Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Incident Streams 2021 off the Deep End: Deeper Annotations and Evaluations in Twitter

Published

Author(s)

Cody Buntain, Richard McCreadie, Ian Soboroff

Abstract

This paper summarizes the final year of the four-year Incident Streams track (TREC-IS), which has produced a large dataset comprising 136,263 annotated tweets, spanning 98 crisis events. Goals of this final year were twofold: 1) to add new categories for assessing messages, with a focus on characterizing the audience, author, and images associated with these messages, and 2) to significantly enlarge the TREC-IS dataset with new events, with an emphasis of deeper pools for sampling. Beyond these two goals, TREC-IS has nearly doubled the number of annotated messages per event for the 26 crises introduced in 2021 and has released a new parallel dataset of 312,546 images associated with crisis content – with 7,297 tweets having annotations about their embedded images. Our analyses of this new crisis data yields new insights about the context of a tweet; e.g., messages intended for a local audience and those that contain images of weather forecasts and infographics have higher than average assessments of priority but are relatively rare. Tweets containing images, however, have significantly higher perceived priorities than tweets without images. Moving to deeper pools, while tending to lower classification performance, also does not generally impact performance rankings or alter distributions of information-types. We end this paper with a discussion of these datasets, analyses, their implications, and how they contribute both new data and insights to the broader crisis informatics community.
Proceedings Title
Proceedings of the 19th Information Systems for Crisis Response and Management Conference
Conference Dates
May 22-25, 2022
Conference Location
Tarbes, FR
Conference Title
Information Systems for Crisis Response and Management

Keywords

crisis informatics, information retrieval, filtering

Citation

Buntain, C. , McCreadie, R. and Soboroff, I. (2022), Incident Streams 2021 off the Deep End: Deeper Annotations and Evaluations in Twitter, Proceedings of the 19th Information Systems for Crisis Response and Management Conference, Tarbes, FR, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=934607 (Accessed April 24, 2024)
Created May 2, 2022, Updated February 24, 2023