Incident Streams 2021 off the Deep End: Deeper Annotations and Evaluations in Twitter

Published

May 2, 2022

Author(s)

Cody Buntain, Richard McCreadie, Ian Soboroff

Abstract

This paper summarizes the final year of the four-year Incident Streams track (TREC-IS), which has produced a large dataset comprising 136,263 annotated tweets, spanning 98 crisis events. Goals of this final year were twofold: 1) to add new categories for assessing messages, with a focus on characterizing the audience, author, and images associated with these messages, and 2) to significantly enlarge the TREC-IS dataset with new events, with an emphasis of deeper pools for sampling. Beyond these two goals, TREC-IS has nearly doubled the number of annotated messages per event for the 26 crises introduced in 2021 and has released a new parallel dataset of 312,546 images associated with crisis content – with 7,297 tweets having annotations about their embedded images. Our analyses of this new crisis data yields new insights about the context of a tweet; e.g., messages intended for a local audience and those that contain images of weather forecasts and infographics have higher than average assessments of priority but are relatively rare. Tweets containing images, however, have significantly higher perceived priorities than tweets without images. Moving to deeper pools, while tending to lower classification performance, also does not generally impact performance rankings or alter distributions of information-types. We end this paper with a discussion of these datasets, analyses, their implications, and how they contribute both new data and insights to the broader crisis informatics community.

Proceedings Title

Proceedings of the 19th Information Systems for Crisis Response and Management Conference

Conference Dates

May 22-25, 2022

Conference Location

Tarbes, FR

Conference Title

Information Systems for Crisis Response and Management

Pub Type

Conferences

Download Paper

Local Download

Keywords

crisis informatics, information retrieval, filtering

Citation

Buntain, C. , McCreadie, R. and Soboroff, I. (2022), Incident Streams 2021 off the Deep End: Deeper Annotations and Evaluations in Twitter, Proceedings of the 19th Information Systems for Crisis Response and Management Conference, Tarbes, FR, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=934607 (Accessed April 26, 2024)