An Exploratory Study of the W3C Mailing List Test Collection for Retrieval of Emails with Pro/Con Arguments
Yejun Wu, Douglas W. Oard, Ian M. Soboroff
The W3C mailing list test collection, an information retrieval test collection for email, was developed for the TREC Enterprise Search track in 2005. One task in that track was to retrieve emails that contribute at least one pro/con related to a specific topic. This paper describes the test collection and presents a preliminary evaluation of its suitability for evaluating such systems, including an analysis of topic types found in the collection, characterization of intrassessor agreement on pro/con judgments, and an example of the evaluation results that can be obtained using the collection. There is clear evidence that the collection is useful in its present form, but several areas for improvement can be identified. In particular, some topic types found in the collection do not seem well suited to pro/con judgement. The paper concludes with suggestions for future work on the design of test collections and information retrieval systems for this task.
email test collections, intrassessor agreement, sentiment analysis, test collection design
, Oard, D.
and Soboroff, I.
An Exploratory Study of the W3C Mailing List Test Collection for Retrieval of Emails with Pro/Con Arguments, Conference on Email and Anti-Spam | 3rd | |, Mountain View, CA, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=50829
(Accessed July 26, 2021)