An Exploratory Study of the W3C Mailing List Test Collection for Retrieval of Emails with Pro/Con Arguments

Published

August 21, 2006

Author(s)

Yejun Wu, Douglas W. Oard, Ian Soboroff

Abstract

The W3C mailing list test collection, an information retrieval test collection for email, was developed for the TREC Enterprise Search track in 2005. One task in that track was to retrieve emails that contribute at least one pro/con related to a specific topic. This paper describes the test collection and presents a preliminary evaluation of its suitability for evaluating such systems, including an analysis of topic types found in the collection, characterization of intrassessor agreement on pro/con judgments, and an example of the evaluation results that can be obtained using the collection. There is clear evidence that the collection is useful in its present form, but several areas for improvement can be identified. In particular, some topic types found in the collection do not seem well suited to pro/con judgement. The paper concludes with suggestions for future work on the design of test collections and information retrieval systems for this task.

Proceedings Title

Conference on Email and Anti-Spam | 3rd | |

Conference Dates

July 27-28, 2006

Conference Location

Mountain View, CA, US

Conference Title

CEAS 2006 Third Conference on Email and Antispam

Pub Type

Conferences

Download Paper

Local Download

Keywords

email test collections, intrassessor agreement, sentiment analysis, test collection design

Citation

Wu, Y. , Oard, D. and Soboroff, I. (2006), An Exploratory Study of the W3C Mailing List Test Collection for Retrieval of Emails with Pro/Con Arguments, Conference on Email and Anti-Spam | 3rd | |, Mountain View, CA, US, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=50829 (Accessed April 24, 2024)