Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

An Exploratory Study of the W3C Mailing List Test Collection for Retrieval of Emails with Pro/Con Arguments

Published

Author(s)

Yejun Wu, Douglas W. Oard, Ian M. Soboroff

Abstract

The W3C mailing list test collection, an information retrieval test collection for email, was developed for the TREC Enterprise Search track in 2005. One task in that track was to retrieve emails that contribute at least one pro/con related to a specific topic. This paper describes the test collection and presents a preliminary evaluation of its suitability for evaluating such systems, including an analysis of topic types found in the collection, characterization of intrassessor agreement on pro/con judgments, and an example of the evaluation results that can be obtained using the collection. There is clear evidence that the collection is useful in its present form, but several areas for improvement can be identified. In particular, some topic types found in the collection do not seem well suited to pro/con judgement. The paper concludes with suggestions for future work on the design of test collections and information retrieval systems for this task.
Proceedings Title
Conference on Email and Anti-Spam | 3rd | |
Conference Dates
July 27-28, 2006
Conference Location
Mountain View, CA
Conference Title
CEAS 2006 Third Conference on Email and Antispam

Keywords

email test collections, intrassessor agreement, sentiment analysis, test collection design

Citation

Wu, Y. , Oard, D. and Soboroff, I. (2006), An Exploratory Study of the W3C Mailing List Test Collection for Retrieval of Emails with Pro/Con Arguments, Conference on Email and Anti-Spam | 3rd | |, Mountain View, CA, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=50829 (Accessed May 14, 2021)
Created August 21, 2006, Updated February 17, 2017