Evaluating the Evaluation: A Case Study Using the TREC 2002 Question Answering Track

Published

May 1, 2003

Author(s)

Ellen M. Voorhees

Abstract

Evaluating competing technologies on a common problem set is a powerful way to improve the state of the art and hasten technology transfer. Yet poorly designed evaluations can waste research effort or even mislead researchers with faulty conclusions. Thus it is important to examine the quality of a new evaluation task to establish its reliability. This paper provides an example of one such assessment by analyzing the task within the TREC 2002 question answering (QA) track. The analysis demonstrates that comparative results from the new task are stable, and empirically estimates the size of the difference required between scores to confidently conclude that two runs are different.

Proceedings Title

Proceedings of the 2003 Human Language Technology Conference (HLT-NAACL 03)

Conference Dates

May 1, 2003

Conference Location

Edmonton, CA

Conference Title

International Conference on Human Language Technology

Pub Type

Conferences

Download Paper

Local Download

Keywords

evaluation, question answering, TREC

Citation

Voorhees, E. (2003), Evaluating the Evaluation: A Case Study Using the TREC 2002 Question Answering Track, Proceedings of the 2003 Human Language Technology Conference (HLT-NAACL 03), Edmonton, CA, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=50781 (Accessed October 15, 2025)