Harman, D.
(2001),
The Importance of Focused Evaluations: A Case Study of TREC and DUC, , , [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=150251 (Accessed May 4, 2026)
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
Secure .gov websites use HTTPS
A lock (
) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.
The Importance of Focused Evaluations: A Case Study of TREC and DUC
Published
Author(s)
Abstract
Evaluation has always been an important part of scientific research, and in information retrieval, this evaluation has mostly been done using test collections. In 1992, a new test collection was built at the National Institute of Standards and Technology (NIST), and a focused evaluation (the Text REtrieval Conference or TREC) was started to use the collection. Results from nearly 12 years of this focused evaluation show significant technology transfer across systems, leading to major improvements in system performance. Focused evaluations also create the ability to target specific problems in language technology, such as retrieval across languages, and to design tasks for evaluation such that issues can be studied concurrently by multiple groups. This chapter will discuss some of the tasks that have been examined in TREC, including critical factors in the design of those evaluations. Additionally a second focused evaluation, the Document Understanding Conference (DUC) which evaluates text summarization, will be discussed.Citation
Progress in Natural Language Processing & Information Retrieval: A Festschrift for Karen Sparck Jones
Publisher Info
,
Pub Type
Books
Download Paper
Keywords
evaluation, information retrieval, TREC
Citation
Issues
If you have any questions about this publication or are having problems accessing it, please contact [email protected].
Created March 1, 2001, Updated February 19, 2017