Skip to main content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.


The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

The Importance of Focused Evaluations: A Case Study of TREC and DUC



Donna K. Harman


Evaluation has always been an important part of scientific research, and in information retrieval, this evaluation has mostly been done using test collections. In 1992 a new test collection was built at the National Institute of Standards and Technology (NIST), and a focused evaluation (the Text REtrieval Conference or TREC) was started to use the collection. Results from nearly 10 years of this focused evaluation show significant technology transfer across systems, leading to major improvements in system performance. Focused evaluations also create the ability to target specific problems in language technology, such as retrieval across languages, and to design tasks for evaluation such that issues can be studied concurrently by multiple groups. This paper will discuss some of the tasks that have been examined in TREC, including critical factors in the design of those evaluations. Additionally, a new focused evaluation, the Document Understanding Conference (DUC) which will examine text summarization, will be introduced.
Proceedings Second NTCIR Workshop on Research in Chinese & Japanese Text Retrieval
Created January 10, 2007, Updated February 17, 2017