Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Evaluation by Highly Relevant Documents



Ellen M. Voorhees


Given the size of the web, the search engine industry has argued that engines should be evaluated by their ability to retrieve highlyrelevant pages rather than all possible relevant pages. To explore the role highly relevant documents play in retrieval system evaluation,assessors for the TREC-9 web track used a three-point relevance scale and also selected best pages for each topic. The relative effectiveness of runs evaluated by different relevant document sets differed, confirming the hypothesis that different retrieval techniques work better for retrieving highly relevant documents. Yet evaluating by highly relevant documents can be unstable since there are relatively few highly relevant documents. TREC assessors frequently disagreed in their selection of the best page, and subsequent evaluation by best page across different assessors varied widely. The discounted cumulative gain measure introduced by Jarvelin and Kekalainen increases evaluation stability by incorporating all relevance judgments while still giving precedence to highly relevant documents.
ACM Special Interest Group in Information Retrieval (SIGIR)


search engine evaluation, text retrieval evaluation


Voorhees, E. (2001), Evaluation by Highly Relevant Documents, ACM Special Interest Group in Information Retrieval (SIGIR) (Accessed July 15, 2024)


If you have any questions about this publication or are having problems accessing it, please contact

Created January 1, 2001, Updated February 17, 2017