Skip to main content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.


The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

On the Behavior of PRES Using Incomplete Judgment Sets



Ellen M. Voorhees


PRES, the Patent Retrieval Evaluation Score, is a family of retrieval system evaluation measures that combines recall and user effort to better reflect the quality of a retrieval run with respect to recall-oriented search tasks. Previous analysis of the measure was done using the test collection for the CLEF-IP 2009 track, a collection that contains a limited range of number of relevant documents, making it difficult to assess the behavior of PRES for varying recall contexts. This paper examines the effect of incomplete judgments on PRES scores using the well-studied TREC-8 ad hoc test collection, a collection with a much more varied number-of-relevants profile. Experiments with small judgment sets created through a typical collection-building process show the PRES measures are resilient to incomplete judgment sets.
Proceedings Title
Proceedings of the 2015 ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR)
Conference Dates
September 27-30, 2015
Conference Location
Northampton, MA


incomplete judgments, information retrieval evaluation, test collections, TREC
Created September 30, 2015, Updated November 10, 2018