On the Behavior of PRES Using Incomplete Judgment Sets

Published

September 30, 2015

Author(s)

Ellen M. Voorhees

Abstract

PRES, the Patent Retrieval Evaluation Score, is a family of retrieval system evaluation measures that combines recall and user effort to better reflect the quality of a retrieval run with respect to recall-oriented search tasks. Previous analysis of the measure was done using the test collection for the CLEF-IP 2009 track, a collection that contains a limited range of number of relevant documents, making it difficult to assess the behavior of PRES for varying recall contexts. This paper examines the effect of incomplete judgments on PRES scores using the well-studied TREC-8 ad hoc test collection, a collection with a much more varied number-of-relevants profile. Experiments with small judgment sets created through a typical collection-building process show the PRES measures are resilient to incomplete judgment sets.

Proceedings Title

Proceedings of the 2015 ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR)

Conference Dates

September 27-30, 2015

Conference Location

Northampton, MA

Pub Type

Conferences

Download Paper

DOI Link

Keywords

incomplete judgments, information retrieval evaluation, test collections, TREC

Citation

Voorhees, E. (2015), On the Behavior of PRES Using Incomplete Judgment Sets, Proceedings of the 2015 ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR), Northampton, MA, [online], https://doi.org/10.1145/2808194.2809484 (Accessed April 25, 2024)