Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Retrieval Evaluation with Incomplete Information

Published

Author(s)

C E. Buckley, Ellen M. Voorhees

Abstract

This paper examines whether the Cranfield evaluation methodology is robust to gross violations of the completeness assumption (i.e., the assumption that all relevant documents within a test collection have been identified and are present in the collection). We show that current evaluation measures are not robust to substantially incomplete relevance judgments. A new measure is introduced that is both highly correlated with existing measures when complete judgments are available and more robust to incomplete judgment sets. This finding suggests that substantially larger or dynamic test collections built using current pooling practices should be viable laboratory tools, despite the fact that the relevance information will be incomplete and imperfect.
Proceedings Title
Proceedings of the Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Sheffield, UK, July 2004
Conference Dates
July 1, 2004
Conference Location
Sheffield, 1, UK
Conference Title
ACM Special Interest Group in Information Retrieval (SIGIR)

Keywords

evaluation, information retrieval, test collections

Citation

Buckley, C. and Voorhees, E. (2004), Retrieval Evaluation with Incomplete Information, Proceedings of the Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Sheffield, UK, July 2004, Sheffield, 1, UK, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=150469 (Accessed July 12, 2024)

Issues

If you have any questions about this publication or are having problems accessing it, please contact reflib@nist.gov.

Created June 30, 2004, Updated October 12, 2021