U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

PUBLICATIONS

Reliable Information Retrieval Evaluation With Incomplete and Biased Judgements

Published

Author(s)

Stefan Buttcher, Charles L. Clarke, Peter C. Yeung, Ian Soboroff

Abstract

Information retrieval evaluation based on the pooling method is inherently biased against systems that did not contribute to the pool of judged documents. This may distort the results obtained about the relative quality of the systems evaluated and thus lead to incorrect conclusions about the performance of a particular ranking technique. We examine the magnitude of this and explore how it can be countered by automatically building an unbiased set of judgements from the original, biased judgements obtained through pooling. We compare the performance of this method with other approaches to the problem of incomplete judgements, such as bpref, and show that the proposed method leads to higher evaluation accuracy, especially if the set of manual judgements is rich in documents, but highly biased against some systems.
Proceedings Title
Proceedings of the Annual International ACM SIGIR Conference on Research and Development inInformation Retrieval
Conference Location
, USA
Conference Title
Annual Conference on Research adn Development in Information Retrieval (SIGIR )
Pub Type
Conferences

Keywords

information retrieval, test collections

Citation

Buttcher, S. , Clarke, C. , Yeung, P. and Soboroff, I. (2007), Reliable Information Retrieval Evaluation With Incomplete and Biased Judgements, Proceedings of the Annual International ACM SIGIR Conference on Research and Development inInformation Retrieval, , USA (Accessed May 4, 2024)

Created July 22, 2007, Updated October 12, 2021