Reliable Information Retrieval Evaluation With Incomplete and Biased Judgements

Published

July 23, 2007

Author(s)

Stefan Buttcher, Charles L. Clarke, Peter C. Yeung, Ian Soboroff

Abstract

Information retrieval evaluation based on the pooling method is inherently biased against systems that did not contribute to the pool of judged documents. This may distort the results obtained about the relative quality of the systems evaluated and thus lead to incorrect conclusions about the performance of a particular ranking technique. We examine the magnitude of this and explore how it can be countered by automatically building an unbiased set of judgements from the original, biased judgements obtained through pooling. We compare the performance of this method with other approaches to the problem of incomplete judgements, such as bpref, and show that the proposed method leads to higher evaluation accuracy, especially if the set of manual judgements is rich in documents, but highly biased against some systems.

Proceedings Title

Proceedings of the Annual International ACM SIGIR Conference on Research and Development inInformation Retrieval

Conference Location

, USA

Conference Title

Annual Conference on Research adn Development in Information Retrieval (SIGIR )

Pub Type

Conferences

Keywords

information retrieval, test collections

Citation

Buttcher, S. , Clarke, C. , Yeung, P. and Soboroff, I. (2007), Reliable Information Retrieval Evaluation With Incomplete and Biased Judgements, Proceedings of the Annual International ACM SIGIR Conference on Research and Development inInformation Retrieval, , USA (Accessed October 10, 2025)