Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

A Comparison of Pooled and Sampled Relevance Judgments

Published

Author(s)

Ian M. Soboroff

Abstract

Test collections are most useful when they are reusable, that is, when they can be reliably used to rank systems that did not contribute to the pools. Pooled relevance judgments for very large collections may not be reusable for two reasons: they will be very sparse and not sufficiently complete, and they may be biased in the sense that they will unfairly rank some class of systems. The TREC 2006 terabyte track judged both a pool and a deep random sample in order to measure the effects of sparseness and bias.
Proceedings Title
Proceedings of the Annual International ACM SIGIR Conference on Research and Development inInformation Retrieval
Conference Title
Annual Conference on Research adn Development in Information Retrieval (SIGIR )

Keywords

nformation retrieval, test collections

Citation

Soboroff, I. (2007), A Comparison of Pooled and Sampled Relevance Judgments, Proceedings of the Annual International ACM SIGIR Conference on Research and Development inInformation Retrieval (Accessed July 20, 2024)

Issues

If you have any questions about this publication or are having problems accessing it, please contact reflib@nist.gov.

Created August 29, 2007, Updated February 19, 2017