Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

A Comparison of Pooled and Sampled Relevance Judgments in the TREC 2006 Terabyte Track

Published

Author(s)

Ian M. Soboroff

Abstract

Pooling is the most common technique used to build modern test collections. Evidence is mounting that pooling may not yield reusable test collections for very large document sets. This paper describes the approach taken in the TREC 2006 Terabyte Track: an initial shallow pool was judged to gather relevance information, which was then used to draw a random sample of further documents to judge. The sample judgments rank systems somewhat differently than the pool. Some analysis and plans for further research are discussed.
Proceedings Title
Proceedings of the First Internation Workshop on Evaluating Information Access (EVIA 2007)
Conference Dates
May 1, 2007
Conference Location
Tokyo, JA
Conference Title
First Internation Workshop on Evaluating Information Access (EVIA 2007)

Keywords

bias, information retrieval evaluation, pooling, random sampling, test collections

Citation

Soboroff, I. (2007), A Comparison of Pooled and Sampled Relevance Judgments in the TREC 2006 Terabyte Track, Proceedings of the First Internation Workshop on Evaluating Information Access (EVIA 2007), Tokyo, JA, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=51135 (Accessed April 18, 2024)
Created May 27, 2007, Updated February 17, 2017