A Comparison of Pooled and Sampled Relevance Judgments in the TREC 2006 Terabyte Track

Published

May 27, 2007

Author(s)

Ian M. Soboroff

Abstract

Pooling is the most common technique used to build modern test collections. Evidence is mounting that pooling may not yield reusable test collections for very large document sets. This paper describes the approach taken in the TREC 2006 Terabyte Track: an initial shallow pool was judged to gather relevance information, which was then used to draw a random sample of further documents to judge. The sample judgments rank systems somewhat differently than the pool. Some analysis and plans for further research are discussed.

Proceedings Title

Proceedings of the First Internation Workshop on Evaluating Information Access (EVIA 2007)

Conference Dates

May 1, 2007

Conference Location

Tokyo, JA

Conference Title

First Internation Workshop on Evaluating Information Access (EVIA 2007)

Pub Type

Conferences

Download Paper

Local Download

Keywords

bias, information retrieval evaluation, pooling, random sampling, test collections

Citation

Soboroff, I. (2007), A Comparison of Pooled and Sampled Relevance Judgments in the TREC 2006 Terabyte Track, Proceedings of the First Internation Workshop on Evaluating Information Access (EVIA 2007), Tokyo, JA, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=51135 (Accessed October 11, 2025)