Soboroff, I.
(2007),
A Comparison of Pooled and Sampled Relevance Judgments in the TREC 2006 Terabyte Track, Proceedings of the First Internation Workshop on Evaluating Information Access (EVIA 2007), Tokyo, JA, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=51135
(Accessed December 10, 2024)
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
Secure .gov websites use HTTPS
A lock (
) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.
A Comparison of Pooled and Sampled Relevance Judgments in the TREC 2006 Terabyte Track
Published
Author(s)
Abstract
Pooling is the most common technique used to build modern test collections. Evidence is mounting that pooling may not yield reusable test collections for very large document sets. This paper describes the approach taken in the TREC 2006 Terabyte Track: an initial shallow pool was judged to gather relevance information, which was then used to draw a random sample of further documents to judge. The sample judgments rank systems somewhat differently than the pool. Some analysis and plans for further research are discussed.Proceedings Title
Proceedings of the First Internation Workshop on Evaluating Information Access (EVIA 2007)
Conference Dates
May 1, 2007
Conference Location
Tokyo, JA
Conference Title
First Internation Workshop on Evaluating Information Access (EVIA 2007)
Pub Type
Conferences
Download Paper
Keywords
bias, information retrieval evaluation, pooling, random sampling, test collections
Citation
Issues
If you have any questions about this publication or are having problems accessing it, please contact reflib@nist.gov.
Created May 27, 2007, Updated February 17, 2017