Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Test Collection Diagnosis and Treatment



Ian M. Soboroff


Test collections are a mainstay of information retrieval research. Since the 1990s, large reusable test collections have been developed in the context of community evaluations such as TREC, NTCIR, CLEF, and INEX. Recently, advances in pooling practice as well as crowdsourcing technologies have placed test collection building back into the hands of the small research group or company. In all of these cases, practitioners should be aware of, and concerned about the quality of test collections. This paper surveys work in test collection quality measures, references case studies to illustrate their use, and provides guidelines on assessing the quality of test collections in practice.
Proceedings Title
Proceedings of the 2010 Workshop on Evaluating Information Access (EVIA 2010)
Conference Dates
June 15-18, 2010
Conference Location


test collections, information retrieval


Soboroff, I. (2010), Test Collection Diagnosis and Treatment, Proceedings of the 2010 Workshop on Evaluating Information Access (EVIA 2010), Tokyo, -1, [online], (Accessed April 21, 2024)
Created June 15, 2010, Updated February 19, 2017