Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Building a Filtering Test Collection for TREC 2002



Ian M. Soboroff, S E. Robertson


Test collections for the filtering track in TREC have typically used either past sets of relevance judgments, or categorized collections such as Reuters Corpus Volume 1 or OHSUMED, because filtering systems need relevance judgments during the experiment for training and adaptation. For TREC 2002, we constructed an entirely new set of search topics for the Reuters Corpus for measuring filtering systems. Our method for building the topics involved multiple iterations of feedback from assessors, and fusion of results from multiple search systems using different search algorithms. We also developed a second set of inexpensive topics based on categories in the document collection. We found that the initial judgments made for the experiment were sufficient; subsequent pooled judging changed system rankings very little. We also found that systems performed very differently on the category topics than on the assessor-built topics.
ACM Special Interest Group in Information Retrieval (SIGIR)


information filtering, relevance feedback, test collections


Soboroff, I. and Robertson, S. (2003), Building a Filtering Test Collection for TREC 2002, ACM Special Interest Group in Information Retrieval (SIGIR) (Accessed June 19, 2024)


If you have any questions about this publication or are having problems accessing it, please contact

Created July 28, 2003, Updated February 19, 2017