Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Dynamic Test Collections: Measuring Search Effectiveness on the Live Web

Published

Author(s)

Ian M. Soboroff

Abstract

Existing methods for measuring the quality of search algorithms use a static collection of documents. A set of queries and a mapping from the queries to the relevant documents allow the experimenter to see how well different search engines or engine configurations retrieve the correct answers. This methodology assumes that the document set and thus the set of relevant documents are unchanging. In this paper, we abandon the static collection requirement. We begin with a recent TEXT REtrieval Conference (TREC) collection created from a web crawl, and analyze how the documents in that collection have changed over time. We determine how the decayed collection to measure a live web search system. We employ novel measures of search effectiveness that are robust despite incomplete relevance information. Lastly, we propose a methodology of "collection maintenance" which supports measuring search effectiveness both for a single system and between systems run at different points in time.
Proceedings Title
Proceedings of the Annual International ACM SIGIR Conference on Research and Development inInformation Retrieval
Conference Title
29th Annual Conference on Research adn Development in Information Retrieval (SIGIR 2006)

Keywords

dynamic document collections, information retrieval evaluation, test collections, web search evaluation

Citation

Soboroff, I. (2007), Dynamic Test Collections: Measuring Search Effectiveness on the Live Web, Proceedings of the Annual International ACM SIGIR Conference on Research and Development inInformation Retrieval, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=50846 (Accessed March 1, 2024)
Created January 22, 2007, Updated February 17, 2017