Comparing Interactive Information Retrieval Systems Across Sites: The TREC-6 Interactive Track Matrix Experiment

Published

July 15, 1998

Author(s)

E S. Lagergren, Paul D. Over

Abstract

This is a case study in the design and analysis of a 9-site TREC-6 experiment aimed at comparing the performance of 12 interactive information retrieval (IR) systems on a shared problem: a question-answering task, 6 statements of information need, and a collection of 210,158 articles from the Financial Times of London 1991-1994. The study discusses the application of experimental design principles and the use of a shared control IR system in addressing the problems of comparing experimental interactive IR systems across sites: isolating the effects of topics, human searchers, and other site specific factors within an affordable design. The results confirm the dominance of the topic effect, show the searcher effect is almost as often absent as present, and indicate that for several sites the 2-factor interactions are negligible. An analysis of variance found the system effect to be significant pairwise differences.

Conference Dates

August 1, 1998

Conference Location

Melbourne, AS

Conference Title

ACM/SIGIR Conference on Research and Development in Information Retrieval

Pub Type

Conferences

Download Paper

Local Download

Keywords

experimental design, information retrieval, user study

Citation

Lagergren, E. and Over, P. (1998), Comparing Interactive Information Retrieval Systems Across Sites: The TREC-6 Interactive Track Matrix Experiment, ACM/SIGIR Conference on Research and Development in Information Retrieval, Melbourne, AS, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=151737 (Accessed July 26, 2024)