Comparing Interactive Information Retrieval Systems Across Sites: The TREC-6 Interactive Track Matrix Experiment
E S. Lagergren, Paul D. Over
This is a case study in the design and analysis of a 9-site TREC-6 experiment aimed at comparing the performance of 12 interactive information retrieval (IR) systems on a shared problem: a question-answering task, 6 statements of information need, and a collection of 210,158 articles from the Financial Times of London 1991-1994. The study discusses the application of experimental design principles and the use of a shared control IR system in addressing the problems of comparing experimental interactive IR systems across sites: isolating the effects of topics, human searchers, and other site specific factors within an affordable design. The results confirm the dominance of the topic effect, show the searcher effect is almost as often absent as present, and indicate that for several sites the 2-factor interactions are negligible. An analysis of variance found the system effect to be significant pairwise differences.
August 1, 1998
ACM/SIGIR Conference on Research and Development in Information Retrieval
and Over, P.
Comparing Interactive Information Retrieval Systems Across Sites: The TREC-6 Interactive Track Matrix Experiment, ACM/SIGIR Conference on Research and Development in Information Retrieval, Melbourne, AS, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=151737
(Accessed June 2, 2023)