Analysis of the Query Track of the Eighth Text Retrieval Conference (TREC-8) shows how the presence or absence of a word or two in a query can make a large difference in information retrieval performance. The Query Track contains results from 8 systems each run on 21 natural-language queries that were formulated for each topic of 50. To compare the queries for a topic, we consider both discrimination against irrelevant documents and ordering of relevant documents in system outputs. Each of these aspects of performance leads to a graph. We present these graphs for 8 topics and summarize what is shown for the other 42. The 50 topics demonstrate the performance effects of word omissions, word additions, meaningful word substitutions, and word changes with only system ramifications. Regarding query formulation, the analysis in this paper shows the need to overcome mismatches between query terms and terms used in the documentation collection.
Citation: ACM/SIGIR Conference on Research and Development in Information Retrieval
Pub Type: Journals
evaluation, field/empirical studies of the informati, information seeking behavior, testing methodology