System Performance and Natural Language Expression of Information Needs

Published

January 5, 2005

Author(s)

Walter S. Liggett Jr, C E. Buckley

Abstract

Consider information retrieval systems that respond to a query (a natural language statement of a topic, an information need) with an ordered list of 1000 documents from the document collection. From the responses to queries that all express the same topic, one can discern how the words associated with a topic result in particular system behavior. From what is discerned from different topics, one can hypothesize abstract topic factors that govern system performance. An example of such a factor is the specificity of the topic's primary key word. We illustrate these ideas with system responses for a group of topics from NIST's Text REtrieval Conference (TREC). We begin by showing how one can analyze the queries and system responses for a particular topic, and how one can formulate topic factors through comparison of different topics. We then show how to choose from a group of topics, pairs likely to offer particularly informative comparisons. Finally, we consider prospects for using topic factors in generalizing from observed system responses for a group of topic to conclusions about system performance on topics as yet unseen.

Citation

Journal of Classification

Volume

1 No 1

Pub Type

Journals

Keywords

archetypal analysis, indicator, multidimensional scaling, natural language

Citation

Liggett, W. and Buckley, C. (2005), System Performance and Natural Language Expression of Information Needs, Journal of Classification (Accessed December 14, 2024)