Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

System Performance and Natural Language Expression of Information Needs



Walter S. Liggett Jr, C E. Buckley


Consider information retrieval systems that respond to a query (a natural language statement of a topic, an information need) with an ordered list of 1000 documents from the document collection. From the responses to queries that all express the same topic, one can discern how the words associated with a topic result in particular system behavior. From what is discerned from different topics, one can hypothesize abstract topic factors that govern system performance. An example of such a factor is the specificity of the topic's primary key word. We illustrate these ideas with system responses for a group of topics from NIST's Text REtrieval Conference (TREC). We begin by showing how one can analyze the queries and system responses for a particular topic, and how one can formulate topic factors through comparison of different topics. We then show how to choose from a group of topics, pairs likely to offer particularly informative comparisons. Finally, we consider prospects for using topic factors in generalizing from observed system responses for a group of topic to conclusions about system performance on topics as yet unseen.
Journal of Classification
1 No 1


archetypal analysis, indicator, multidimensional scaling, natural language


Liggett, W. and Buckley, C. (2005), System Performance and Natural Language Expression of Information Needs, Journal of Classification (Accessed July 19, 2024)


If you have any questions about this publication or are having problems accessing it, please contact

Created January 5, 2005, Updated February 17, 2017