Consider information retrieval systems that respond to a query (a natural language statement of a topic, an information need) with an ordered list of 1000 documents from the document collection. From the responses to queries that all express the same topic, one can discern how the words associated with a topic result in particular system behavior. From what is discerned from different topics, one can hypothesize abstract topic factors that govern system performance. An example of such a factor is the specificity of the topic's primary key word. We illustrate these ideas with system responses for a group of topics from NIST's Text REtrieval Conference (TREC). We begin by showing how one can analyze the queries and system responses for a particular topic, and how one can formulate topic factors through comparison of different topics. We then show how to choose from a group of topics, pairs likely to offer particularly informative comparisons. Finally, we consider prospects for using topic factors in generalizing from observed system responses for a group of topic to conclusions about system performance on topics as yet unseen.
Citation: Journal of Classification
Volume: 1 No 1
Pub Type: Journals
archetypal analysis, indicator, multidimensional scaling, natural language