Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Using Cross-Evaluation to Evaluate Interactive QA Systems



Emile L. Morse, Ying Sun, Paul B. Kantor


Abstract: In this paper, we report on an experiment to assess the possibility of rigorous evaluation of interactive Question-Answering (QA) systems using the Cross-Evaluation method. This method takes into account the effects of tasks and context, and of the users of the systems. Statistical techniques are used to remove these effects, isolating the effect of the system itself. The results show that this approach yields meaningful measurements of the impact of systems on user task performance using a surprisingly small number of subjects and without relying on pre-determined judgments of the quality, or of the relevance of materials. We conclude that the method is indeed effective for comparing end-to-end question-answering systems, and for comparing interactive systems with high efficiency.
Journal of the American Society for Information Science and Technology


Morse, E. , Sun, Y. and Kantor, P. (2011), Using Cross-Evaluation to Evaluate Interactive QA Systems, Journal of the American Society for Information Science and Technology, [online], (Accessed June 12, 2024)


If you have any questions about this publication or are having problems accessing it, please contact

Created July 5, 2011, Updated February 19, 2017