Using Cross-Evaluation to Evaluate Interactive QA Systems
Emile L. Morse, Ying Sun, Paul B. Kantor
Abstract: In this paper, we report on an experiment to assess the possibility of rigorous evaluation of interactive Question-Answering (QA) systems using the Cross-Evaluation method. This method takes into account the effects of tasks and context, and of the users of the systems. Statistical techniques are used to remove these effects, isolating the effect of the system itself. The results show that this approach yields meaningful measurements of the impact of systems on user task performance using a surprisingly small number of subjects and without relying on pre-determined judgments of the quality, or of the relevance of materials. We conclude that the method is indeed effective for comparing end-to-end question-answering systems, and for comparing interactive systems with high efficiency.
Journal of the American Society for Information Science and Technology
, Sun, Y.
and Kantor, P.
Using Cross-Evaluation to Evaluate Interactive QA Systems, Journal of the American Society for Information Science and Technology, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=908567
(Accessed December 3, 2023)