Morse, E.
, Sun, Y.
and Kantor, P.
(2011),
Using Cross-Evaluation to Evaluate Interactive QA Systems, Journal of the American Society for Information Science and Technology, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=908567
(Accessed December 16, 2024)
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
Secure .gov websites use HTTPS
A lock (
) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.
Using Cross-Evaluation to Evaluate Interactive QA Systems
Published
Author(s)
, Ying Sun, Paul B. Kantor
Abstract
Abstract: In this paper, we report on an experiment to assess the possibility of rigorous evaluation of interactive Question-Answering (QA) systems using the Cross-Evaluation method. This method takes into account the effects of tasks and context, and of the users of the systems. Statistical techniques are used to remove these effects, isolating the effect of the system itself. The results show that this approach yields meaningful measurements of the impact of systems on user task performance using a surprisingly small number of subjects and without relying on pre-determined judgments of the quality, or of the relevance of materials. We conclude that the method is indeed effective for comparing end-to-end question-answering systems, and for comparing interactive systems with high efficiency.Citation
Journal of the American Society for Information Science and Technology
Volume
62
Issue
9
Pub Type
Journals
Download Paper
Citation
Issues
If you have any questions about this publication or are having problems accessing it, please contact reflib@nist.gov.
Created July 5, 2011, Updated February 19, 2017