Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Effects of Speech Recognition Accuracy on Performance of DARPA Communicator Spoken Dialogue Systems

Published

Author(s)

Gregory A. Sanders, A N. Le

Abstract

The DARPA Communicator program explored ways to construct better spoken-dialogue systems, with which users interact via speech alone to perform relatively complex tasks such as travel planning. During 2000 and 2001 two large data sets were collected from sessions in which paid users did travel planning using the Communicator systems that had been built by eight research groups. The research groups improved their systems intensively during the ten months between the two data collections. In this paper, we analyze those data sets to estimate the effects of speech recognition accuracy, as measured by Word Error Rate (WER), on other metrics. We found correlation between WER and Task Completion. That correlation, unexpectedly, remained more or less linear even for high values of WER. The picture for User Satisfaction metrics is more complex: we found little effect of WER on User Satisfaction for WER less than about 35% to 40% in the 2001 data. The size of the effect of WER on Task Completion was less in 2001 than in 2000, and we believe this difference is due to improved strategies for accomplishing tasks despite speech recognition errors, which is an important accomplishment of the research groups who built the Communicator implementations. We show that additional factors must account for much of the variability in task success, and we present multivariate linear regression models for task success on the 2001 data. We also discuss the apparent gaps in the coverage of metrics for spoken dialogue systems.
Citation
International Journal of Speech Technology (ISSN 1381-2416)
Volume
7
Issue
No. 4

Keywords

communicator, completion, efficiency, spoken dialogue, task success, word error rate user satisfaction

Citation

Sanders, G. and Le, A. (2004), Effects of Speech Recognition Accuracy on Performance of DARPA Communicator Spoken Dialogue Systems, International Journal of Speech Technology (ISSN 1381-2416) (Accessed June 21, 2024)

Issues

If you have any questions about this publication or are having problems accessing it, please contact reflib@nist.gov.

Created October 1, 2004, Updated February 17, 2017