Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Effects of Word Error Rate in the DARPA Communicator Data During 2000 and 2001

Published

Author(s)

Gregory A. Sanders, A N. Le, John S. Garofolo

Abstract

During 2000 and 2001 two large data collections were performed, with paid users doing travel planning using eight Communicator spoken dialogue systems. We analyze the effects of speech recognition accuracy, as measured by Word Error Rate (WER), on other metrics. Analysis shows a linear correlation between WER and task completion metrics, which unexpectedly remained linear even at high levels of WER. The determinants of user satisfaction are more complex, but we present evidence suggesting a somewhat linear correlation between WER and satisfaction for WER less than 35% or 40% in 2001 (stronger correlation in 2000). The size of the effect of increasing WER on task completion is about half as large in 2001 as in 2000, demonstrating improved ability to accomplish tasks despite high WER, and we consider this to be an important accomplishment of the research groups who built these Communicator systems.
Proceedings Title
International Conference on Spoken Language Processing, 2002
Conference Location
FL

Keywords

communicator, efficiency, speech recognition, spoken dialogue systems, task completion, user satisfaction, word error rate

Citation

Sanders, G. , Le, A. and Garofolo, J. (2002), Effects of Word Error Rate in the DARPA Communicator Data During 2000 and 2001, International Conference on Spoken Language Processing, 2002, FL, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=51049 (Accessed July 24, 2024)

Issues

If you have any questions about this publication or are having problems accessing it, please contact reflib@nist.gov.

Created September 16, 2002, Updated February 17, 2017