Effects of Word Error Rate in the DARPA Communicator Data During 2000 and 2001

Published

September 16, 2002

Author(s)

Gregory A. Sanders, A N. Le, John S. Garofolo

Abstract

During 2000 and 2001 two large data collections were performed, with paid users doing travel planning using eight Communicator spoken dialogue systems. We analyze the effects of speech recognition accuracy, as measured by Word Error Rate (WER), on other metrics. Analysis shows a linear correlation between WER and task completion metrics, which unexpectedly remained linear even at high levels of WER. The determinants of user satisfaction are more complex, but we present evidence suggesting a somewhat linear correlation between WER and satisfaction for WER less than 35% or 40% in 2001 (stronger correlation in 2000). The size of the effect of increasing WER on task completion is about half as large in 2001 as in 2000, demonstrating improved ability to accomplish tasks despite high WER, and we consider this to be an important accomplishment of the research groups who built these Communicator systems.

Proceedings Title

International Conference on Spoken Language Processing, 2002

Conference Location

FL

Pub Type

Conferences

Download Paper

Local Download

Keywords

communicator, efficiency, speech recognition, spoken dialogue systems, task completion, user satisfaction, word error rate

Citation

Sanders, G. , Le, A. and Garofolo, J. (2002), Effects of Word Error Rate in the DARPA Communicator Data During 2000 and 2001, International Conference on Spoken Language Processing, 2002, FL, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=51049 (Accessed January 22, 2025)