Evaluation Methodology and Metrics Employed to Assess the TRANSTAC Two-way, Speech-to-Speech Translation Systems

Published

February 1, 2013

Author(s)

Gregory A. Sanders, Brian A. Weiss, Craig I. Schlenoff, Michelle P. Steves, Sherri Condon

Abstract

One of the most difficult challenges that military personnel face when operating in foreign countries is clear and successful communication with the local population. To address this issue, the Defense Advanced Research Projects Agency (DARPA) is funding academic institutions and industrial organizations through the Spoken Language Communication and Translation System for Tactical Use (TRANSTAC) program to develop practical machine translation systems. The goal of the TRANSTAC program is to demonstrate capabilities to rapidly develop and field free-form, two-way, speech-to-speech translation systems that enable speakers of different languages to communicate with one another in real-world tactical situations without an interpreter. Evaluations of these technologies are a significant part of the program and DARPA has asked the National Institute of Standards and Technology (NIST) to lead this effort. This article presents the experimental design of the TRANSTAC evaluations and the metrics, both quantitative and qualitative, that were captured to comprehensively assess the systems’ performance.

Citation

Computer Speech and Language

Volume

27

Issue

2

Pub Type

Journals

Download Paper

DOI Link

Keywords

Evaluatio, Performance Metrics, Speech-to-Speech Translation, TRANSTAC

Citation

Sanders, G. , Weiss, B. , Schlenoff, C. , Steves, M. and Condon, S. (2013), Evaluation Methodology and Metrics Employed to Assess the TRANSTAC Two-way, Speech-to-Speech Translation Systems, Computer Speech and Language, [online], https://doi.org/10.1016/j.csl.2011.05.001 (Accessed April 23, 2024)