Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Evaluation of 2-Way Iraqi Arabic-English Speech Translation Systems Using Automated Metrics

Published

Author(s)

Gregory A. Sanders, Sherri Condon, Mark Arehart, Dan Parvaz, Christy Doran, John Aberdeen

Abstract

The Defense Advanced Research Projects (DARPA) Spoken Language Communication and Translation System for Tactical Use (TRANSTAC) program faced many challenges in applying automated measures of translation quality to Iraqi Arabic-English speech translation dialogues. Features of speech data in general and of Iraqi Arabic data in particular undermine basic assumptions of automated measures that depend on matching system outputs to reference translations. We show that scores for translation into Iraqi Arabic exhibit higher correlations with human judgments when they are computed from normalized system outputs and reference translations. Orthographic normalization, lexical normalization, and operations involving light stemming resulted in higher correlations with human judgments. Another challenge for use of automated metrics in the TRANSTAC program was the relatively small amount of test data available for evaluation. We present evidence that the datasets of 500-600 utterances for each language which we used to evaluate the systems are adequate for scoring and comparing among different systems.
Citation
Machine Translation
Volume
26
Issue
1-2

Keywords

Arabic, machine translation, evaluation, automated metrics, speech translation

Citation

Sanders, G. , Condon, S. , Arehart, M. , Parvaz, D. , Doran, C. and Aberdeen, J. (2011), Evaluation of 2-Way Iraqi Arabic-English Speech Translation Systems Using Automated Metrics, Machine Translation, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=907859 (Accessed March 28, 2024)
Created September 22, 2011, Updated February 19, 2017