Machine Translation

Summary

The Multimodal Information Group's machine translation (MT) program includes several activities contributing to machine translation technology and metrology advancements, primarily through systematic and targeted annual evaluations.

Since 2002, the Multimodal Information Group has coordinated evaluations of text-to-text MT technology through our OpenMT series. NIST-led open evaluations such as OpenMT provides a test bed for experimentation of evaluation techniques that may then be applied to sponsored MT technology evaluations. Similarly, NIST's Metrics for Machine Translation Challenge (MetricsMaTr) provides a forum to research and promote innovative techniques that advance the measurement sciences used in MT evaluations. Participation in these open evaluation activities is open to all researchers that find the tasks of interest and are able to abide by the particular evaluation's task protocols and rules.

Description

Current MT technology evaluation activities:

• OpenMT: A biannual NIST evaluation of text-to-text MT technology. Focus is placed on the core task of MT technology where information learned is applicable to other MT technology types.

Past MT evaluation activities:

• LORELEI/LOREHLT, 2016-2019: Low Resource Languages for Emergent Incidents (LORELEI) was an interdisciplinary DARPA-sponsored program with an MT component. The goal of the program was to dramatically advance the state of computational linguistics and human language technology to enable rapid, low-cost development of capabilities for low-resource languages.  The Low Resource HLT (LoReHLT) open evaluations served to evaluate component technologies relevant to LORELEI.
• BOLT, 2011-2015: Broad Operational Language Translation (BOLT) was an interdisciplinary DARPA-sponosored program with an MT component. It aimed at enabling English-speaking persons to communicate with non-English-speaking populations and identify important information in foreign-language sources -- specifically by: 1) allowing English-speakers to understand foreign-language sources of all genres, including chat, messaging and informal conversation; 2) providing English-speakers the ability to quickly identify targeted information in foreign-language sources using natural-language queries; and 3) enabling multi-turn communication in text and speech with non-English speakers. There was some emphasis on delivering these capabilities free from domain or genre limitations. NIST organized and implemented BOLT's evaluations of speech-to-text and text-to-text MT technology as well as the evaluation of end-to-end MT systems enabling live speech communication between two speakers of different languages.
• GALE, 2006-2011: Global Autonomous Language Exploitation (GALE) was an interdisciplinary five-year DARPA program that included both an MT component and a Distillation (information extraction) component. NIST organized and implemented GALE's yearly evaluations for speech-to-text and text-to-text MT technology.
• MADCAT, 2008-2013: Multilingual Automatic Document Classification Analysis and Translation (MADCAT) was a DARPA program to evaluate technologies that translate Arabic document images into English text. NIST evaluated the overall system as well as the major components (OCR and MT) of the system.
• OpenHaRT, 2011-2013: The NIST Handwriting Recognition and Translation evaluation focused on evaluating technologies that contribute to document understanding, with emphasis on core tasks such as recognition and translation.
• TRANSTAC, 2006-2010: Spoken Language Communication and Translation System for Tactical Use (TRANSTAC) was a DARPA program to develop and field speech-to-speech MT technology, enabling two-way spoken communication between U.S. Soldiers and Marines (speaking only English) and civilian populations who speak only other languages. NIST evaluated the performance of the TRANSTAC systems, including the systems as a whole, as well as their speech recognition, machine translation, and text-to-speech components.

Past MT metrology evaluation activities:

• MetricsMaTr: An evaluation of MT metrology. Focus was placed on the improvement of automated measurement techniques of MT technology, specifically towards providing insight into the quality of a translation.
• MFLTS: This activity was sponsored by the US Army Machine Foreign Language Translation System program. NIST chaired the Metrics-IPT, whose task was to develop a new metric that is grounded to the Interagency Language Roundtable (ILR) rating system Metrics-IPT working group.