NIST 2006 Machine Translation Evaluation Official Results

Date of Updated Release: Tuesday, November 1, 2006, version 4

The NIST 2006 Machine Translation Evaluation (MT-06) was part of an ongoing series of evaluations of human language translation technology. NIST conducts these evaluations in order to support machine translation (MT) research and help advance the state-of-the-art in machine translation technology. These evaluations provide an important contribution to the direction of research efforts and the calibration of technical capabilities. The evaluation was administered as outlined in the official MT-06 evaluation plan.

Disclaimer

These results are not to be construed, or represented as endorsements of any participant's system or commercial product, or as official findings on the part of NIST or the U.S. Government. Note that the results submitted by developers of commercial MT products were generally from research systems, not commercially available products. Since MT-06 was an evaluation of research algorithms, the MT-06 test design required local implementation by each participant. As such, participants were only required to submit their translation system output to NIST for uniform scoring and analysis. The systems themselves were not independently evaluated by NIST.

There is ongoing discussion within the MT research community regarding the most informative metrics for machine translation. The design and implementation of these metrics are themselves very much part of the research. At the present time, there is no single metric that has been deemed to be completely indicative of all aspects of system performance.

The data, protocols, and metrics employed in this evaluation were chosen to support MT research and should not be construed as indicating how well these systems would perform in applications. While changes in the data domain, or changes in the amount of data used to build a system, can greatly influence system performance, changing the task protocols could indicate different performance strengths and weaknesses for these same systems.

Because of the above reasons, this should not be interpreted as a product testing exercise and the results should not be used to make conclusions regarding which commercial products are best for a particular application.

Evaluation Tasks

The MT-06 evaluation consisted of two tasks. Each task required a system to perform translation from a given source language into the target language. The source languages were Arabic and Chinese, and the target language was English.

• Translate Arabic text into English text
• Translate Chinese text into English text

Evaluation Conditions

MT research and development requires language data resources. System performance is strongly affected by the type and amount of resources used. Therefore, two different resource categories were defined as conditions of evaluation. The categories differed solely by the amount of data that was available for use in system training and development. The evaluation conditions were called "Large Data Track" and "Unlimited Data Track".

• Large Data Track – limited the training data to data in the LDC public catalogue existing before February 1st, 2006.
• Unlimited Data Track – extended the training data to any publicly available data existing before February 1st, 2006.
• Unlimited Plus Data Track – further extended the training data to include non-publicly available data existing before February 1st, 2006. [see end of page]

Other submissions not in categories described above are not reported here.

Evaluation Data

Source Data

In an effort to reduce data creation costs, the MT-06 evaluation made use of GALE-06 evaluation data (GALE subset). NIST augmented the GALE subset with additional data of equal or greater size for most of the genres (NIST subset). This provided a larger and more diverse test set. Each set contained documents drawn from newswire text documents, web-based newsgroup documents, human transcription of broadcast news, and human transcription of broadcast conversations. The source documents were encoded in UTF-8.

The test data was selected from a pool of data collected by the LDC during February 2006. The careful selection process sought to have a variety of sources (see below), publication dates, and difficulty ratings while hitting the target test set size.

Reference Data

The GALE subset had one adjudicated high quality translation that was produced by the National Virtual Translation Center. The NIST subset had four independently generated high quality translations that were produced by professional translation companies. In both subsets, each translation agency was required to have native speaker(s) of the source and target languages, working on the translations.

Performance Measurement

Machine translation quality was measured automatically using an N-gram co-occurrence statistic metric developed by IBM and referred to as BLEU. BLEU measures translation accuracy according to the N-grams or sequence of N-words that it shares with one or more high quality reference translations. Thus, the more co-occurrences the better the score. BLEU is an accuracy metric, ranging from "0" to "1" with "1" being the best possible score. A detailed description of BLEU can be found in the paper Papineni, Roukos, Ward, Zhu (2001). "Bleu: a Method for Automatic Evaluation of Machine Translation" (keyword = RC22176).

Although BLEU was the official metric for MT-06, measuring translation quality is an ongoing research topic in the MT community. At the present time, there is no single metric that has been deemed to be completely indicative of all aspects of system performance. Three additional automatic metrics METEOR, TER, and BLEU-refinement as well as human assessment were used to report the system performance. As stated in the evaluation specification document, this official public version of the results will report only the scores as measured by BLEU.

Evaluation Participants

The table below lists the organizations involved in submitting MT-06 evaluation results. Most submitted results representing their own organizations, some participated only in a collaborative effort (marked by the @ symbol), and some did both (marked by the + symbol).

• DFKI GmbH registered but dropped out of the evaluation on July 28, 2006.
• Fitchburg State College registered but dropped out of the evaluation on August 3, 2006.

Evaluation Systems

Each site/team could submit one or more systems for evaluation with one system marked as its primary system. The primary system indicated the site/team's best effort. This official public version of the results report the results only for the primary systems.

Evaluation Results

The tables below list the results of the NIST 2006 Machine Translation Evaluation. The results are sorted by the BLEU scores and reported separately for the GALE subset and the NIST subset because they do not have the same number of reference translations. The results are also reported for each data domain. Note that these scores reflect case-errors.

Friedman's Rank Test for k Correlated Samples was used to test for significant difference among the systems. The initial null hypothesis was that all systems were the same. If the null hypothesis was rejected at the 95% level of confidence, the lowest scoring system was taken out of the pool of systems to be tested, and the Friedman's Rank Test was repeated for the remaining systems until no significant difference was found. The remaining systems that were not removed from the pool were deemed to be statistically equivalent. The process was repeated for the systems taken out of the pool. Alternating colors (white and yellow backgrounds) show the different groups.

Key:

• An asterisk (*) indicates the submission was submitted late.
• A pound sign (#) indicates the submission was a bug-fix. Bug-fix means that error(s) was found in the system during the testing period and sites fixed the error(s) and reran the test. Format errors do not count as bug-fixes.

Note: Site 'nlmp' was unable to process the entire test set. No result is listed for that site.

Arabic-to-English Results

Large Data Track

NIST Subset

Overall BLEU Scores

Newswire BLEU Scores

Newsgroup BLEU Scores

Broadcast News BLEU Scores

GALE Subset

Overall BLEU Scores

Newswire BLEU Scores

Newsgroup BLEU Scores

Broadcast News BLEU Scores

Broadcast Conversation BLEU Scores

Unlimited Data Track

NIST Subset

Overall BLEU Scores

Newswire BLEU Scores

Newsgroup BLEU Scores

Broadcast News BLEU Scores

GALE Subset

Overall BLEU Scores

Newswire BLEU Scores

Newsgroup BLEU Scores

Broadcast News BLEU Scores

Broadcast Conversation BLEU Scores

Chinese-to-English Results

Large Data Track

NIST Subset

Overall BLEU Scores

Newswire BLEU Scores

Newsgroup BLEU Scores

Broadcast News BLEU Scores

GALE Subset

Overall BLEU Scores

Newswire BLEU Scores

Newsgroup BLEU Scores

Broadcast News BLEU Scores

Broadcast Conversation BLEU Scores

Unlimited Data Track

NIST Subset

Overall BLEU Scores

Newswire BLEU Scores

Newsgroup BLEU Scores

Broadcast News BLEU Scores

GALE Subset

Overall BLEU Scores

Newswire BLEU Scores

Newsgroup BLEU Scores

Broadcast News BLEU Scores

Broadcast Conversation BLEU Scores

Unlimited Plus Data track

Release History

• Version 1: Initial release of preliminary results to evaluation participants
• Version 2: Added late and bug fixed submissions, added METEOR, TER, and BLEU-refinement scores.
• Version 3: Public version of the results (included only the BLEU scores for primary systems)
• Version 4: Public version of the results, updated disclaimer