OpenMT12 Evaluation Results

Officlal Release of Results

Date of release: August 28, 2012
Version: mt12_official_v1

The NIST 2012 Open Machine Translation Evaluation (OpenMT12) was part of an ongoing series of evaluations of human language translation technology. NIST conducts these evaluations in order to support machine translation (MT) research and help advance the state-of-the-art in machine translation technology. These evaluations provide an important contribution to the direction of research efforts and the calibration of technical capabilities. The evaluation was administered as outlined in the official OpenMT12 evaluation plan.

Disclaimer

These results are not to be construed, or represented as endorsements of any participant's system or commercial product, or as official findings on the part of NIST or the U.S. Government. Note that the results submitted by developers of commercial MT products were generally from research systems, not commercially available products. Since OpenMT12 was an evaluation of research algorithms, the OpenMT12 test design required local implementation by each participant. As such, participants were only required to submit their translation system output to NIST for uniform scoring and analysis. The systems themselves were not independently evaluated by NIST.

Certain commercial equipment, instruments, software, or materials are identified in this paper in order to specify the experimental procedure adequately. Such identification is not intended to imply recommendation or endorsement by NIST, nor is it intended to imply that the equipment, instruments, software or materials are necessarily the best available for the purpose. There is ongoing discussion within the MT research community regarding the most informative metrics for machine translation. The design and implementation of these metrics are themselves very much part of the research. At the present time, there is no single metric that has been deemed to be completely indicative of all aspects of system performance.

The data, protocols, and metrics employed in this evaluation were chosen to support MT research and should not be construed as indicating how well these systems would perform in applications. While changes in the data domain, or changes in the amount of data used to build a system, can greatly influence system performance, changing the task protocols could indicate different performance strengths and weaknesses for these same systems.

Because of the above reasons, this should not be interpreted as a product testing exercise and the results should not be used to make conclusions regarding which commercial products are best for a particular application.

History

• 2012/08/28: v1 of official release.

Evaluation Tasks

OpenMT12 was a test of text-to-text MT technology. The evaluation consisted of tests for five language pairs:

• Arabic-to-English
• Chinese-to-English
• Dari-to-English
• Farsi-to-English
• Korean-to-English

Evaluation Conditions

MT research and development requires language data resources. System performance is strongly affected by the type and amount of resources used. Therefore, two different resource categories were defined as conditions of evaluation. The categories differ solely by the amount of data that was available for use in the training and development of the core MT engine. These evaluation conditions were called "Constrained Training" and "Unconstrained Training". See the evaluation specification document for a complete description of allowable resources for each.

Evaluation Tracks

In recent years, performance improvements have been demonstrated through the use of system combination techniques. For OpenMT12, two evaluation tracks were supported that were called "Single System Track" and "System Combination Track". Results are reported separately for each track. As the names of each track implies, a key feature of systems entered in the Single System Track is that the resulting translations are produced by primarily one algorithmic approach, while translations from the System Combination Track result from a combination technique where two or more core algorithmic approaches are used.

Evaluation Data

The evaluation data of OpenMT12 consisted of several subsets of data. All available subsets per language pair were processed by participants for that language pair. All source data and reference translations were made available for inspection and experimentation by participants after the evaluation. This means that the data sets labeled as Progress tests here will not serve as Progress tests again in the future, as they will have been exposed.

• Original Progress test: The Progress test sets used in OpenMT08 and OpenMT09, unchanged, with data to be released after OpenMT12. Data genres: Newswire and Web.
• Expanded Progress test: The Progress test set used in OpenMT08 and OpenMT09, expanded with additional data from the same data genres.
• Original Progress test new source subset: A combined subset of the OpenMT08 and OpenMT09 Progress tests, with new source data generated by human translation of the English reference translation. The source data was provided in two styles:
  • English-true: A more English-oriented translation; requires that the text reads well and does not use any idiomatic expressions in the foreign language to convey meaning, unless absolutely necessary.
  • Foreign-true: A translation as close as possible to the foreign language, as if the text had originated in that language.
• New domain: Data from a variety of genres, varying in formality and structured, with a common theme that will not be made public before evaluation.

The following table lists which data subsets were part of the evaluation set and the approximate reference word count comprising all subsets for each language pair:

Performance Measurement 
 

• BLEU-4 (mteval-v13a, the official OpenMT12 evaluation metric)
  • Invocation line: perl mteval-v13a.pl -r REFERENCE_FILE -s SOURCE_FILE -t CANDIDATE_FILE -c -b
  • Option -c : case-sensitive scoring
  • Option -b : BLEU score only
• IBM BLEU (bleu-1.04a)
  • Invocation line: perl bleu-1.04.pl -r REFERENCE_FILE -t CANDIDATE_FILE
  • By default, scoring is case-sensitive
• NIST (mteval-v13a)
  • Invocation line: perl mteval-v13a.pl -r REFERENCE_FILE -s SOURCE_FILE -t CANDIDATE_FILE -c -n
  • Option -c : case-sensitive scoring
  • Option -n : NIST score only
• TER (tercom-0.7.25)
  • Invocation line: java -jar tercom.7.25.jar -r REFERENCE_FILE -h CANDIDATE_FILE -N -s
  • Option -N : enables normalization
  • Option -s : case-sensitive scoring
• METEOR (meteor-0.7)
  • Invocation line: perl meteor.pl -s SYSTEM_ID -r REFERENCE_FILE -t CANDIDATE_FILE --modules "exact porter_stem wn_stem wn_synonymy"
  • Option --modules "exact porter_stem wn_stem wn_synonymy" : uses all four METEOR matching modules, in that order

Participants

The following table lists the organizations participating in OpenMT12 and the test sets they registered to process.

Results Section

[  Arabic-To-English ]   [  Chinese-To-English ]   [  Dari-To-English ]   [  Farsi-To-English ]   [  Korean-To-English ]   [  Progress Test History ]  

Arabic-To-English Results

Scores reported are limited to primary, on-time, non-debugged submissions.

Scores are ordered alphabetically by siteID.

Arabic-To-English Participants

The following table lists the organizations participating in OpenMT12 and the test sets they registered to process.

1a. Arabic-To-English Progress Test New Source Subset - "English-True" source 
(143 documents containing 1537 segments)

Arabic-To-English / Constrained

1b. Arabic-To-English Progress Test New Source Subset - "Foreign-True" source 
(143 documents containing 1537 segments)

Arabic-To-English / Constrained

2. Arabic-To-English Original Progress Test 
(135 documents containing 1378 segments)

Arabic-To-English / Constrained

⁽¹⁾A late debugged submission was also made; scores for that submission are not reported here.

[  Top ]   [  Arabic-To-English ]   [  Chinese-To-English ]   [  Dari-To-English ]   [  Farsi-To-English ]   [  Korean-To-English ]   [  Progress Test History ]  

Chinese-To-English Results

Scores reported are limited to primary, on-time, non-debugged submissions.

Scores are ordered alphabetically by siteID.

Chinese-To-English Participants

The following table lists the organizations participating in OpenMT12 and the test sets they registered to process.

1a. Chinese-To-English Progress Test New Source Subset - "English-True" source 
(143 documents containing 1537 segments)

Chinese-To-English / Constrained

Chinese-To-English / UnConstrained

Chinese-To-English / Original System Combination / Constrained

1b. Chinese-To-English Progress Test New Source Subset - "Foreign-True" source 
(143 documents containing 1537 segments)

Chinese-To-English / Constrained

Chinese-To-English / UnConstrained

Chinese-To-English / Original System Combination / Constrained

2. Chinese-To-English Original Progress Test 
(122 documents containing 1370 segments)

Chinese-To-English / Constrained

Chinese-To-English / UnConstrained

Chinese-To-English / Original System Combination / Constrained

3. Chinese-to-English OpenMT12 Current Test 
(73 documents containing 820 segments)

Chinese-To-English / Constrained

Chinese-To-English / UnConstrained

Chinese-To-English / Original System Combination / Constrained

4. Chinese-to-English Original Progress Test + OpenMT12 Current Test 
(195 documents containing 2190 segments)

Chinese-To-English / Constrained

Chinese-To-English / UnConstrained

Chinese-To-English / Original System Combination / Constrained

5. Chinese-to-English OpenMT12 Novel Domain Test 
(149 documents containing 2184 segments)

Chinese-To-English / Constrained

Chinese-To-English / UnConstrained

Chinese-To-English / Original System Combination / Constrained

⁽²⁾A late debugged submission was also made; scores for that submission are not reported here.

⁽³⁾An on-time debugged submission was also made; scores for that submission are not reported here.

⁽⁴⁾An on-time submission was made but withdrawn; scores for that submission are not reported here.

[  Top ]   [  Arabic-To-English ]   [  Chinese-To-English ]   [  Dari-To-English ]   [  Farsi-To-English ]   [  Korean-To-English ]   [  Progress Test History ]  

Dari-To-English Results

Scores reported are limited to primary, on-time, non-debugged submissions.

Scores are ordered alphabetically by siteID.

Dari-To-English Participants

The following table lists the organizations participating in OpenMT12 and the test sets they registered to process.

1a. Dari-To-English Progress Test New Source Subset - "English-True" source 
(143 documents containing 1537 segments)

Dari-To-English / UnConstrained

1b. Dari-To-English Progress Test New Source Subset - "Foreign-True" source 
(143 documents containing 1537 segments)

Dari-To-English / UnConstrained

[  Top ]   [  Arabic-To-English ]   [  Chinese-To-English ]   [  Dari-To-English ]   [  Farsi-To-English ]   [  Korean-To-English ]   [  Progress Test History ]  

Farsi-To-English Results

Scores reported are limited to primary, on-time, non-debugged submissions.

Scores are ordered alphabetically by siteID.

Farsi-To-English Participants

The following table lists the organizations participating in OpenMT12 and the test sets they registered to process.

1a. Farsi-To-English Progress Test New Source Subset - "English-True" source 
(143 documents containing 1537 segments)

Farsi-To-English / UnConstrained

1b. Farsi-To-English Progress Test New Source Subset - "Foreign-True" source 
(143 documents containing 1537 segments)

Farsi-To-English / UnConstrained

[  Top ]   [  Arabic-To-English ]   [  Chinese-To-English ]   [  Dari-To-English ]   [  Farsi-To-English ]   [  Korean-To-English ]   [  Progress Test History ]  

Korean-To-English Results

Scores reported are limited to primary, on-time, non-debugged submissions.

Scores are ordered alphabetically by siteID.

Korean-To-English Participants

The following table lists the organizations participating in OpenMT12 and the test sets they registered to process.

1a. Korean-To-English Progress Test New Source Subset - "English-True" source 
(143 documents containing 1537 segments)

Korean-To-English / UnConstrained

1b. Korean-To-English Progress Test New Source Subset - "Foreign-True" source 
(143 documents containing 1537 segments)

Korean-To-English / UnConstrained

[  Top ]   [  Arabic-To-English ]   [  Chinese-To-English ]   [  Dari-To-English ]   [  Farsi-To-English ]   [  Korean-To-English ]   [  Progress Test History ]  

Progress Test History

This page of results is limited to those of the Progress Test data for the OpenMT12 tests of Arabic-to-English and Chinese-to-English text-to-text translation.

The Progress test was designed as a means to demonstrate true system improvement for a particular site over time, irrespective of inherent data differences that come with each new test set. In order to keep the focus on progress over time, only results from sites that participated in OpenMT12 and at least one prior evaluation are reported.

There may be issues related to processing older test sets that make across-site comparisons less meaningful.

Scores reported are limited to primary, on-time, non-debugged submissions.

Scores are ordered by BLEU-4 score on the Overall test set.

Note that all BLEU-4 scores reported on this page were computed using mteval-v13a. (The original OpenMT08 scores were computed using mteval-v11b.)

Participants

The following table lists the organizations participating in OpenMT12 and the test sets they registered to process.

1. Arabic-To-English Original Progress Test History 


 

2. Chinese-To-English Original Progress Test History 


 

⁽⁵⁾The MT08 submission was a system combination.

⁽⁶⁾A late debugged submission was made; scores for that submission are not reported here.

[  Top ]   [  Arabic-To-English ]   [  Chinese-To-English ]   [  Dari-To-English ]   [  Farsi-To-English ]   [  Korean-To-English ]   [  Progress Test History ]  

[  NIST OpenMT12 Home  ]