OpenMT12 Evaluation

The NIST Open Machine Translation 2012 Evaluation (OpenMT12) was implemented according to the OpenMT12 evaluation plan. It took place in the spring of 2012, followed by a summer of 2012 workshop.

Highlights

• Evaluation on prior years' Arabic-to-English and Chinese-to-English Progress test data, with the data to be made available to participants after OpenMT12
• Evaluation on a parallel data set based on prior years' Progress tests for Arabic, Chinese, Dari, Farsi, and Korean to English, in two source data styles
• Evaluation on new domain data for Chinese-to-English.
• Support for both a single system and a system combination track (provided sufficient interest by the participants)
• Evaluation by automatic metrics and coordination of volunteer human assessments using a new tool to rank alternative translations

Documentation

• Evaluation plan (v1.2, updated July 9 2012)
• Registration form
• Human assessment registration form
• LDC data license agreement
• System description template 
• Official release of results

Schedule

• (See Table 1 of eval plan): Training data off-limits periods
• October 28 2011: Evaluation plan available
• October 28 2011 – February 3 2012: Registration period (early registration highly encouraged)
• November 4 2011: Training data available from LDC
• January 16 – February 24 2012: Dry run period (early submission highly encouraged); output due at NIST February 24 11.59am EST
• April 2 – 6 2012: Main evaluation period; output due at NIST April 6 11.59am EDT
• April 16 – 20 2012: System combination evaluation period; output due at NIST April 20 11.59am EDT
• April 20 2012: Preliminary release of main evaluation results to participants
• April 27 2012: System descriptions due
• April 30 – June 1 2012: Human assessment period
• June 27 – 28 2012: Workshop in the Washington DC area
• August 31 2012: Official public release of results

Software and Resources

• Current mteval scoring utility: mteval-v13a.pl
• Current OpenMT XML DTD: mteval-xml-v1.6.dtd
• OpenMT12 submission checker: OpenMT12.tar.gz

Contact

mt_poc [at] nist.gov (mt_poc[at]nist[dot]gov)