Skip to main content
U.S. flag

An official website of the United States government

Dot gov

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Https

Secure .gov websites use HTTPS
A lock ( ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

The NIST 2008 Metrics for Machine Translation Challenge - Overview, Methodology, Metrics, and Results

Published

Author(s)

Mark A. Przybocki, Kay Peterson, P. S. Bronsart, Gregory A. Sanders

Abstract

This paper discusses the evaluation of automated metrics developed for the purpose of evaluating machine translation (MT) technology. A general discussion of the usefulness of automated metrics is offered. The NIST MetricsMATR evaluation of MT metrology is described, including its objectives, protocols, participants, and test data. The methodology employed to evaluate the submitted metrics is reviewed. The general classes of metrics that were evaluated are summarized. Overall results of this evaluation are presented, primarily by means of correlation statistics, showing the degree of agreement between the automated metric scores and the scores of human judgments. Metrics are analyzed at the sentence, document, and system level with results conditioned by various properties of the test data. This paper concludes with some perspective on the improvements that should be incorporated into future evaluations of metrics for MT evaluation.
Citation
Machine Translation

Keywords

MT metrics, evaluation, automated metrics, machine translation, MT, MetricsMATR
Created March 10, 2010, Updated February 19, 2017