Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Evaluating Speech Translation Systems: Applying SCORE to TRANSTAC Technologies



Craig I. Schlenoff, Brian A. Weiss, Michelle P. Steves, Gregory A. Sanders, Frederick M. Proctor, Ann M. Virts


The Spoken Language Communication and Translation System for Tactical Use (TRANSTAC) program is a Defense Advanced Research Projects Agency (DARPA) advanced technology research and development program. The goal of the TRANSTAC program is to demonstrate capabilities to rapidly develop and field free-form, two-way translation systems that enable speakers of different languages to communicate with one another in real-world tactical situations without an interpreter. The National Institute of Standards and Technology (NIST), along with support from MITRE and Appen Pty Ltd., have been funded to serve as the Independent Evaluation Team (IET) for the TRANSTAC Program. As the IET, NIST is responsible for analyzing the performance of the TRANSTAC systems by designing and executing multiple TRANSTAC evaluations and analyzing the results of the evaluation. To accomplish this, NIST has applied the SCORE (System, Component, and Operationally Relevant Evaluations) Framework. SCORE is a unified set of criteria and software tools for defining a performance evaluation approach for complex intelligent systems. It provides a comprehensive evaluation blueprint that assesses the technical performance of a system and its components through isolating and changing variables as well as capturing end-user utility of the system in realistic use-case environments. This document describes the TRANSTAC program and explains how the SCORE framework was applied to assess the technical and utility performance of the TRANSTAC systems.
Proceedings Title
Proceedings of the Performance Metrics for Intelligent Systems (PerMIS) 2009
Conference Dates
September 21-23, 2009
Conference Location
Gaithersburg, MD


SCORE, TRANSTAC, speech translation, performance evaluation


Schlenoff, C. , Weiss, B. , Steves, M. , Sanders, G. , Proctor, F. and Virts, A. (2009), Evaluating Speech Translation Systems: Applying SCORE to TRANSTAC Technologies, Proceedings of the Performance Metrics for Intelligent Systems (PerMIS) 2009, Gaithersburg, MD, [online], (Accessed April 20, 2024)
Created September 29, 2009, Updated February 19, 2017