Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Evolution of the SCORE Framework to Enhance Field-Based Performance Evaluations of Emerging Technologies

Published

Author(s)

Brian A. Weiss, Craig I. Schlenoff

Abstract

Designing and implementing a performance evaluation of an emerging technology to present a broad picture of how this technology would perform in its typical operating environment is a very challenging goal. Personnel from the National Institute of Standards and Technology (NIST) have developed the System, Component, and Operationally-Relevant Evaluation (SCORE) framework as formal guide for designing technology evaluations to capture both technical performance and end-user utility assessments of systems and their components within controlled and realistic environments to present an extensive (but not necessarily exhaustive) picture of how a system would behave in a set of realistic use-case domains. The framework has been applied to numerous evaluation efforts over the past three years producing valuable quantitative and qualitative metrics. This paper will present the building blocks of the SCORE methodology including the system goals and design criteria that drive the evaluation design process. An evolution of the SCORE framework in capturing utility assessments at the capability level of a system will also be presented. Examples will be shown of SCORE’s successful application to the evaluation of the soldier-worn sensor systems and two-way, free-form spoken language translation technologies.
Proceedings Title
Proceedings of the 2008 Performance Metrics for Intelligent Systems (PerMIS) Workshop
Conference Dates
August 19-21, 2008
Conference Location
Gaithersburg, MD

Keywords

SCORE, DARPA, ASSIST, TRANSTAC, performance evaluation, elemental tests, vignette tests, task tests, speech translation, soldier-worn sensor

Citation

Weiss, B. and Schlenoff, C. (2008), Evolution of the SCORE Framework to Enhance Field-Based Performance Evaluations of Emerging Technologies, Proceedings of the 2008 Performance Metrics for Intelligent Systems (PerMIS) Workshop, Gaithersburg, MD, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=824720 (Accessed March 29, 2024)
Created August 21, 2008, Updated February 17, 2017