Evolution of the SCORE Framework to Enhance Field-Based Performance Evaluations of Emerging Technologies

Published

August 21, 2008

Author(s)

Brian A. Weiss, Craig I. Schlenoff

Abstract

Designing and implementing a performance evaluation of an emerging technology to present a broad picture of how this technology would perform in its typical operating environment is a very challenging goal. Personnel from the National Institute of Standards and Technology (NIST) have developed the System, Component, and Operationally-Relevant Evaluation (SCORE) framework as formal guide for designing technology evaluations to capture both technical performance and end-user utility assessments of systems and their components within controlled and realistic environments to present an extensive (but not necessarily exhaustive) picture of how a system would behave in a set of realistic use-case domains. The framework has been applied to numerous evaluation efforts over the past three years producing valuable quantitative and qualitative metrics. This paper will present the building blocks of the SCORE methodology including the system goals and design criteria that drive the evaluation design process. An evolution of the SCORE framework in capturing utility assessments at the capability level of a system will also be presented. Examples will be shown of SCORE’s successful application to the evaluation of the soldier-worn sensor systems and two-way, free-form spoken language translation technologies.

Proceedings Title

Proceedings of the 2008 Performance Metrics for Intelligent Systems (PerMIS) Workshop

Conference Dates

August 19-21, 2008

Conference Location

Gaithersburg, MD

Pub Type

Conferences

Download Paper

Local Download

Keywords

SCORE, DARPA, ASSIST, TRANSTAC, performance evaluation, elemental tests, vignette tests, task tests, speech translation, soldier-worn sensor

Citation

Weiss, B. and Schlenoff, C. (2008), Evolution of the SCORE Framework to Enhance Field-Based Performance Evaluations of Emerging Technologies, Proceedings of the 2008 Performance Metrics for Intelligent Systems (PerMIS) Workshop, Gaithersburg, MD, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=824720 (Accessed December 6, 2024)