Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

THE MULTI-RELATIONSHIP EVALUATION DESIGN FRAMEWORK: DESIGNING TESTING PLANS TO COMPREHENSIVELY ASSESS ADVANCED AND INTELLIGENT TECHNOLOGIES

Published

Author(s)

Brian A. Weiss, Linda C. Schmidt, Harry A. Scott, Craig I. Schlenoff

Abstract

As new technologies are developed and mature, it becomes extremely important to provide both formative and summative assessments on their performance. Performance assessment events range in form from a few simple tests of key elements of the technology to highly complex and extensive evaluation exercises targeting specific levels and capabilities of the system under scrutiny. Typically the more advanced the system, the more often performance evaluations are warranted and the more complex the evaluation planning. Numerous evaluation frameworks have been developed to generate evaluation designs intent on characterizing the performance of intelligent systems. Many of these frameworks enable the design of extensive evaluations, but each has its own focused objectives presenting a range of boundaries. This paper introduces the Multi-Relationship Evaluation Design (MRED) framework whose ultimate goal is to automatically generate an evaluation design based upon multiple inputs. The MRED framework takes input goal data and outputs an evaluation blueprint complete with specific evaluation elements including level of technology to be tested, metric type, user type, and, evaluation environment. Some of MRED's unique features are that it characterizes these relationships and manages these uncertainties along with those associated with evaluation input. The authors will introduce MRED by first presenting relationships between four main evaluation design elements.. This will be further supported through the definition of key terms. An example will be presented in which these terms and relationships are applied to the evaluation design of an automobile technology. An initial validation step follows where MRED is applied to the speech translation technology whose evaluation design was inspired by the successful use of a pre-existing evaluation framework.
Proceedings Title
ASME 2010 International Design Engineering Technical Conferences (IDETC) 22nd International Conference on Design Theory and Methodology (DTM)
Conference Dates
August 15-18, 2010
Conference Location
Montreal, Quebec

Keywords

MRED, Performance Metrics, Evaluation Framework, Uncertainty

Citation

Weiss, B. , Schmidt, L. , Scott, H. and Schlenoff, C. (2010), THE MULTI-RELATIONSHIP EVALUATION DESIGN FRAMEWORK: DESIGNING TESTING PLANS TO COMPREHENSIVELY ASSESS ADVANCED AND INTELLIGENT TECHNOLOGIES, ASME 2010 International Design Engineering Technical Conferences (IDETC) 22nd International Conference on Design Theory and Methodology (DTM), Montreal, Quebec, -1, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=905063 (Accessed March 28, 2024)
Created May 5, 2010, Updated February 19, 2017