THE MULTI-RELATIONSHIP EVALUATION DESIGN FRAMEWORK: DESIGNING TESTING PLANS TO COMPREHENSIVELY ASSESS ADVANCED AND INTELLIGENT TECHNOLOGIES

Published

May 5, 2010

Author(s)

Brian A. Weiss, Linda C. Schmidt, Harry A. Scott, Craig I. Schlenoff

Abstract

As new technologies are developed and mature, it becomes extremely important to provide both formative and summative assessments on their performance. Performance assessment events range in form from a few simple tests of key elements of the technology to highly complex and extensive evaluation exercises targeting specific levels and capabilities of the system under scrutiny. Typically the more advanced the system, the more often performance evaluations are warranted and the more complex the evaluation planning. Numerous evaluation frameworks have been developed to generate evaluation designs intent on characterizing the performance of intelligent systems. Many of these frameworks enable the design of extensive evaluations, but each has its own focused objectives presenting a range of boundaries. This paper introduces the Multi-Relationship Evaluation Design (MRED) framework whose ultimate goal is to automatically generate an evaluation design based upon multiple inputs. The MRED framework takes input goal data and outputs an evaluation blueprint complete with specific evaluation elements including level of technology to be tested, metric type, user type, and, evaluation environment. Some of MRED's unique features are that it characterizes these relationships and manages these uncertainties along with those associated with evaluation input. The authors will introduce MRED by first presenting relationships between four main evaluation design elements.. This will be further supported through the definition of key terms. An example will be presented in which these terms and relationships are applied to the evaluation design of an automobile technology. An initial validation step follows where MRED is applied to the speech translation technology whose evaluation design was inspired by the successful use of a pre-existing evaluation framework.

Proceedings Title

ASME 2010 International Design Engineering Technical Conferences (IDETC) 22nd International Conference on Design Theory and Methodology (DTM)

Conference Dates

August 15-18, 2010

Conference Location

Montreal, Quebec

Pub Type

Conferences

Download Paper

Local Download

Keywords

MRED, Performance Metrics, Evaluation Framework, Uncertainty

Citation

Weiss, B. , Schmidt, L. , Scott, H. and Schlenoff, C. (2010), THE MULTI-RELATIONSHIP EVALUATION DESIGN FRAMEWORK: DESIGNING TESTING PLANS TO COMPREHENSIVELY ASSESS ADVANCED AND INTELLIGENT TECHNOLOGIES, ASME 2010 International Design Engineering Technical Conferences (IDETC) 22nd International Conference on Design Theory and Methodology (DTM), Montreal, Quebec, -1, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=905063 (Accessed December 3, 2024)