The Multi-Relationship Evaluation Design Framework: Creating Evaluation Blueprints to Assess Advanced and Intelligent Technologies

Published

October 29, 2010

Author(s)

Brian A. Weiss, Linda C. Schmidt

Abstract

Technological evolutions are constantly occurring across advanced and intelligent systems within a range of fields including those within the military, law enforcement, automobile, and manufacturing industries. Testing the performance of these technologies is critical to update the system designers of areas for improvement and validate the technology's ultimate capabilities. Evaluations can be minimally designed to include a few basic tests of key technology capabilities or they can evolve into extensive test events that emphasize multiple components and capabilities along with the complete system, itself. Tests of advanced and intelligent systems typically assume the latter and can occur frequently based upon system complexity. Numerous evaluation design frameworks have been produced to create test designs to appropriately assess the performance of intelligent systems. While most of these frameworks allow broad evaluation plans to be created, each framework has been focused to address specific project and/or technological needs and therefore has bounded applicability. This paper presents and expands upon the current development of the Multi-Relationship Evaluation Design (MRED) framework. Development of MRED is motivated by the desire to automatically create an evaluation framework capable of producing detailed evaluation blueprints while receiving uncertain input information. The authors will build upon their previous work in developing MRED through an initial discussion of key evaluation design elements. Additionally, the authors will elaborate upon their previously-defined relationships among evaluation personnel to define evaluation structural components pertaining to the evaluation scenarios, test environment, and data collection methods. These terms and their relationships will be demonstrated in an example evaluation design of an emerging technology.

Proceedings Title

Proceedings of the 2010 Performance Metrics for Intelligent Systems (PerMIS) Workshop

Conference Dates

September 28-30, 2010

Conference Location

Baltimore, MD

Pub Type

Conferences

Download Paper

Local Download

Keywords

MRED, SCORE, Performance Metrics, Evaluation Framework, Uncertainty

Citation

Weiss, B. and Schmidt, L. (2010), The Multi-Relationship Evaluation Design Framework: Creating Evaluation Blueprints to Assess Advanced and Intelligent Technologies, Proceedings of the 2010 Performance Metrics for Intelligent Systems (PerMIS) Workshop, Baltimore, MD, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=906659 (Accessed April 23, 2024)