Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

The Multi-Relationship Evaluation Design Framework: Creating Evaluation Blueprints to Assess Advanced and Intelligent Technologies

Published

Author(s)

Brian A. Weiss, Linda C. Schmidt

Abstract

Technological evolutions are constantly occurring across advanced and intelligent systems within a range of fields including those within the military, law enforcement, automobile, and manufacturing industries. Testing the performance of these technologies is critical to update the system designers of areas for improvement and validate the technology's ultimate capabilities. Evaluations can be minimally designed to include a few basic tests of key technology capabilities or they can evolve into extensive test events that emphasize multiple components and capabilities along with the complete system, itself. Tests of advanced and intelligent systems typically assume the latter and can occur frequently based upon system complexity. Numerous evaluation design frameworks have been produced to create test designs to appropriately assess the performance of intelligent systems. While most of these frameworks allow broad evaluation plans to be created, each framework has been focused to address specific project and/or technological needs and therefore has bounded applicability. This paper presents and expands upon the current development of the Multi-Relationship Evaluation Design (MRED) framework. Development of MRED is motivated by the desire to automatically create an evaluation framework capable of producing detailed evaluation blueprints while receiving uncertain input information. The authors will build upon their previous work in developing MRED through an initial discussion of key evaluation design elements. Additionally, the authors will elaborate upon their previously-defined relationships among evaluation personnel to define evaluation structural components pertaining to the evaluation scenarios, test environment, and data collection methods. These terms and their relationships will be demonstrated in an example evaluation design of an emerging technology.
Proceedings Title
Proceedings of the 2010 Performance Metrics for Intelligent Systems (PerMIS) Workshop
Conference Dates
September 28-30, 2010
Conference Location
Baltimore, MD

Keywords

MRED, SCORE, Performance Metrics, Evaluation Framework, Uncertainty

Citation

Weiss, B. and Schmidt, L. (2010), The Multi-Relationship Evaluation Design Framework: Creating Evaluation Blueprints to Assess Advanced and Intelligent Technologies, Proceedings of the 2010 Performance Metrics for Intelligent Systems (PerMIS) Workshop, Baltimore, MD, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=906659 (Accessed June 20, 2024)

Issues

If you have any questions about this publication or are having problems accessing it, please contact reflib@nist.gov.

Created October 29, 2010, Updated February 19, 2017