Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Multi-Relationship Evaluation Design: Modeling an Automatic Test Plan Generator



Brian A. Weiss, Linda C. Schmidt


Advanced and intelligent systems within the manufacturing, military, homeland security, and automotive fields are constantly emerging and progressing. Testing these technologies is crucial to (1) inform the technology developers of targeted areas for improvement, (2) capture end-user feedback, and (3) substantiate the degree of the technology's capabilities. Evaluation designers have put forth considerable effort in developing methods to speed test-plan generation. The Multi-Relationship Evaluation Design (MRED) methodology is being created to gather multiple inputs from several source categories and automatically output evaluation blueprints that identify the pertinent test-plan characteristics. MRED captures input from three categories including the evaluation stakeholders, the technology state, and the available resources. This information and the relationships among these inputs are combined as input into an algorithm that will yield specific test plan characteristics. This paper reviews the MRED methodology as it enters its final stages of development including new discussion of: the relationships among the various inputs and the chosen method of Evaluative Voting to capture Stakeholder Preferences. An example focusing on the design of test plans to evaluate a robotic arm is also presented to bring further clarity to the latest MRED developments.
Proceedings Title
2012 Performance Metrics for Intelligent Systems Workshop
Conference Dates
March 20-22, 2012
Conference Location
College Park, MD


MRED, Performance Metrics, Evaluation Framework, Uncertainty


Weiss, B. and Schmidt, L. (2012), Multi-Relationship Evaluation Design: Modeling an Automatic Test Plan Generator, 2012 Performance Metrics for Intelligent Systems Workshop, College Park, MD, [online], (Accessed May 22, 2024)


If you have any questions about this publication or are having problems accessing it, please contact

Created April 11, 2012, Updated February 19, 2017