Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Experimental Studies of Integrated Cognitive Systems



P Langley, Elena R. Messina


In this paper, we examine the issues that arise in the experimental study of integrated cognitive systems. We review the reasons why such artifacts are dicult to evaluate, then consider some dependent measures that can be used to characterize their behavior. Next we discuss independent variables that can in uence this behavior, in particular features of the domain and characteristics of the system, including its knowledge and experience. We then turn to domains and testbeds that support experiments with such systems, giving examples of some promising candidates. We conclude with a discussion of the scienti c goals of experimentation, which involve understanding the mapping from domain and system characteristics onto behavior.
Conference Dates
August 24-26, 2004
Conference Location
Gaithersburg , MD, USA
Conference Title
2004 Performance Metrics for Intelligent Systems (PerMIS) Workshop


cognitive systems, intelligent systems, performance evaluation, Performance Metrics, performance metrics, Robotics & Intelligent Systems


Langley, P. and Messina, E. (2004), Experimental Studies of Integrated Cognitive Systems, 2004 Performance Metrics for Intelligent Systems (PerMIS) Workshop, Gaithersburg , MD, USA, [online], (Accessed June 18, 2024)


If you have any questions about this publication or are having problems accessing it, please contact

Created August 25, 2004, Updated October 12, 2021