Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Performance Evaluation and Benchmarking of Intelligent Systems Book

Performance Evaluation and Benchmarking of Intelligent Systems Book
2009, XIX, 338 p., Hardcover ISBN: 978-1-4419-0491-1

Edited by:

R. Madhavan, Oak Ridge National Laboratory, Oak Ridge, TN, USA;

E. Tunstel, Johns Hopkins University Applied Physics Laboratory, Laurel, MD, USA;

E. Messina, National Institute of Standards and Technology, Gaithersburg, MD, USA


About this book:

  • Defines measures and methodologies of evaluating performance of intelligent systems
  • Covers a wide range of subject areas related to intelligent system
  • Written by leading experts from their presentations at a workshop held at the National Institute of Standards and Technology (NIST)

Performance Evaluation and Benchmarking of Intelligent Systems presents research dedicated to the subject of performance evaluation and benchmarking of intelligent systems by drawing from the experiences and insights of leading experts gained both through theoretical development and practical implementation of intelligent systems in a variety of diverse application domains. Scientific methodologies for standardization and benchmarking are crucial for quantitatively evaluating the performance of emerging robotic and intelligent systems' technologies. This contributed volume offers a detailed and coherent picture of state-of-the-art, recent developments, and further research areas in intelligent systems.

The chapters cover a broad range of applications, such as assistive robotics, planetary surveying, urban search and rescue, and line tracking for automotive assembly. Subsystems or components described in this book include human-robot interaction, multi-robot coordination, communications, perception, and mapping. Chapters are also devoted to simulation support and open source software for cognitive platforms, providing examples of the type of enabling underlying technologies that can help intelligent systems to propagate and increase in capabilities.

Performance Evaluation and Benchmarking of Intelligent Systems serves as a professional reference for researchers and practitioners in the field. This book is also applicable to advanced courses for graduate level students and robotics professionals in a wide range of engineering and related disciplines including computer science, automotive, healthcare, manufacturing, and service robotics.

Written for:

Researchers and practitioners in the field of intelligent systems and robotics


  • benchmarks
  • intelligent systems
  • performance metrics
  • quantitative performance evaluation
  • robotics
  • standardized test methods

Table of Contents:

Chapter 1. Metrics for Multiagent Systems

Chapter 2. Evaluation Criteria for Human-Automation Performance Metrics

Chapter 3. Performance Evaluation Methods for Assistive Robotic Technology

Chapter 4. Issues in Applying Bio-Inspiration, Cognitive Critical Mass and Developmental-Inspired Principles to Advanced Intelligent Systems

Chapter 5. Evaluating Situation Awareness of Autonomous Systems

Chapter 6. From Simulation to Real Robots with Predictable Results: Methods and Examples

Chapter 7. Cognitive Systems Platforms using Open Source

Chapter 8. Assessing Coordination Demand in Cooperating Robots

Chapter 9. Measurements to Support Performance Evaluation of Wireless Communications in Tunnels for Urban Search and Rescue Robots

Chapter 10. Quantitative Assessment of Robot-Generated Maps

Chapter 11. Mobile Robotic Surveying Performance for Planetary Surface Site Characterization

Chapter 12. Performance Evaluation and Metrics for Perception in Intelligent Manufacturing

Chapter 13. Quantification of Line Tracking Solutions for Automotive Applications


Created October 7, 2009, Updated September 21, 2016