Take a sneak peek at the new NIST.gov and let us know what you think!
(Please note: some content may not be complete on the beta site.).

View the beta site
NIST logo
Bookmark and Share

Performance Metrics for Intelligent Systems (PerMIS'10) Workshop


The 2010 Performance Metrics for Intelligent Systems workshop will be the tenth in a series dedicated to defining measures and methodologies of evaluating performance of intelligent systems. Started in 2000, the PerMIS series focuses on applications of performance measures to applied problems in commercial, industrial, homeland security, and military applications.

PerMIS’10 will emphasize the key role of performance assessment in developing intelligent systems that can co-exist with humans towards improving the quality of our lives intertwined with automation. Adaptability to human-centered collaboration, the ability to cope with unstructured, dynamic environments, and keeping humans out of harm’s way have been widely accepted as critical prerequisites. Designing such flexible, smart, and safe systems requires that their performance be quantifiable thereby facilitating emerging technologies and societal acceptance.

Selected papers from PerMIS’10 will be considered for publication in a special issue with the International Journal of Intelligent Control and Systems. A National Science Foundation Poster Session is also being planned with travel support to undergraduate and graduate students. The Proceedings of PerMIS are indexed by INSPEC, Compendex, ACM Digital Library, and are released as a NIST Special Publication.

In relation to the main theme, topic areas include, but are not limited to:

  • Defining & Measuring Aspects and Capabilities of a Co-X (e.g. Co-Workers, Co-Inhabitants, …) System:
    • Human-Robot Interaction, Collaboration and Coordination
    • Mobility
    • Responsiveness, Reliability, Trustworthiness, Interchangeability, Durability
    • Variable Levels of Autonomy
    • Taxonomies
    • Biologically Inspired Models
  • Evaluating Components within Intelligent Systems:
    • Sensing and Perception
    • Knowledge Representation, World Models, Ontologies
    • Planning and Control
    • Learning, Adapting and Reasoning
  • Underlying Infrastructural Support for Performance Assessment:
    • Testing and Evaluation (including testbeds and competitions for inter-comparisons)
    • Instrumentation and Other Measurement Tools
    • Simulation and Modeling
  • Technology Readiness Measures for Intelligent Systems
  • Benchmarks and Applied Performance Measures in Various Domains:
    • Manufacturing, Logistics, and Industrial Systems
    • Service: Domestic, Mining, Agriculture, …
    • Intelligent Transportation Systems
    • Defense and Security
    • Emergency Response Robots (e.g. search and rescue, bomb disposal)
    • Intelligent systems for Hazardous Environments (e.g. nuclear remediation)
    • Smart Grid
    • Space Robotics
    • Medical & Healthcare


Past Workshops


Start Date: Tuesday, September 28, 2010
End Date: Thursday, September 30, 2010
Location: Hyatt Regency Baltimore at Baltimore Inner Harbor
Format: Workshop


NIST, DARPA, NSF, Association for Computing Machinery Special Interest Group on Artificial Intelligence (ACM-SIGART), IEEE Technical Committee on Performance Evaluation and Benchmarking of Robotic and Automation Systems (TC-PEBRAS), IEEE Wash./NoVA Sensors Council Chapter



Student Poster Session

Plenary Addresses

Submission Information

Travel and Accommodations

Special Sessions

PerMIS Program (pdf)

Program at a Glance (pdf)

Digests (pdf)

Technical Contact:

General Chair
Elena Messina NIST

Program Chair
Raj Madhavan UMD-CP/NIST

Publicity Chair
B. Grabowski MITRE USA

Publications Chair

Poster Session Chair
Holly Yanco U of Mass - Lowell USA

Program Committee
S. Balakirsky NIST USA
G. Berg-Cross EM & I USA
G. Blankenship UMD-CP USA
F. Bonsignorio UC3M Spain
R. Bostelman NIST USA
M. Childers ARL USA
M. Fields ARL USA
J. Gunderson GammaTwo USA
L. Gunderson GammaTwo USA
A. Kleiner UFreiburg Germany
R. Lakaemper Temple USA
M. Lewis UPitt USA
A. del Pobil UJaume-I Spain
E. Prassler UAppSci-BRS Germany
F. Proctor NIST USA
D. Prokhorov Toyota USA
C. Schlenoff NIST USA
C. Scrapper MITRE USA
M. Shneier NIST USA
E. Tunstel JHU-APL USA