Take a sneak peek at the new NIST.gov and let us know what you think!
(Please note: some content may not be complete on the beta site.).

View the beta site
NIST logo
Bookmark and Share

2009 Performance Metrics for Intelligent Systems (PerMIS'09) Workshop


The Performance Metrics for Intelligent Systems workshop is the only one of its kind dedicated to defining measures and methodologies of evaluating performance of intelligent systems. Started in 2000, the PerMIS series focuses on applications of performance measures to practical problems in commercial, industrial, homeland security, and military applications. It has proved to be an excellent forum for discussions and partnerships, dissemination of ideas, and future collaborations between researchers, graduate students, and practitioners from industry, academia, and government agencies.

The main theme of the ninth iteration of the workshop, PerMIS’09, seeks to address the question: Does performance measurement accelerate the pace of advancement for intelligent systems? In addition to the main theme, as in previous years, the workshop will focus on applications of performance measures to practical problems in commercial, industrial, homeland security, and military applications. Topic areas include, but are not limited to:

  • Defining and measuring aspects of a system:
    • The level of autonomy
    • Human-robot interaction
    • Collaboration & coordination
    • Taxonomies
    • Biologically inspired models
  • Evaluating components within intelligent systems:
    • Sensing and perception
    • Knowledge representation, world models, ontologies
    • Planning and control
    • Learning and adapting
    • Reasoning
  • Infrastructural support for performance evaluation:
    • Testbeds and competitions for intercomparisons
    • Instrumentation and other measurement tools
    • Simulation and modeling support
  • Technology readiness measures for intelligent systems
  • Benchmarks and applied performance measures in various domains, e.g.,
    • Industrial and manufacturing systems
    • Intelligent transportation systems
    • Emergency response robots (e.g., search and rescue, bomb disposal)
    • Homeland security systems
    • Defense robotics
    • Intelligent systems for hazardous environments (e.g., nuclear remediation)
    • Smart grid
    • Space robotics
    • Medical robotics & assistive devices
The Proceedings of PerMIS are indexed by INSPEC, Compendex, ACM Digital Library, and are released as a NIST Special Publication. Selected papers from past workshops have been published as special issues with the Journal of Integrated Computer-Aided Engineering (PerMIS’04) and the Journal of Field Robotics (PerMIS’06) and as a Springer Edited Book Volume (PerMIS’08).


Past Workshops


Start Date: Monday, September 21, 2009
End Date: Wednesday, September 23, 2009
Location: Gaithersburg
Format: Workshop


NIST, DARPA, IEEE Technical Committee on Performance Evaluation and Benchmarking of Robotic and Automation Systems (TC-PEBRAS), IEEE Washington Section Robotics and Automation Society (RAS) Chapter, and the Association for Computing Machinery Special Interest Group on Artificial Intelligence (ACM-SIGART)

CALL for Papers

Plenary Addresses

PerMIS Program (pdf)

Program at a Glance (pdf)

Digests (pdf)

Final Proceedings (pdf)

Technical Contact:

General Chair
Elena Messina, NIST 

Program Chair
Raj Madhavan, ORNL/NIST  

Publicity Chair
E. Tunstel, JHU-APL

Program Committee
S. Balakirsky NIST USA
G. Berg-Cross EM & I USA
F. Bonsignorio Heron Robots Italy
R. Bostelman NIST USA 
S. Carpin UCMerced USA
P. Courtney PerkinElmer UK
G. Dimitoglou Hood College USA
J. Evans John Evans, LLC USA
P. Evans SwRI USA 
J. Gunderson GammaTwo USA
L. Gunderson GammaTwo USA
J. Hallam SDU Denmark 
T. Kalmar-Nagy Texas A& M USA 
R. Lakaemper TempleU USA 
M. Lewis UPitt USA
G. Moses UMD USA
A. del Pobil UJaume-I Spain
E. Prassler UAppSci-BRS Germany
F. Proctor NIST USA
D. Prokhorov Toyota USA
L. Reeker NIST USA
M. Reggiani UPadua Italy
C. Schlenoff NIST USA
M. Shneier NIST USA
J. Steele, CSM USA