2002 Performance Metrics for Intelligent Systems (PerMIS'02) Workshop

Purpose:

As expectations for intelligent systems continue to grow, the need for quantitative evaluation of system performance becomes more critical. This third workshop in a series will bring together leading researchers to address methods for measuring the abilities of intelligent systems.

We wish to discuss ideas for quantitative engineering approaches to measuring intelligence. Performance tests and competitions are in this class. These measure the overall system performance in structured situations. There are mathematical approaches to quantifying the abilities of a system, be it through complexity measures, entropy computations, or other calculations of either internal factors or external manifestations.

Many issues remain. Must performance tests be domain specific? How can tests be propagated throughout the community? Is it reasonable to expect that researchers publish their results? Can systems with fundamentally different designs be compared? Who determines what the criteria for evaluation, or “success” are?

Among the topic areas to be considered for this workshop are:

Adaptive and Learning Systems
Unmanned Autonomous Systems
Knowledge Intensive Subsystems
Cognitive and Neural Modelling
Large Systems with Human-Computer Interaction for Decision Making
Evolutionary Computations and Activities
Hierarchical and Distributed Controllers with Elements of Autonomy
Image Processing, Classification and Interpretation
Cooperating Autonomous Robots
Multi Agent Systems
Optimization, Heuristics and Search Methods
Pattern Recognition and Classification
Behavior based Control
Self-organizing Systems
Measuring Systems for Integration Purposes
Heuristic Interpretation of test results
Automated Interpretation of test results
Modeling of Neuro-biological Autonomic Systems
Mapping Design Specifications into Performance of Intelligent Systems
Understanding Incomplete and Ambiguous Assignments
Interpreting and Performing the Assignment Under Conditions of Reduced Technological Capabilities
How Performance Depends on Knowledge Representation · Multiresolutional Ontology of Performance
Linkage Between Multiple Sensor Modalities and Performance in Intelligent Systems
Can Natural Language Communication with an Intelligent System Affect Performance?
Development of SELF in Intelligent Systems

Past Workshops

Details:

Start Date: Tuesday, August 13, 2002

End Date: Thursday, August 15, 2002

Format: Workshop

Sponsor(s):

Co-Sponsored by

The National Institute of Standards and Technology

Defense Advanced Research Projects Agency

Institute of Electrical and Electronics Engineers Control Systems Society

National Aeronautics & Space Administration

In Cooperation with

The IEEE Neural Network Council

Call for Papers (pdf)

Plenary Addresses

PerMIS 2002 Conference Schedule (pdf)

Technical Contact:

Elena Messina, NIST
Alex Meystel, Drexel University
Larry Reeker, NIST

Advisory Board
G. Adorni, University of Parma, Italy
J. Albus, NIST, USA
P. Antsaklis, University of Notre Dame, USA
M. Asada, Osaka University, Japan
G. A. Bekey, University of Southern California, USA
K. Bellman, Aerospace Integration Science Corp., USA
J. G. Blitch, SAIC, USA
P. Borne, Ecole Centrale de Lille, France
H.-H. Bothe, Technical University of Denmark, Denmark
B. Chandrasekaran, Ohio State University, USA
J. Cherniavsky, NSF, USA · M. Cotsaftis, LTME/ECE, France
R. Cottam, ETRO VUB, Belgium
F. Darema, NSF, USA
P. Dario, Scuola Superiore, Italy
P. Davis, RAND Graduate School., USA
J. Fetzer, University of Minnesota, USA
D. Filev, Ford, USA
R. Finkelstein, Robotic Technology, Inc., USA
D. Fogel, Natural Selection, Inc., USA
N. Foo, University of New South Wales, Australia
W. Freeman, University of California at Berkeley, USA
E. Fromm, Drexel University, USA
T. Fukuda, University of Nagoya, Japan
D. Gage, DARPA, USA
R. Garner, Loebner Prize Winner for 1998 and 1999, USA
G. Gerhart, US Army TACOM, USA
E. Grant, CRIM, North Carolina State University, USA
S. Grossberg, Boston University, USA
R. Gudwin, State Univerity of Campinas, Brazil
W. Hamel, University of Tennessee, USA
W. Hargrove, Oak Ridge National Laboratory, USA
M. Herman, NIST, USA
E. Horvitz, Microsoft Research, USA
M. Jabri, University of Sydney, Australia
D. Jaron, Drexel University, USA
A. Jones, NIST, USA
R. Jordan, Lockheed Martin, USA
C. Joslyn, Los Alamos National Laboratory, USA
S. Kak, Louisiana State University, USA
O. Kaynak, Bogazici University, Istanbul, Turkey
H. Kitano, Sony Computer Science Labs, Japan
K. Kreutz-Delgado, University of California at San Diego
F. Kurfess, California Polytechnic State University
J. E. Laird, University of Michigan, USA
C. Landauer, Aerospace Integration Science Corp., USA
S. Lee, Samsung Advanced Inst. of Technology, Korea
C. S. George Lee, Purdue University, USA
P. B. Luh, University of Connecticut, USA
B. Mirkin, Birkbeck College, GB
U. Ozguner, Ohio State University, USA
T. Parisini, Politecnico di Milano, Italy
K. Passino, Ohio State University, USA
L. Perlovsky, AFRL/SNHE, USA
L. Pouchard, Oak Ridge National Lab, USA
J. Pustejovsky, Brandeis University, USA
D. Repperger, AFRL/HECP, USA
E. H. Ruspini, SRI International, USA
T. Samad, Honeywell, USA
A. Sanderson, RPI, USA
R. Sanz, University of Madrid, Spain
G. Saridis, RPI, USA
A. Schultz, Naval Research Laboratory, USA
T. Shih, Tamkang University, Taiwan
R. Simmons, Carnegie Mellon, USA
M. Swinson, Sandia National Lab, USA
H. Szu, ONR Navy, USA
M. Tilden, Los Alamos National Lab., USA
S. Tzafestas, National Techical University of Athens, Greece
L. Tsoukalas, Purdue Universtity, USA
I. B. Turksen, University of Toronto, Canada
C. Weisbin, NASA, USA
T. Whalen, Georgia State University, USA
A. Wild, Motorola, USA
V. Winter, University of Omaha, USA
R. Yager, Iona College, USA
A. Yavnai, RAFAEL, Israel
Y. Ye, IBM T. J. Watson Research Center, USA
B. Zeigler, University of Arizona, USA
L. Zadeh, University of California at Berkeley, USA