As expectations for intelligent systems continue to grow, the need for quantitative evaluation of system performance becomes more critical. This third workshop in a series will bring together leading researchers to address methods for measuring the abilities of intelligent systems.
We wish to discuss ideas for quantitative engineering approaches to measuring intelligence. Performance tests and competitions are in this class. These measure the overall system performance in structured situations. There are mathematical approaches to quantifying the abilities of a system, be it through complexity measures, entropy computations, or other calculations of either internal factors or external manifestations.
Many issues remain. Must performance tests be domain specific? How can tests be propagated throughout the community? Is it reasonable to expect that researchers publish their results? Can systems with fundamentally different designs be compared? Who determines what the criteria for evaluation, or "success" are?Among the topic areas to be considered for this workshop are:
The National Institute of Standards and Technology
Defense Advanced Research Projects Agency
Institute of Electrical and Electronics Engineers Control Systems Society
National Aeronautics & Space Administration
In Cooperation with
The IEEE Neural Network Council
Call for Papers (pdf)