Developing World Model Data Specifications as Metrics for Sensory Processing for On-Road Driving Tasks

Published

October 1, 2003

Author(s)

Anthony J. Barbera, John A. Horst, Craig I. Schlenoff, Evan K. Wallace, David Aha

Abstract

The building of knowledge-intensive real-time intelligent control systems is one of the most difficult tasks humans attempt. It is motivated by the desire to create an artificial reasoning system that is capable of intelligent behavior, i.e., replicating the ability to act upon the world and to successfully accomplish activities that are only possible with the levels of knowledge processing exhibited by human beings. A critical question to be answered is how is the success of this effort to be measured and evaluated. Measurement of the outward observable system behavior, while somewhat indicative does not really measure the correctness or quality of the system's capabilities. This is especially true in complex real-time control systems such as autonomous on-road driving. Here, the system may look like it is behaving reasonably, but in reality its decisions could be faulty such that it might produce sudden unexpected behavior that leaves the developer asking why did it do that? Complex real-time control systems are characterized by the major components of sensory processing to measure entities and events of interest in the environment; internal world model processing that derives world representations from sensory processing and task context internal states; and the behavior generation processing that reasons from this world model, develops alternate plans, and makes value judgments to decide on the next appropriate output action to accomplish the goal tasks. What is needed are performance metrics at the level of these internal processing components that can be used to judge their quality and correctness. This paper describes an on-going effort at NIST, funded by the DARPA MARS On-Road Driving Project, to do a task analysis of autonomous on-road driving which includes the development of performance metrics.This project uses the NIST Real-time Control System (RCS, now referred to as 4D/RCS) [1] design methodology and reference architecture to develop a task decomposition representational format for the on-road driving task knowledge. This task decomposition representation is used as the framework to further specify the world model entities, attributes, features, and events required for proper reasoning about each of the subtask activities. These world model specifications, in turn, are used as the requirements for the sensory processing system. These requirements identify those things that have to be measured in the environment, including their resolutions, accuracy tolerances, detection timing, and detection distances for each subtask activity. From these can be developed a set of performance metrics that allow validation of sensory processing by evaluating the world model representations it produces for each individual component subtask activity. In this way, taxonomies of autonomous capabilities can be developed and tested against these sensory processing and world model building performance metrics. Additional metrics can be developed to measure the performance characteristics of the behavior generation component with its planning and value judgment operations, but these additional metrics are not the topic of this paper.

Proceedings Title

Performance Metrics for Intelligent Systems, Workshop||PerMIS 2003|

Conference Dates

September 16-18, 2003

Conference Location

Undefined

Conference Title

Performance Metrics for Intelligent Systems

Pub Type

Conferences

Keywords

finite state machines, on-road driving, performance metrics, sensory processing, task decomposition

Citation

Barbera, A. , Horst, J. , Schlenoff, C. , Wallace, E. and Aha, D. (2003), Developing World Model Data Specifications as Metrics for Sensory Processing for On-Road Driving Tasks, Performance Metrics for Intelligent Systems, Workshop||PerMIS 2003|, Undefined (Accessed April 18, 2024)