NIST-Built Urban Ruin Put Search-and-Rescue Robots to the Test

For the sake of science, robotics researchers at the National Institute of Standards and Technology, an agency of the Commerce Department's Technology Administration, carefully designed and built a scene of urban devastation. Now they want to share it with the rest of the world.

Next week, the researchers will debut their re-creation of a collapsed building in Austin, Texas, site of the world's first competition for search-and-rescue robots. After its debut at AAAI 2000 (July 30-Aug. 2, 2000), the annual conference of the American Association for Artificial Intelligence, NIST's portable ruin is slated to be installed—or duplicated—in Melbourne, Australia, venue for the 4th RoboCup World Championships, Aug. 28-Sept. 2000.

The new test course is the first output of a beginning NIST effort to promote development and use of standard measurements for evaluating the performance of robots and other machines with electronic "brains."

"The lack of objective performance measures makes it difficult to compare approaches to accomplishing specific tasks and to generalize from one situation to another," explains John Evans, head of the Intelligent Systems Division in NIST's Manufacturing Engineering Laboratory. "Researchers are plowing some of the same ground again and again. Widely accepted performance metrics could accelerate progress in improving the capabilities of robots and in the re-use of algorithms and other system components."

To explore this perspective, NIST is hosting a "Workshop on Performance Metrics for Intelligent Systems," Aug. 14-16, 2000. Initially, NIST expected 25 to 30 participants, but about 70 researchers from around the world already have submitted papers for presentation. "There appears to be lots of interest in improving our ability to measure the capabilities of intelligent systems," Evans says.

One organization with strong interest is the Defense Advanced Research Projects Agency, which funds innovative, high-risk projects addressing future national security needs. One DARPA thrust is to promote revolutionary advances in robotics technology in support of a variety of applications, such as countering terrorist threats and military attacks in urban settings. DARPA supported development of NIST's standardized test course for urban- search and- rescue robots, and it is a co-sponsor of the upcoming workshop.

Measuring 20 meters on a side, the square test course is composed of three successively more difficult modules. The first module consists of a largely two-dimensional landscape, interspersed with pieces of overturned furniture, closets with doors ajar, floors with a uniform surface, and occasional jutting walls. The third-and most challenging--module is a chaotic environment: piles of rubble block doorways; a buckled cinder-block wall is on the verge of collapse; pipes, rebar and broken boards are strewn about; a pancaked floor section is propped perilously on a pile of debris; and floors with different textured surfaces are punctuated with scattered holes. The intermediate module has two floors, a pancaked section, and moderate difficulty for traversability, including a variety of floor coverings and alternative routes that connect the first and second stories.

In each module, collaborators from the Naval Research Laboratory will place a variety of overt and hidden "targets" for robots to locate and identify. These will include mannequins, infrared emitters (tuned to the temperature of the human body), clothing, moving body parts, hoses that emit compressed air (to represent gas leaks) and recorded sounds of human voices.

Robots will be judged on a variety of performance factors as they carry out assigned tasks. In a typical scenario, robots will enter the damaged building, maneuver through the module, find and identify human victims, place a package near a "victim," and then exit the structure. Adjusted for degree of difficulty, scores will be based on number of victims found and the time it takes to locate the targets and exit.

Robots will receive credits for conveying accurate information to rescuers outside the building, identifying structural hazards, and determining the severity of victims' injuries. The quality and usefulness of the maps of the interior areas surveyed by the machines also will be evaluated. Robots will be penalized for causing additional disruptions inside the building, such as knocking over a large debris pile.

NIST engineer Adam Jacoff, one of the test-course designers, likened the competition to an Olympic decathalon because some robots may excel in some areas and do poorly in others. Jacoff says he does not expect any high scores in the most difficult module and, perhaps, not even the intermediate one.

"Unstructured environments are very challenging for robots," he explains. "Our intention is to provide a useful benchmark so that researchers can accurately gage the performance of their technology, understand the improvements that are needed, make those improvements, and then come back to the test course and re-evaluate."

Evans and Jacoff would like to see versions of the test course built at several sites, so that robotics researchers would have reasonable access and a common basis for assessing progress. According to Evans, the NIST test course has drawn considerable interest from Japanese robotics researchers, who played a major role in launching the international RoboCup competition. However, that annual event has largely featured sports-like contests, such as modified soccer matches. Since the catastrophic 1995 Kobe, Japan earthquake, which killed more than 5,000 people and injured 35,000, Japan has launched an initiative to develop robotic technologies to aid rescue efforts at major urban disasters.

As a non-regulatory agency of the U.S. Department of Commerce's Technology Administration, NIST strengthens the U.S. economy and improves the quality of life by working with industry to develop and apply technology, measurements and standards through four partnerships: the Measurement and Standards Laboratories, the Advanced Technology Program, the Manufacturing Extension Partnership and the Baldrige National Quality Program