Performance of Emergency Response Robots

Summary

Emergency responders risk life and limb interacting with known hazards to protect the public, rescue potential victims, and enhance the resilience of communities. Firefighters, bomb technicians, and urban search and rescue specialists typically wear only conventional personal protective equipment while dealing with a variety of extreme hazards for which robots should be well suited. Examples include establishing situational awareness around large structure fires; disabling or dismantling improvised explosive devices (pipes, packages, vehicles); searching for survivors in collapsed or compromised structures; assessing large scale industrial or transportation accidents; or mitigating potential terror attacks using chemical, biological, or radiological sources. Responders need capable robotic systems that can be deployed from safe standoff distances and that can reduce cognitive loads on operators through increased autonomous capabilities. As advanced capabilities are developed and improved, emergency responders need quantitative ways to measure whether a particular robot is capable and reliable enough to perform specific mission tasks remotely. Additionally, to ensure operator skills are developed and maintained, structured training methods are also needed to measure operator proficiency for reliable performance.

This project will develop the measurement science necessary to evaluate the capabilities of emergency response robotic systems, including autonomous functionalities and operator proficiency. The results of this research will help guide robot manufacturers toward developing advanced capabilities, accelerate innovation, and improve reliability. The project will also inform the development of robot performance standards to support emergency responders (federal, state, and local) in their procurements and training.

Test methods evaluate quickly deployable autonomous and remotely operated ground robots, aerial drones, and aquatic vehicles for complex and hazardous environments.

Description

Objective
Develop the measurements and standards infrastructure necessary to quantitatively evaluate robotic system capabilities along with the related proficiency of remote operators and pilots deploying into hazardous environments. We also seek to:

• Inspire innovation in robot designs, operator interfaces, and autonomy.

• Increase the competitiveness of commercial robot manufacturers.

• Improve the effectiveness and safety of emergency responders performing extremely hazardous tasks in complex environments.

Technical Idea
Emergency responders need quantitative ways to measure whether a particular robot is capable and reliable enough to perform specific missions. These missions decompose into sets of elemental robot tasks that can be represented individually as standard test methods. Categories of elemental robot tasks include maneuvering, mobility, dexterity, sensing, endurance, radio communication, durability, reliability, autonomy, logistics, and safety. Each standard test method enables repeatable testing to establish statistically significant levels of reliability and confidence that the robot can perform the task. Standard test methods essentially define the test apparatuses, procedures, and performance metrics so they can be fabricated and practiced by robot manufacturers and user groups alike. They provide a tangible language to communicate responder requirements and demonstrate robot capabilities. The apparatuses typically include increasingly challenging settings to determine the maximum capability of the robot in a given task. Examples include stair inclines of 30°, 35°, 40°, 45°. Variables are controlled initially to measure baseline performance. They are then introduced incrementally to determine the effect. Examples of variables include lighted environments (>300 lux) versus dark environments (< 0.1 lux) or dry apparatuses versus wet apparatuses. A combination of 20-30 different standard test methods can be selected with appropriate apparatus settings and environmental variables to define a particular mission profile. Repeated testing within this set of standards establishes confidence that the robot can perform the overall mission.

The project results will help inform purchasing and deployment decisions and enable specification of available combinations of robotic capabilities demonstrated to statistical significance within a set of standard test methods that represent a potential user’s mission tasks. This will align emergency responder expectations prior to purchasing by fully informing them about what the robot can and cannot be expected to do in the field.

The project deliverables will help foster innovation as robot manufacturers use the standard test methods to guide research, measure progress, and make engineering decisions regarding new sensors, coordinated manipulators, autonomous behaviors, etc. This will quicken the evolution of advanced robotic capabilities toward field deployment and improve reliability. Project deliverables will also help emergency responders improve and sustain very perishable operator skills. These standards provide inherent measures of operator proficiency that can focus training and track progress. When these standards are adopted widely for training, they can begin to isolate and measure deficiencies in equipment, including operator interfaces, and help identify technology gaps.

Research Plan
The research plan includes three phases of development for particular mission tasks:

1. Generate and validate a comprehensive suite of standard test methods to quantify component level and system level capabilities.

2. Foster innovation in robotics research and manufacturing by using the standards to guide and measure advancements in robotic system capabilities using new sensors, coordinated manipulators, autonomous behaviors, etc.

3. Disseminate the standards to support responder training and tactics with measures of operator proficiency.

During the first phase, we generate and validate a series of elemental test methods to characterize robot capabilities for specific missions defined by responder organizations. Elemental test methods typically include some combination of maneuvering, mobility, dexterity, sensing, endurance, radio communication, durability, reliability, autonomy, logistics, and safety for remotely operated ground systems, small aquatic systems, and small aerial systems under 25 kg (55 lbs). For example, a set of standard test methods will be generated to perform missions around a burning structure to establish situational awareness, identify hazardous materials, and locate potential victims. Another set of standard test methods will include manipulator reach, strength, and dexterity to measure potential fire service robots’ ability to open doors, turn valves, manipulate hose line connections, and retrieve objects. Many of the same standard test methods also apply to improvised explosive devices and other hazardous materials, and visa versa. Any given mission can provide the initial impetus to generate and validate a set of tests. The resulting standard test methods often apply to other responder missions as well.

These test methods are being standardized through the ASTM International Standards Committee on Homeland Security Applications; Response Robots (E54.09) which includes equal representation of robot manufacturers, emergency responders, and civilian/military test administrators. The resulting standards are agreed upon ways to measure individual capabilities without identifying what level of performance is required by a particular user community for a given mission task. That is, they are not equipment standards that typically define the necessary outcomes of testing to meet a defined “standard.” Any emergency responder organization can use these standard test methods to specify their mission-oriented equipment or training objectives. The various responder organizations then set thresholds of performance within each applicable standard test method to clearly articulate their mission expectations. Operational mission descriptions specified using combinations of 20-30 standard test methods provide clear guidance to robot manufacturers. They also provide clear guidance to trainees by providing targeted levels of proficiency considered necessary by their responder organizations to be effective during deployments.

Project stakeholders extend across the U.S. and internationally and test method validation exercises and robot competitions are hosted as public events around the world.

 

Quantitative results from different standard terrains evaluate overall system performance, design trade-offs for different conditions, and remote operator proficiency.

 

The Autonomous Quadruped Robot Challenge (AQRC) at the 2023 IEEE International Conference on Robotics and Automation is the newest robot competition focused on four-legged robots or “quadrupeds.”

 

The 2023 RoboCup Rescue International Championship was the latest competition in a series that has been held in different countries each year since 2000, with several regional opens per year. These competitions use 20 standard ground robot test methods developed by this project as challenge tasks.

---

Major Accomplishments
To date, the project has developed more than 50 test methods for ground robots, aerial drones, and aquatic vehicles. More than 20 test methods have been standardized so far, the rest are being validated with manufacturers, emergency responders, and soldiers prior to balloting. Draft-standard test methods can also help coordinate innovation, commercialization, and deployment of quickly emerging technologies.

Published Standard Test Methods for Evaluating Ground Response Robots

Terminology

ASTM-E2521M-23 Standard Terminology for Evaluating Response Robot Capabilities

Logistics

ASTM-E3132M-17 System Configuration

ASTM-E2592M-16 Packaging for Urban Search and Rescue Task Force Equipment Caches

Maneuvering

ASTM-E2829M-20 Sustained Speed

ASTM-E2830M-20 Tasks with Grasped Sleds

ASTM-E3310M-22 Align Ground Contacts with Parallel Rails

Mobility: Reproducible Terrains

ASTM-E2826M-20 Continuous Pitch/Roll Ramps

ASTM-E2827M-20 Crossing Pitch/Roll Ramps

ASTM-E2828M-20 Symmetric Stepfields

ASTM-E2991M-17 Gravel

ASTM-E2992M-17 Sand

ASTM-E3349M-22 K-Rails

Mobility: Confined Area Obstacles (variable)

ASTM-E2801M-20 Variable Gap Lengths

ASTM-E2802M-21 Variable Height Hurdles

ASTM-E2803M-20 Variable Incline Planes

ASTM-E2804M-20 Variable Stairs/Landings

ASTM-E3311M-22 Variable Height Rail Obstacles

Dexterity

ASTM-E3408M-23 Linear Inspection

Situational Awareness

ASTM-E2853M-22 Search Tasks

Sensing

ASTM-E2566M-17 Visual Acuity

Endurance

ASTM-E3380M-23 Endurance Using Reproducible Terrains

Radio Communications

ASTM-E2854M-21 Line-of-Sight Range

ASTM-E2854M-21 Non-Line-of-Sight Range

Published Standard Test Methods for Evaluating Aerial Drones

Endurance

ASTM-E3426M-24 Endurance

Draft Test Methods for Evaluating Aerial Drones 
A series of tests for measuring 5 levels of system capability and remote pilot proficiency in increasingly difficult environments: Open, Obstructed, and Confined (indoors). These test methods enable remote pilot credentialing that can help ensure safety (e.g., when flying in the national airspace). The same test lanes used during dark conditions evaluate system performance and pilot proficiency for night operations.

Draft Test Methods for Evaluating Aquatic Vehicles
The project has developed a series of aquatic vehicle tests that can be floating on the surface, placed on the bottom, or omnidirectional. They are typically initially deployed in clear water tanks or pools to limit the variables before adding in each variable purposely to measure performance degradation. Examples include lighted/dark environments, clear/opaque water, placid/turbulent flow.