Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Response Robot Evaluation Exercise (#7)

Project Overview

The National Institute of Standards and Technology (NIST), with sponsorship from the Department of Homeland Security Science and Technology Directorate, is developing a suite of DHS-NIST-ASTM International Standard Test Methods for Response Robots to quantitatively evaluate the performance of robots for emergency response applications. These standard test methods measure robot capabilities in mobility/maneuvering, energy/power, sensing, radio communication, manipulation, human-robot interaction, logistics, and safety to provide point of comparison for a variety of robot sizes and configurations prior to testing in more realistic scenarios. Statistically significant robot performance data captured within standard test methods measure incremental system improvements, highlight break-through capabilities, and support procurement and deployment decisions. More than thirty such test methods are under development with associated apparatuses, procedures, and performance metrics. They are being standardized through the ASTM International standards committee on Homeland Security Applications; Operational Equipment; Robots (E54.08.01).

This suite of standard test methods is being developed to measure the range of capabilities of response robots independent of robot size. The working definition of a "response robot" is a remotely deployed device intended to perform operational tasks at operational tempos that can serve as an extension of the operator to improve remote situational awareness, provide means to project operator intent through the equipped capabilities, improve effectiveness and efficiency of the mission, and reduce risk to the operator. Key features include:

  • Rapidly deployed
  • Remotely operated from an appropriate standoff
  • Mobile in complex environments
  • Sufficiently hardened against harsh environments
  • Reliable and field serviceable
  • Durable and/or cost effectively disposable
  • Equipped with operational safeguards

These test methods address high-priority tasks identified by emergency responders including: fast, light and mobile reconnaissance tasks for throwable robots; wide area survey tasks for hazmat or other events for packable or luggable robots; counter IED, VBIED, and PBIED tasks for mobile manipulators; aerial reconnaissance for small unmanned aerial systems (sUAS) conforming to the emerging FAA Group I class weighing less and 2kg (4.4lbs), less than 30 knots maximum speed in horizontal flight, and frangible or otherwise harmless upon impact; underwater reconnaissance for small unmanned underwater vehicles (sUUV) or remotely operated vehicles (ROV). For each of these application domains the standard test methods enable quantitative robot evaluations, provide practice tasks, and measure operator proficiency.

The standards development process involves hosting periodic robot requirements workshops, standards committee meetings, and response robot evaluation exercises at responder training facilities to gather emergency responders, robot developers, and test administrators around draft standard test methods and practice deployment scenarios. These events allow emergency responders to articulate essential robot capabilities, validate proposed test methods, and refine performance thresholds and objectives based on objective performance data captured across a class of robots. Emergency responders involved in the process learn about the state-of-the-science in robotic capabilities and help ensure that the test method apparatuses and procedures address their application needs. These events also inform robot developers regarding the reliability and applicability of their robots for actual deployment scenarios, and the ease of use of their systems as they train responders within the test apparatuses. Robot developers involved in the process learn about emerging operational requirements and can demonstrate robotic capabilities by capturing statistically significant performance data within the resulting standard test methods.

The seventh in a series of DHS/NIST Response Robot Evaluation Exercises will be hosted at the emergency responder training facility known as Disaster City in College Station, TX. Thirty emergency responders from across the country will participate -- half representing FEMA urban search and rescue teams and half representing bomb squads from across the country. They will help validate emerging standard robot test methods, become familiar with available robot capabilities, and advise robot developers regarding operational requirements. All applicable robots are invited to take part in these exercises including ground, aquatic, and small VTOL aerials under 2kg (4.4lbs). Robots will capture performance data within emerging standard test methods, familiarize and train responders within the test methods, then be deployed with responders to perform operational tasks in practice scenarios including:

  • Test methods for mobility, obstacles, endurance, radio communications (line of sight, non line of sight, and structure penetration), sensors (video acuity, pan-tilt-zoom tasks, 2-way speech intelligibility, range imager resolution, thermal imager resolution) will prepare robots to perform operational tasks for down-range reconnaissance of a hazardous materials and passenger train wrecks from stand-offs greater than 150m/500ft.
  • Test methods for navigating, searching, and mapping (2D and 3D) complex environments will prepare robots for operational tasks in building interiors and exteriors, partially collapsed structures, and confined spaces in rubble piles.
  • Test methods for mobile manipulation (non-contact inspection, access tool for window breaking and boring, and grasping/removal tasks) will prepare robots for operational tasks to counter improvised explosive devices (C-IED), vehicle-born improvised explosive devices (C-VBIED), and personnel-borne improvised explosive devices (C-PBIED).
  • Test methods for towing trailers and gripper-dragging objects will prepare robots for operational tasks in C-IED, C-VBIED, C-PBIED and US&R scenarios.
  • Test methods for underwater navigation, station-keeping, and sensor acuity will prepare for operational tasks in vehicle reconnaissance in the on-sight pond.
  • Test methods for air-worthiness, station-keeping, and sensor acuity will prepare small unmanned aerial systems with vertical take-off, hovering, and landing capabilities for operational tasks supporting several scenarios noted above.

These response robot evaluation exercises introduce emerging robotic capabilities to emergency responders within training facilities, while educating robot developers regarding the performance requirements necessary to be effective. They also help correlate the draft standard test methods with envisioned deployment tasks and lay the foundation for usage guides identifying a robot's applicability to particular response scenarios. The result will be validated or refined draft standard test methods, quantitative robot performance data captured to support balloting within the ASTM International Committee on Homeland Security, Operational Equipment (E54.08.01), and an updated Response Robot Capabilities Compendium to inform responders and soldiers regarding what robots can and cannot be expected to do reliably in the field.

These events help refine the proposed standard test apparatuses, procedures, and metrics while allowing robot developers to practice critical capabilities and measure performance in ways that are relevant to emergency responders. The associated responder training scenarios help correlate the proposed standard test methods with envisioned deployment tasks and lay the foundation for usage guides identifying a robot's applicability to particular response scenarios.

These events help refine the proposed standard test apparatuses, procedures, and metrics while allowing robot developers to practice critical capabilities and measure performance in ways that are relevant to emergency responders. The associated responder training scenarios help correlate the proposed standard test methods with envisioned deployment tasks and lay the foundation for usage guides identifying a robot's applicability to particular response scenarios.

Disaster City is a 52-acre training facility designed to deliver the full array of skills and techniques needed by urban search and rescue professionals. As part of the Texas Engineering Extension Service (TEEX) at Texas A&M University and a training site for TX-TF1, the facility features full-size collapsible structures that replicate community infrastructure, including a strip mall, office building, industrial complex, assembly hall/theater, single family dwelling, train derailment, three rubble piles, a C-VBIED scenario, and underwater vehicle reconnaissance scenario.



  • Intelligent Systems Division, National Institute of Standards and Technology 
  • Texas Engineering Extension Service (TEEX)


  • Science and Technology Directorate, Department of Homeland Security
  • National Institute of Justice


  • Southwest Research Institute
  • University of Massachusetts - Lowell
  • Pennsylvania State University
  • Space and Naval Warfare Systems Command

Expected Participants

Important Event Information

Incident Response Scenarios

Test Methods


Participant Checklist
  • Other items to bring
    • Operator interface video adapters (BNC or RCA) so we can include your interface within our quad screen video capture which includes robot video, tracking data, etc. (we'll have PAL to NTSC converter available)
    • Sensors for victim identification or hazardous material detection along with preferred simulants we can "hide" in the test methods and scenarios for you to find. Or be sure to tell us what typical simulants to have available (ammonia, for example).


This event will be held on Monday through Friday, including two days of robot practice and testing within the DHS-NIST-ASTM International Standard Test Methods for Response Robots, two days of robots deploying in operational scenarios with responders, and a final half-day standards committee meeting to capture feedback.  

Day 1 and Day 2: Robot Practice and Testing
November 14-15, 8:00am Safety Briefing - 5:00pm Hotwash

On site are robot developers and NIST administrators only. All participating robots will be run through all applicable test methods, providing practice sessions prior to arrival of the emergency responders. "Expert" operators, chosen by the robot developers to capture baseline performance data and provide developer feedback regarding the test apparatuses and test methods, should operate the robots. The robot performance data will not be published. Rather the robot developers will be exposed to the entire suite of responder-validated test methods and provided an opportunity to help refine the test methods prior to standardization (this event will be the final opportunity for this set of tests).  

Day 3: Robot Testing and Operator Training
November 16, 8:00am Safety Briefing - 5:00pm Hotwash

On site are emergency responders representing FEMA Task Force Teams from across the country, robot developers, and NIST administrators. The assembled responders will rotate in small groups through all test methods to train on robots prior to deploying them into the US&R training props on site. They will become familiar with robotic capabilities using the best performing robots in any given test method. While being exposed to the latest emerging technologies, the responders will provide feedback to developers regarding necessary capabilities, operator interfaces, and realistic usage scenarios. 

Day 4: Operational Scenarios
November 17, 8:00am Safety Briefing - 5:00pm Hotwash 

The emergency responders will focus on the most applicable robots to perform targeted tasks in the operational practice scenarios around the site, which will include embedded test methods practiced in the previous days. Robot developers will accompany the responders on scenario deployments as observers, advisors, and as operators in particularly difficult deployments to show the potential of robot capabilities. All robot developers on-site, even if your robot is not selected by responders for deployment, may watch the incident response scenarios to observe the robot deployments and absorb the lessons.  

Day 5: ASTM International Standards Committee Symposium
November 18, 8:00am 1:00pm

An ASTM International Standards Committee Symposium will be held for all participants to provide feedback on the proposed standard test methods. We also want to assess the operational impact and potential improvements necessary for robots to become useful tools for fielding. 

SESSION 1: Program Overview

  • DHS S&T Goals for the ASTM International Standards Committee (E54.08.01) and the Resulting Robot Capabilities Compendium
  • Development Process: Identifying Robot Capability Requirements; Prototyping Apparatuses, Procedures and Metrics; and Capturing Statistically Significant Robot Performance Data
  • Test Administration Policies and Regional Robot Test Facilities
  • Robot Competitions used to Refine and Disseminate Standard Test Methods: RoboCupRescue and MAGIC2010
SESSION 2: Responder/Military Use Cases:
  • National Capital Region Bomb Squad Working Group and the Nation Bomb Squad Commanders Advisory Board
  • Joint Improvised Explosive Device Defeat Organization
SESSION 3: Robot Developer Use Cases:
  • Small Robot Mobility - Pointman, Applied Research Associates. USA
  • Mobile Manipulation - Telemax, Telerob, Germany
  • Quad-Robot Aerials - AirRobot, Germany
  • Underwater ROVs - SeaBotix, USA

Developers: A block of rooms have been reserved at:
Towne Place Suites

300 University Drive East
College Station, TX 77840

Room Rate: $93.00 + tax (Please book your rooms before the cut off date to insure the room rate.)
Use Group Code: USR/NIST Response
Cut off date: October 23, 2011

Holiday Inn Express & Suites
1203 University Dr. East
College Station, TX 77840

Courtyard Marriott
3939 Highway 6 South
College Station, TX 77845

Created September 14, 2011, Updated June 2, 2021