Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

2017.06 ASTM E54.09, Toronto, Canada

2017 - June 12-14

ASTM E54.09 Standard Committee Meeting

Hamilton Port Authority; Toronto, Canada


Committee Overview

Over the past few years our suite of 30 ground robot test methods have been adopted widely across the U.S. and internationally to support purchasing and training. We are now expanding our efforts to develop test methods for small unmanned aircraft systems (sUAS) defined by the FAA as under 25kg (55lbs) and for underwater remotely operated vehicles for hazardous environments.

Remotely operated robots enable emergency responders to perform extremely hazardous tasks from safer stand-off distances. Standard test methods help robot manufacturers and users objectively evaluate system capabilities to align with mission requirements. This improves the safety and effectiveness of emergency responders as they attempt to save lives and protect property in our communities.

The ASTM International Standards Committee on Homeland Security Applications: Response Robots (E54.09) is developing the standards infrastructure necessary to inspire innovation, inform purchasing decisions, and focus training with measures of operator proficiency. Adam Jacoff from NIST’s Intelligent Systems Division chairs the sub-committee with his team of robotics researchers and collaborators from all over the world. The standards committee is made up of robot manufacturers, emergency responders, test administrators, program managers, and others interested in developing the underlying measurement science necessary to quantitatively evaluate and compare robotic system capabilities and operator proficiencies.

This effort has been sponsored by the Department of Homeland Security, Science & Technology Directorate, Capability Development Support Group, Office of Standards. They along with other sponsors have facilitated these standards committee meetings and associated test method validation exercises for nearly fifty test methods for ground, aerial, and aquatic systems.


Meeting Outcomes

Summary of Standards Committee Meeting and Validation Exercise hosted in Hamilton, Ontario

This meeting and exercise was hosted at the Hamilton Port Authority, Ontario, Canada. More than twenty robots provided by manufacturers, emergency responders, and military organizations evaluated their robotic capabilities and operator proficiencies within the test methods. Test administrators from collaborating facilities around the world conducted trials to learn about and validate the test methods. They also helped devise new tests to address emerging requirements articulated by responders at the meeting.



Hamilton Port Authority Building
A) Front view of the Hamilton Port Authority boat storage facility used to host the event. It was mostly empty for the summer. B) Back view of the same facility. They also provided dock space in the Lake Ontario to embed our underwater test methods to conduct trials in the opaque water more typically associated with response operations.



Ground Robot Test Methods
The apparatuses set up to the left are the Basic Maneuvering and Dexterity suite of test methods. They are performed sequentially in time limited trials. B) The Access Buildings suite of test methods includes Terrains, Stoops, Stairs, Doors, and Hallway Labyrinths all with embedded Dexterity and Mapping tasks.



Ground Robot Test Methods
The main service robot used by these regional bomb squads is manufactured in Canada. A) It is shown performing the Dexterity: Rotate task using the parallel apparatus at door knob elevation. B) Here it is performing the Dexterity: Inspect task using the omni-directional apparatus.



Standard Test Methods for Aquatic
The C-frame apparatuses provide five different orientations for each task (0°-45°-90°-135°-180°). B) The walkways on top of the tank appear helpful, but they actually hindered inserting the large apparatus frames into the tank. So open top tanks are preferred.  C) The same test methods were replicated off the dock in the brackish water with natural currents and sediment in the harbor. We embed the apparatuses into these operationally relevant scenarios to measure the degradation in performance due to the uncontrolled environmental variables.



Standard Test Methods for Aerial
 The indoor netted aviaries provided safe flight testing while validating the test methods and collecting essential data for the standardization process. B) A wall of fans provided turbulence for the aerial systems, encouraging automatic holding of position and orientation to improve performance of tasks.



Standard Test Methods for Aerial
The outdoor testing took place in a more remote section of Ontario where the local responders are cleared to fly their systems outdoors. B) We set up our Situational Awareness test methods to practice and validate them using an aerial system manufactured in Canada and used by all the responders. C) Because aerial systems need to handle weather conditions, we adapted a sprinkler system to the Spiral Inspect test apparatus to enable testing resilience of the system to rain


To download the complete report:



Media Highlights


News interview with President of the Canadian Explosives Technicians Association

Local news

Other media




The ASTM International Standards Committee on Homeland Security Applications; Response Robots (E54.09) will be hosting a meeting and associated response robot exercise to validate test methods for remotely operated and autonomous ground, aerial, and aquatic systems. The resulting standard test methods can be used by anybody to measure and compare system capabilities and operator proficiency.

Key discussion topics will include:

  • Fifty test methods including apparatuses, procedures, and performance metrics
  • Collaborating test facilities worldwide that host the standards
  • Test method balloting status and priorities going forward
  • Response Robots Capabilities Compendium allowing filtering of test results to address mission requirements
  • Training within the test methods as standard measures of operator proficiency

The associated test method validation exercise will involve most of the test methods under development. International test administrators from collaborating facilities will help set up and administer the test methods.  We will validate and refine the test methods with regional emergency responders and manufacturers using their own systems to practice the tests. Manufacturers will measure baseline system capabilities. Regional responders will focus their training and measure their proficiency in head-to-head trials. They will be using their own robots to perform a sequence of standard tasks used during four other regional Robot Rodeos with more than one hundred bomb technicians throughout the U.S. in 2016-2017.  This enables direct comparison of their proficiency to regional and national averages.

The test methods will include:

  • 15 ground system test methods focusing on Maneuvering, Mobility, Dexterity, Tool Deployment, Sensing, Endurance, and Radio Comms necessary to conduct missions involving improvised explosive devices, urban search and rescue, accessing and mapping buildings, and others.
  • 15 small unmanned aircraft system (sUAS) test methods for  Maneuvering, Situational Awareness, Sensing, Endurance, and Mapping necessary to conduct missions involving search and rescue, infrastructure inspection, accident reconstruction, and others.
  • 15 Aquatic system test methods focusing on maneuvering, dexterity, tool deployment, and other capabilities necessary to conduct missions involving underwater improvised explosive devices.




Online Pre-Registration is REQUIRED to participate in the ASTM International Standards Committee on Homeland Security Applications; Response Robots (E54.09) Meeting and Associated Test Method Validation Exercise: 
Register for the upcoming meeting and exercise online. It is FREE if you choose not to pay the VOLUNTARY ACTIVITY FEE on the registration page.  When the dialogue box asks again you can simply PROCEED WITH PRE-REGISTERING.  Of course, you may contribute to ASTM if you wish. You do not need to join ASTM to attend the meeting and exercise, it is open to the public.
Our new ASTM International Standards Committee on Homeland Security Applications; Response Robots (E54.09) is official.
Join or transfer to our new sub-committee on Response Robots (E54.09) here:
If you were a member of the Operational Equipment (E54.08) sub-committee, you are not automatically a member of the new Response Robots (E54.09) sub-committee. So please go online and either shift or add your affiliation to our new sub-committee. You do not need to give up your position on any other sub-committee if your interests remain.




Monday, June 12 2017

Robots begin operating in ground, aerial, and aquatic test methods either for practice or to capture proctored test method validation trials. Regional responder VIPs meet and tour prior to meeting.

0900 Regional Responder VIP Meeting and Tour

  • Travel for attendees
  • Test administrator training
  • Robot practice in test methods

1200 Lunch served on site, robots continue testing so everybody can watch.

1300 E54.09 Committee Business

  • Introductions around the room
  • Collaborating lest facilities worldwide, particular use cases, and other standard test method initiatives.
  • Robot purchases recently specified within our suite of standard test methods
  • Response Robot Capabilities Compendium (v2015-16) overview and filtering interface, bar charts, etc.
  • Robot competitions using our test methods as challenge tasks to guide innovation and measure progress in emerging capabilities and autonomy.
  • Responder and military training events using our standard test methods to focus training and measure operator proficiency.

1600 Tour Test Method Exercise

1800 Done.


Tuesday, June 13 2017

Robots continue operating  ground, aerial, and aquatic test methods either for practice or to capture proctored test method validation trials.

0830 SAFETY BRIEFING for all the day's attendees

(Ground system testing will be continuous all day every day)

  • Configuration Identification and Sensors
  • Maneuvering (basic training)
  • Radio Comms
  • Mission Combo Sequences: Building Access

1200 Lunch served on site, robots continue testing so everybody can watch.

(Aerial system testing will be continuous all day every day)

  • Impact Forces (video only)
  • Maneuvering Suite
  • Situational Awareness Suite
  • Outline of next day's outside aerial testing (off-site)

1800 Done.


Wednesday, June 14 2017

Robots continue operating in dozens of chosen standard test methods attempting to establish statistically significant performance for responders to consider. (Note: 0800-1300 Adam is at Exec Committee meeting in Toronto.)

0830 SAFETY BRIEFING for all the day's attendees


(Aquatic system testing will be continuous all day every day)

  • C-IED suite

1200 Lunch served on site, robots continue testing so everybody can watch.


  • Relocate and set up test apparatuses
  • Situational Awareness Suite
  • Mapping Test Method (stitched images containing 10-30 objects of interest)

1800 Done.




Hamilton Port Authority



Harbor West Marina



210 Hillyard Street



Hamilton, Ontario L8L 6B6






They are providing access to a new 150 x 270 ft boat hangar to contain our ground tests, netted aviary, and a 21,000 gallon water tank.




 Dress Code

This event will consist of equal parts meetings and test method evaluations with robots indoors and outside.  So proper dress is required, including long pants and closed toe shoes. Responders please wear your insignias.






Hotel Information

Waterfront Hotel



2020 Lakeshore Road, L7R 4G8 Burlington, Canada



Only has rooms starting on June 11. Onsite Restaurant, lakefront area, walkable, 15 minutes from venue.






Hilton Garden Inn Toronto/Burlington



985 Syscon Road, Burlington, Ontario, L7L 5S3, Canada



Phone: 1-905-631-7000



 (link is external




In addition to ASTM staff, you can visit  and/or email RobotTestMethod [at] (RobotTestMethod[at]nist[dot]gov (link sends e-mail)).



Contact Information:

Adam Jacoff - Project Leader
RobotTestMethods [at] (RobotTestMethods[at]nist[dot]gov)
Adam.Jacoff [at] (Adam[dot]Jacoff[at]nist[dot]gov)


Created March 8, 2017, Updated February 25, 2022