Aerial Program Overview

DHS-NIST-ASTM-NFPA Aerial Test Methods, Army Camp Shelby 7183

• Background

• Definition of Response Robot Includes sUAS

• Difference Between Standard Test Method and a Standard Robot

• Approach

• Impacts

• Initial Test Method Suite

• Frequently Asked Questions

The National Institute of Standards and Technology, Engineering Laboratory, Intelligent Systems Division, is developing the measurement and standards infrastructure necessary to evaluate robotic capabilities for emergency responders and military organizations addressing critical national security challenges. This includes leading development of a comprehensive suite of ASTM International Standard Test Methods for Response Robots that includes more than 50 test methods for remotely operated ground, aerial, and maritime systems. Several different civilian and military sponsors have supported this effort. And the test methods have been replicated and used in dozens of locations worldwide to measure and evaluate response robot capabilities to meet the following objectives:

• Facilitate communication between user communities and commercial developers, or between development program managers and performers, by representing essential capabilities in the form of tangible test apparatuses, procedures, and performance metrics that produce quantifiable measures of success.
• Inspire innovation and guide developers toward implementing the combinations of capabilities necessary to perform essential mission tasks.
• Measure progress, highlight break-through capabilities, and encourage hardening of developmental systems through repeated testing and comparison of quantitative results.
• Inform purchasing and deployment decisions with statistically significant capabilities data. To date, the suite of ground robot test methods have been used to specify more than $60M worth of robot procurements for military and civilian organizations performing C-IED missions.
• Focus operator training and measure proficiency to track very perishable skills over time and enable comparison across squads, regions, or national averages. To date, the suite of ground robot test methods have been used by more than 250 civilian and military bomb technicians.

The suite includes 15 draft standard test methods for evaluating small initial emphasis on vertical take-off and landing systems and small hand launched fixed wing systems. For the VTOL systems, testing and practice starts within netted aviaries indoors and outdoors to avoid issues of flying in the national airspace. The test methods measure essential capabilities of robots and operator proficiency for hazardous missions defined by emergency responders and soldiers.

[Top]

The working definition of a “response robot” is a remotely deployed device intended to perform operational tasks at operational tempos. It should serve as an extension of the operator to improve remote situational awareness and provide means to project operator intent through the equipped capabilities. It should also improve effectiveness of the mission while reducing risk to the operator. Key features include:

• Rapidly deployed
• Remotely operated from an appropriate standoff
• Maneuverable in complex environments
• Sufficiently hardened against harsh environments
• Reliable and field serviceable
• Durable or cost effectively disposable
• Equipped with operational safeguards

[Top]

Standard test methods are simply agreed upon ways to objectively measure robot capabilities. They isolate certain robot requirements and enable repeatable testing. They are developed and validated through a consensus process with equal representation of users, developers, and test administrators. In the case of sUAS systems, one key standards committee is the ASTM International Standards Committee on Unmanned Aircraft Systems (F38). The resulting robot capabilities data captured within standard test methods can be directly compared even when the tests are performed at different sites at different times. They help establish confidence in a robot and remote operator’s ability to reliably perform particular tasks. Each standard test method includes the following elements:

• Apparatus (or prop): A repeatable, reproducible, and inexpensive representation of tasks that the system is expected to perform. The apparatus should challenge the system with increasing difficulty or complexity and be easy to fabricate internationally to ensure all robots are measured similarly.
• Procedure: A script for the system operator to follow (along with a Test Administrator when appropriate).  These tests are not intended to surprise anybody.  They should be practiced to refine designs and improve techniques.
• Metric: A quantitative way to measure the capability. For example, completeness of 10 continuous repetitions of a task or distance traversed within an environmental condition, etc. Together with the elapsed time, a resulting rate in tasks/time or distance/time can be calculated.
• Fault Condition: A failure of the robotic system preventing completion of 10 or more continuous repetitions. This could include a stuck or disabled vehicle requiring maintenance, or software issues at the remote operator control unit. All such failures are catalogued during testing to help identify recurring issues.

A “standard robot” is a completely different concept and should be carefully considered. A standard robot would presumably meet a well-defined equipment specification for size, shape, capabilities, interfaces, and/or other features. Such equipment standards are typically intended to improve compatibility, enable interoperability, increase production, lower costs, etc. This can be important for many reasons, but can also hinder innovation. In this effort, we are not proposing to develop a standard sUAS specification of any kind.  Rather, we are proposing to develop standard test methods that can be used to evaluate and compare entire classes of such systems in objective and quantifiable ways. The resulting suite of standard test methods could indeed be used to specify a standard sUAS at some point, similar to how they are used to specify purchases. But that is left to each user community to define for themselves based on their particular mission objectives. We use standard test methods to encourage implementation of new technologies to improve capabilities and measure progress along the way.

[Top]

NIST is leading an international effort to develop the measurements and standards infrastructure necessary to evaluate and compare robotic capabilities for emergency responders and military organizations. The resulting standard test methods and performance metrics address critical national security challenges by enabling quantitative measurement and comparison of system capabilities and operator proficiency. They have been replicated by dozens of organizations worldwide to measure and evaluate response robot capabilities. They have helped inform procurement and deployment decisions with statistically-significant robot capabilities data for a variety of essential mission tasks. They have also helped guide robot manufacturers toward innovations that answer responder needs while encouraging hardening of developmental systems. To date, these standards have been used to specify more than $70M worth of ground response robot procurements for firefighters, bomb squads, and soldiers. These standards are now beginning to also focus operator training with newly developed measures of operator proficiency.

There are roughly twenty sUAS test methods under development within the overall suites of test methods. The initial sUAS emphasis is on vertical take-off and landing systems and hand launched fixed wing systems. Ten of these test methods for Maneuvering and Payload Functionality are described in this document, having been adopted as measures of operator proficiency for Job Performance Requirements (JPR) within the NFPA 2400

Standard for sUAS Used for Public Safety Operations.