The National Institute of Standards and Technology, Engineering Laboratory, Intelligent Systems Division, is developing the measurement and standards infrastructure necessary to evaluate robotic capabilities for emergency responders and military organizations addressing critical national security challenges. This includes leading development of a comprehensive suite of ASTM International Standard Test Methods for Response Robots that includes more than 50 test methods for remotely operated ground, aerial, and maritime systems. Several different civilian and military sponsors have supported this effort. And the test methods have been replicated and used in dozens of locations worldwide to measure and evaluate response robot capabilities to meet the following objectives:
Facilitate communication between user communities and commercial developers, or between development program managers and performers, by representing essential capabilities in the form of tangible test apparatuses, procedures, and performance metrics that produce quantifiable measures of success.
Inspire innovation and guide developers toward implementing the combinations of capabilities necessary to perform essential mission tasks.
Measure progress, highlight break-through capabilities, and encourage hardening of developmental systems through repeated testing and comparison of quantitative results.
Inform purchasing and deployment decisions with statistically significant capabilities data. To date, the suite of ground robot test methods have been used to specify more than $60M worth of robot procurements for military and civilian organizations performing C-IED missions.
Focus operator training and measure proficiency to track very perishable skills over time and enable comparison across squads, regions, or national averages. To date, the suite of ground robot test methods have been used by more than 250 civilian and military bomb technicians.
The suite includes 15 draft standard test methods for evaluating small initial emphasis on vertical take-off and landing systems and small hand launched fixed wing systems. For the VTOL systems, testing and practice starts within netted aviaries indoors and outdoors to avoid issues of flying in the national airspace. The test methods measure essential capabilities of robots and operator proficiency for hazardous missions defined by emergency responders and soldiers.
The working definition of a “response robot” is a remotely deployed device intended to perform operational tasks at operational tempos. It should serve as an extension of the operator to improve remote situational awareness and provide means to project operator intent through the equipped capabilities. It should also improve effectiveness of the mission while reducing risk to the operator. Key features include:
Standard test methods are simply agreed upon ways to objectively measure robot capabilities. They isolate certain robot requirements and enable repeatable testing. They are developed and validated through a consensus process with equal representation of users, developers, and test administrators. In the case of sUAS systems, one key standards committee is the ASTM International Standards Committee on Unmanned Aircraft Systems (F38). The resulting robot capabilities data captured within standard test methods can be directly compared even when the tests are performed at different sites at different times. They help establish confidence in a robot and remote operator’s ability to reliably perform particular tasks. Each standard test method includes the following elements:
Apparatus (or prop): A repeatable, reproducible, and inexpensive representation of tasks that the system is expected to perform. The apparatus should challenge the system with increasing difficulty or complexity and be easy to fabricate internationally to ensure all robots are measured similarly.
Procedure: A script for the system operator to follow (along with a Test Administrator when appropriate). These tests are not intended to surprise anybody. They should be practiced to refine designs and improve techniques.
Metric: A quantitative way to measure the capability. For example, completeness of 10 continuous repetitions of a task or distance traversed within an environmental condition, etc. Together with the elapsed time, a resulting rate in tasks/time or distance/time can be calculated.
Fault Condition: A failure of the robotic system preventing completion of 10 or more continuous repetitions. This could include a stuck or disabled vehicle requiring maintenance, or software issues at the remote operator control unit. All such failures are catalogued during testing to help identify recurring issues.
A “standard robot” is a completely different concept and should be carefully considered. A standard robot would presumably meet a well-defined equipment specification for size, shape, capabilities, interfaces, and/or other features. Such equipment standards are typically intended to improve compatibility, enable interoperability, increase production, lower costs, etc. This can be important for many reasons, but can also hinder innovation. In this effort, we are not proposing to develop a standard sUAS specification of any kind. Rather, we are proposing to develop standard test methods that can be used to evaluate and compare entire classes of such systems in objective and quantifiable ways. The resulting suite of standard test methods could indeed be used to specify a standard sUAS at some point, similar to how they are used to specify purchases. But that is left to each user community to define for themselves based on their particular mission objectives. We use standard test methods to encourage implementation of new technologies to improve capabilities and measure progress along the way.
NIST is leading an international effort to develop the measurements and standards infrastructure necessary to evaluate and compare robotic capabilities for emergency responders and military organizations. The resulting standard test methods and performance metrics address critical national security challenges by enabling quantitative measurement and comparison of system capabilities and operator proficiency. They have been replicated by dozens of organizations worldwide to measure and evaluate response robot capabilities. They have helped inform procurement and deployment decisions with statistically-significant robot capabilities data for a variety of essential mission tasks. They have also helped guide robot manufacturers toward innovations that answer responder needs while encouraging hardening of developmental systems. To date, these standards have been used to specify more than $70M worth of ground response robot procurements for firefighters, bomb squads, and soldiers. These standards are now beginning to also focus operator training with newly developed measures of operator proficiency.
There are roughly twenty sUAS test methods under development within the overall suites of test methods. The initial sUAS emphasis is on vertical take-off and landing systems and hand launched fixed wing systems. Ten of these test methods for Maneuvering and Payload Functionality are described in this document, having been adopted as measures of operator proficiency for Job Performance Requirements (JPR) within the NFPA 2400
Standard for sUAS Used for Public Safety Operations.
Guiding System Purchases with Comprehensive, Repeatable Evaluations
This project has pioneered the capture and use of quantitative robot capabilities data to inform and specify system purchases. This helps measure technical progress, highlight break-through capabilities, and encourage hardening of systems through repeated testing.
Focusing Training with Quantitative Measures of Operator Proficiency
This project has developed standard measures of operator proficiency using a circuit training model that enables operators to compare themselves to “expert” performance, or regional and national averages. The test methods help focus training and track very perishable skills over time. This approach has been validated using ground robot test methods with more than 300 bomb technicians internationally in more than 30 locations.
Inspiring Innovation and Measuring Progress in Technology Integration
This project has hosted dozens of requirements workshops and robot competitions with thousands of participants to refine and validate test methods. These competitions guide developers toward implementing the combinations of capabilities necessary to perform essential mission tasks. Several “best-in-class” ground robots have emerged and been commercialized to deploy advanced mobility, dexterity, and mapping into complex and hazardous environments.
The suite of test methods for sUASs includes the following:
Orbit a Point (Move and Rotate) (WK58932) - This test method evaluates the capability to move and rotate around a point. The system performs a series of basic maneuvers using an onboard camera to align with centrally located bucket targets from a defined radius and altitude. Surrounding bucket targets are used to define the intended radius and altitude. This test method can be conducted manually using discrete move and rotate maneuvers or automatically using orbit features of the system.
Avoid Obstacles (Figure-8s) (WK58933) - This test method evaluates the capability to maneuver around vertical obstacles (shown as yellow posts) and horizontal obstacles (shown as red bars). The system performs a series of figure-8 paths in various orientations including nose-forward, nose-left, and nose-right.
Fly Straight and Level (WK58934) - This test method evaluates the capability to fly straight and level using a visual target as a guide. The system performs a series flights toward such targets either from multiple directions or in a back and forth manner between two targets using the recessed bucket target to assess deviations from the linear trajectory.
Land Accurately (WK58935) - This test method evaluates the capability to land accurately from vertical and downward 45 degree descending approaches. The system performs a series landings on a metered platform from a defined range, altitude, and four different approach directions. When performing the angled approaches, the recessed targets are used to guide the descent.
Point and Zoom Cameras (WK ####) - This test method evaluates the capability to point and zoom cameras at near- field and far-field visual acuity targets from a specified hover position. The system performs a series of target identifications alternating between near- field and far-field visual acuity targets separated by a 180-degree rotations.
Inspect Objects (WK58936)- This test method evaluates the capability to move and rotate around an object of interest to identify key features. The system performs a series of basic maneuvers using an onboard camera to align with centrally located bucket targets from a defined radius and altitude. Surrounding bucket targets are used to define the intended radius and altitude. This test method can be conducted manually using discrete move and rotate maneuvers or automatically using orbit features of the system.
Inspect Objects (WK58937) - This test method evaluates the capability to move around an object of interest to inspect key details from close proximity. The system performs a series of basic maneuvers using an onboard camera to align with bucket targets to inspect Downward, Forward, Omni-directional, and Upward (note shown) objects.
Map Wide Areas (WK58938) - This test method evaluates the capability to localize and map a variety of known and unknown objects across a wide area. The system performs its mapping function from a prescribed altitude intended to force extensive stitching of images. Ground targets are placed at known locations throughout a scenario. The embedded ground objects are made of standard test apparatuses used in other test methods and operationally significant items.
Map Wide Areas (WKXXXX) - This test method evaluates the capability to drop a payload accurately from a defined altitude. The system performs a series of drops on a metered platform from different altitudes. The payloads can be weighted surrogates or operationally significant delivery items.