Difference Between Standard Test Method and a Standard Robot
The National Institute of Standards and Technology, Engineering Laboratory, Intelligent Systems Division, is developing the measurement and standards infrastructure necessary to evaluate robotic capabilities for emergency responders and military organizations addressing critical national security challenges. This includes leading development of a comprehensive suite of ASTM International Standard Test Methods for Response Robots that includes more than 50 test methods for remotely operated ground, aerial, and maritime systems. Several different civilian and military sponsors have supported this effort. And the test methods have been replicated and used in dozens of locations worldwide to measure and evaluate response robot capabilities to meet the following objectives:
The suite includes 15 draft standard test methods for evaluating small initial emphasis on vertical take-off and landing systems and small hand launched fixed wing systems. For the VTOL systems, testing and practice starts within netted aviaries indoors and outdoors to avoid issues of flying in the national airspace. The test methods measure essential capabilities of robots and operator proficiency for hazardous missions defined by emergency responders and soldiers.
The working definition of a “response robot” is a remotely deployed device intended to perform operational tasks at operational tempos. It should serve as an extension of the operator to improve remote situational awareness and provide means to project operator intent through the equipped capabilities. It should also improve effectiveness of the mission while reducing risk to the operator. Key features include:
Standard test methods are simply agreed upon ways to objectively measure robot capabilities. They isolate certain robot requirements and enable repeatable testing. They are developed and validated through a consensus process with equal representation of users, developers, and test administrators. In the case of sUAS systems, one key standards committee is the ASTM International Standards Committee on Unmanned Aircraft Systems (F38). The resulting robot capabilities data captured within standard test methods can be directly compared even when the tests are performed at different sites at different times. They help establish confidence in a robot and remote operator’s ability to reliably perform particular tasks. Each standard test method includes the following elements:
A “standard robot” is a completely different concept and should be carefully considered. A standard robot would presumably meet a well-defined equipment specification for size, shape, capabilities, interfaces, and/or other features. Such equipment standards are typically intended to improve compatibility, enable interoperability, increase production, lower costs, etc. This can be important for many reasons, but can also hinder innovation. In this effort, we are not proposing to develop a standard sUAS specification of any kind. Rather, we are proposing to develop standard test methods that can be used to evaluate and compare entire classes of such systems in objective and quantifiable ways. The resulting suite of standard test methods could indeed be used to specify a standard sUAS at some point, similar to how they are used to specify purchases. But that is left to each user community to define for themselves based on their particular mission objectives. We use standard test methods to encourage implementation of new technologies to improve capabilities and measure progress along the way.
NIST is leading an international effort to develop the measurements and standards infrastructure necessary to evaluate and compare robotic capabilities for emergency responders and military organizations. The resulting standard test methods and performance metrics address critical national security challenges by enabling quantitative measurement and comparison of system capabilities and operator proficiency. They have been replicated by dozens of organizations worldwide to measure and evaluate response robot capabilities. They have helped inform procurement and deployment decisions with statistically-significant robot capabilities data for a variety of essential mission tasks. They have also helped guide robot manufacturers toward innovations that answer responder needs while encouraging hardening of developmental systems. To date, these standards have been used to specify more than $70M worth of ground response robot procurements for firefighters, bomb squads, and soldiers. These standards are now beginning to also focus operator training with newly developed measures of operator proficiency.
There are roughly twenty sUAS test methods under development within the overall suites of test methods. The initial sUAS emphasis is on vertical take-off and landing systems and hand launched fixed wing systems. Ten of these test methods for Maneuvering and Payload Functionality are described in this document, having been adopted as measures of operator proficiency for Job Performance Requirements (JPR) within the NFPA 2400
Standard for sUAS Used for Public Safety Operations.