The National Institute of Standards and Technology is a Federal research laboratory within the U.S. Department of Commerce. Its mission is to promote U.S. Innovation and industrial competitiveness by advancing measurement science, standards, and technology in ways that enhance economic security and improve our quality of life. NIST leads international efforts toward creating critical measurement solutions and promoting equitable standards.
NIST’s test methods for small unmanned aircraft systems (sUAS) provide a way to objectively measure and compare system capabilities and remote pilot proficiency. They are low cost, easy to fabricate, and simple to conduct so they can proliferate widely. There are more than twenty test methods being standardized through the ASTM International Standards Committee on Homeland Security Applications; Response Robots (E54.09). They are also referenced as Job Performance Requirements in the National Fire Protection Association Standard for Small Unmanned Aircraft Systems Used For Public Safety Operations (NFPA 2400) and the ASTM Standard Guide for Training for Remote Pilot in Command of Unmanned Aircraft Systems Endorsement (ASTM F38.03).
It is analogous to getting a driver’s license for automobiles. Basic skills tests provide some assurance that the pilot is operating safely within the national airspace, and is not a hazard to nearby ground personnel or manned aircraft in the area. These suites of standard test methods provide common measures of performance with quantitative results. They can be conducted individually, in sequences, or embedded into operational training scenarios as repeatable tasks with scores to augment qualitative assessments. Organizations using these tests set their own thresholds of acceptable system and pilot performance to align with their airspace, environment, and mission complexities.
These tests can be used to quantitatively evaluate sUAS capabilities to align with various mission tasks. For example, the mission may require a combination of a precise position hold near large structures that might degrade GPS or radio comms while capturing detailed images or applying other sensors like thermal imagers. The measurement of the resulting image acuity is affected by all sorts of system and environmental components. The results of the associated standard test methods help users objective compare performance when operated by an “expert” pilot provided by the manufacturer, presuming that’s the best possible performance for the system. The same tests using the same aircraft can also measure any remote pilot’s proficiency as compared to that “expert” pilot. The standard tests do not identify what “good” performance is, they only describe how to measure performance in a standard way. But any organization can set their own minimum levels of remote pilot proficiency and align each system and pilot to the essential mission tasks required to be successful.
Similar benefits apply to various user groups. The key is that they all reference the exact same standard test methods so their results are comparable. Each user group will have its own specific combination of applicable standard test methods and scenarios with the tests embedded as objective scoring. But various commercial aircraft often get deployed across different mission types with different user groups. So they can learn from each other and leverage performance data when captured in the same standard test methods.
The test methods can be conducted by any individual or group. Anybody can perform a series of trials to capture their own scores. Those scores can be compared to others in their group, to regional/national averages, or to the “expert” pilot provided by the manufactures. This is a simple way of quantitatively measuring everybody’s proficiency in a widely distributed way. The basic circuit training lane called the Open Test Lane has five different flight paths to perform in individually timed trials. They can be completed in either 30 or 60 minutes overall depending on the amount of camera pointing and zooming their system can perform. Everybody tracks their own scores across all the applicable tests. Tracking and averaging scores over time provides a statistically significant measure of your proficiency. The results of different test measure each pilot their strengths and weaknesses on individual aircraft, so can lead toward a more focused training plan for improvement.
The standard test lanes are where everyone should start their training and evaluations. But the same test apparatuses are just as useful to quantify scoring throughout more operationally significant training scenarios. They simply provide a 100 point scoring system using the same targets and techniques practiced in the standard test lanes. They can be used to augment typically qualitative assessments by trainers watching each pilot closely. Also, these standard test methods can provide a quantitative basis for credentialing remote pilots. This is important so pilots can be called on for mutual aid in other jurisdictions. The standard test lanes can be set up concurrently with separate small groups of pilots helping each other as a visual observer and a forms filler to capture scores. After all have conducted the circuit training trials, they can then “attest” to each other’s scores like golfers do in tournaments. The tests require capturing images of each bucket alignment so the results can also be scored after the trial at any time by the Proctor or others, and saved along with the form as documentation of the trial outcomes supporting the resulting credential. These credentials can be general, or specific to individual aircraft classes, like a commercial driver’s license. Or they can be specific to the particular environment or mission profiles such as for operations beyond visual line of sight (BVLOS) or at night.
No, NIST does not train or certify anybody. NIST develops and validates test methods that get adopted through standards development organizations such as ASTM International, NFPA, and others. These test methods are used by others to evaluate system performance or remote operator/pilot proficiency.
NIST’s test method development process starts by gathering requirements from emergency responders and public safety organizations, along with a variety of Federal and military stakeholders. NIST then develops the underlying measurement science and related test methods to objectively evaluate key system capabilities. This process typically includes hosting exercises to validate the tests with robot manufactures and responders at
their training facilities, conferences, or competitions. The test methods also get replicated by a wide variety of users nationally and internationally to ensure they can be easily sourced and fabricated, and implemented correctly. This ensures that the test methods are “reproducible” at various locations and that the results are comparable, which is an essential step in the standardization process. This also ensures that the resulting standard test methods are useful to the widest audience of emergency responders and others, such as industrial, commercial, and even recreational users from other domains.
Think of the Payload in this case as a sensor systems, some of which are interchangeable. There are zoom cameras, or thermal imagers, or both as interchangeable payloads, etc.
The Payload Functionality variant of each test introduces some small amount of operational workload to perform in addition to maneuvering. That is, as you’re paused in front of a bucket (or two) and aligned correctly, then work your system interface to zoom in as far as possible to measure your maximum effective visual acuity including the stability of the aircraft, vibrations, twitchy pointing, digital zoom interface, contrast, focus, etc. The score for visual acuity is separate from the score for maneuvering. The trial time probably increases as well. We advocate 5 minute time limits for Maneuvering trials and 10 minute time limits for Payload Functionality trials. But even that might not be enough if your system is slow or unstable to zoom fully – that’s what you’ll learn by performing the test over and over again and comparing scores with other aircraft or with other pilots who perform better on your same aircraft.
If your aircraft has no zoom capability, or limited zoom, the Payload Functionality variant of each test is not applicable.
The Payload Functionality variant of the tests are also helpful when evaluating other sensors such as thermal imagers. If your system is ready for Payload Functionality tests, then the Point and Zoom Cameras test should be performed first to measure the baseline capability of the system. See the Usage Guide or Forms Book for more information.
Adam Jacoff - Project Leader
RobotTestMethods [at] nist.gov
Adam.Jacoff [at] nist.gov