Obstructed Test Lane and Scenarios

• Obstructed Test Lane

• Obstructed Scenario

  • Hostage Situation

  • Vehicle Takedown

  • Hazardous Tanker Truck Accident and Fire

  • Structure Exterior Inspection

  • Flooded House and Collapsed Bridge

  • Night Operations House Surveillance and Search

 

• Sticker Files (Obstructed Test Lane and Scenarios)(8/29/22)

• NIST sUAS Obstructed Test Lane - Scoresheets

• NIST sUAS Obstructed Test Lane - Pocket Guides

The Obstructed Test Lanes enable remote pilots to fly safe and repeatable flight paths to inspect objects within close proximity to obstructions. They include a comprehensive set of 5 different tests that guide remote pilots through various standoff positions, orientations, and perches at 2-3 m (6-10 ft) from objects. They can be performed outdoors or indoors to control lighting, weather, and access to the Global Positioning System (GPS).

 

The Obstructed test apparatuses use “dual bucket alignments” to enable remote pilots to triangulate a safe standoff position by simultaneously aligning with a perpendicular bucket and its associated angled bucket. The dual bucket alignments form a right triangle with equal dimensions for the bucket separation and the aircraft standoff. They use pairs of 7.5 liter (2 gallon) buckets with 20 cm (8 in) diameter recessed targets inside at a 2 m (6 ft) spacing.

Alignment rings inside the perpendicular (90 degree) white buckets visually guide safe flight paths toward and away from the objects. Alignment rings inside the associated angled (45 degree) buckets indicate when the aircraft is in the designated safe and repeatable standoff position. That’s where a remote pilot can maintain position and operate their interface to identify increasingly small features on acuity targets inside the buckets. This modest operational workload includes zoom, focus, and exposure to capture images for scoring after the trial. White and black buckets require maximum control of exposure levels or thermal palettes to discern more details and score more points.

There are 5 different tests with increasing difficulty called Perch, Wall, Ground, Alley, and Post. The dual bucket alignments guide remote pilots through a series of 10 positions, orientations, and perches within both the standard test lanes and the operational scenarios embedded with scoring tasks. All tests and scenarios result in quantitative scores up to 100 points maximum to facilitate measurement, tracking, and comparison across different aircraft and/or remote pilots.

Procedure: The procedure is the same for all Obstructed tests. Each test has dual bucket alignments designated 1 – 4 that at are performed in a sequence. The sequence includes some backtracking to ensure the tasks are performed in various directions relative to the obstacles involved. The sequence of positions is 1 2 3 4 – 3 2 1 – 2 3 4 with the red underlined numbers indicating the backtracking part of the sequence. That results in 10 dual bucket alignments or 20 bucket alignments total.

Maneuvering Trials: A complete trial totals up to 100 points maximum for 20 bucket alignments. Points are scored using a single no zoom image of each bucket showing either a full alignment ring (5 points), a partial alignment ring (1 point), or no alignment ring (0 point).

Acuity Targets: All trials result in maneuvering scores. But Payload Functionality trials add an operational workload to identify acuity targets while aligned with buckets. The level of detail the system can discern should be known and set prior to conducting a Payload Functionality trial. Each acuity target has 5 increasingly small gap orientations to identify correctly. The smallest features are 1 mm (0.04 in) needed to read small text on shipping labels, for example. Each identifiably gap orientation is verbally conveyed to a Proctor during the trial, or when operating alone a single full zoom image of the acuity target can be captured to score after the trial.

Time Limits: Test trials are not intended to be races. But trial time limits can be imposed to minimize fatigue across multiple tests or to maintain a schedule. Trial time limits should be long enough for an “expert” to complete a perfect trial. Scores of incomplete trials due to expired time limits can only be compared to trials with the same elapsed time limits. Typical time limits are typically set to 5 minutes for Maneuvering trials and 10 minutes for Payload Functionality trials. Although any time limits may be used depending on the drone, the pilot, and the environment.

Faults: A fault is any contact with a test apparatus or any safety issue such as crossing a designated boundary between test lanes or the remote pilot flight line. Any fault results in an end of trial.

Metrics: A complete trial requires performing all the designated bucket alignments in order with no faults (contact with an apparatus) or safety issues (exceeding a boundary). If the trial is not complete, the metrics below do not apply. Keep practicing until complete trials are routinely achieved before applying these metrics:

• Score (total points) – primary: Presuming a complete Maneuvering trial or Payload Functionality trial, the total points are a measure of the combined effectiveness of the aircraft system and the remote pilot to maneuver through all the positions, orientations, and perches necessary to align with all the buckets. Trial scores add up to 100 points maximum. These scores can be used to compare remote pilot proficiencies when using the same drone and interface in the same test method. They also can be compared to the score of an “expert” pilot, typically provided by the manufacturer, which is considered the 100th percentile of proficiency on a particular aircraft system.
• Efficiency (elapsed time) – optional: Presuming a perfect score for a Maneuvering Trial or a Payload Functionality trial, the elapsed time is a measure of the combined efficiency of the aircraft system and the remote pilot. Elapsed trial times can help distinguish between perfect scores to identify more efficient techniques or approaches.

Note: The verge points of the dual bucket alignments designate the most efficient locations to score all buckets. However, stable hovers at the verge points between buckets can be difficult to enforce similarly for various drone sizes and pilot proficiencies. So each bucket can be scored individually from any desired proximity, understanding that the resulting scores and trial times may be negatively affected.

[TOP]

The bus hostage scenario included role players as perpetrators and victims with actual law enforcement organizations from the region demonstrating simulated tactics including drones and a ground robot.

We added quantitative measures of performance to this scenario using dual bucket alignments to guide remote pilots safely through a designated series of inspection tasks. This enables comparison of scores up to 100 points for pilots and aircraft that can reliably perform the various bucket alignments and identify the smallest visual/thermal acuity features across all available targets. The trial time limit was set to 20 minutes to remain within one battery charge and to maintain a schedule for multiple pilots. Time limited trials also enable direct comparisons of scores for completeness and efficiency. Only scores using similar aircraft and time times are directly comparable to evaluate pilot proficiency, but a variety of different aircraft can be used to compare overall scores and ease of use.

The flight paths included safe and repeatable standoff positions, orientations, and perches within 2 m (6 ft) of objects for aircraft flying the most efficient flight path. But bucket alignments can be achieved from any standoff for larger aircraft using more capable zoom lenses. The embedded scoring tasks are 7 liter (2 gallon) buckets with 20 cm (8 in) diameter recessed targets inside. The perpendicular (90 degree) white buckets provide visual alignments for the remote pilot to trust as safe vectors to approach and leave the object being inspected. The associated angled (45 degree) buckets enable triangulation to maintain a safe proximity from the object while operating their zoom, focus, and exposure to capture images for scoring after the trial.

Each set of tasks included 10 positions and orientations worth 10 points each distributed throughout the scenario. The pilot scores 5 points for getting aligned with each perpendicular bucket plus another 1-5 points for correctly identifying increasingly small features on the acuity target located inside the associated angled bucket. The smallest features to identify are 1 mm (0.04 in) representing small text on shipping labels, for example. The white and black pairs of buckets require maximum exposure control to capture images for scoring after the trial.

Large vehicles like buses can require interior searches as well. These relate to the Confined Test Lanes described later, but are typically part of these types of scenarios. They are intended only for scoring by aircraft that can safely fly inside the bus.

[TOP]

[TOP]

The tanker truck accident and fire scenario used 5 large bucket apparatuses to guide remote pilots to points of view around the vehicles for a fast tempo initial assessment. We also embedded 5 small bucket apparatuses near features needing detailed inspections such as the inside of the cab, dripping valves, and gauges to read. This scenario included drop accuracy tasks to emplace remote sensors trying to determine the nature of the hazard (see the sensors on the ground).

[TOP]

Structure exterior inspection tasks include looking through windows and doors along with surrounding ground objects of interest. In this case, the tasks were embedded around a partially collapsed structure. The objective was to safely fly in close proximity of about 2 m (6 ft) from the windows and doors to perform a window/door clearing maneuver with high/low and left/right views inside the structure.

[TOP]

Several houses were submerged with victims on the roof and others in the water nearby. The tasks to perform included delivery of personal floatation devices to those stranded on the roof, searching in windows for survivors, and identifying/tracking moving swimmers. The omni bucket apparatuses replaced the actors as objects to identify and the landing served as target for the dropped payload accuracy.

[TOP]

This scenario was actually conducted the week prior at a different event, but nicely augments all of the scenarios above. This house surveillance at night used two sets of horizontal and vertical Obstructed test apparatuses on all four sides of the house, guiding the remote drone pilot to safe locations among a variety of very difficult obstacles such as the overhanging roof, trees, shrubs, power lines, shrubs, etc. A set of 5 vertical test apparatuses with 10 positions and orientations were attached to all sides of the house totaling 100 points. A set of and 5 horizontal test apparatuses with 10 positions and orientations were also placed on or near objects of interest around the house for another 100 points.

[TOP]

 

Level 4  Obstructed Lane - Payload Functionality:

Note: Obstructed and Confined use the same stickers. 

• NIST sUAS 4 Inch Obstructed Lane - Stickers for Linear Rails - WHITE and BLACK - 3.5in Diam (v2020B Answer Key) (4/19/23)

Level 4 Obstructed Scenario - Payload Functionality:

• NIST sUAS 4 Inch Obstructed Scenario - Stickers for Linear Rails - WHITE and BLACK - 3.5in Diam (v2020B Answer Key) (4/19/23)

Read First

• Directions for printing stickers (8/29/22)

Level 4  Obstructed Lane - Payload Functionality:

• NIST sUAS Obstructed Lane - Stickers for Linear Rails - WHITE and BLACK - 8in Diam (v2020B Answer Key)
• NIST sUAS Obstructed Lane - Stickers for Linear Rails - WHITE and BLACK - 8in Diam (v2020B Answer Key) (ROUND STICKERS ONLY)
• NIST sUAS Obstructed Lane - Stickers for Linear Rails - WHITE and BLACK - 8in Diam (v2020B Answer Key) (RECTANGULAR STICKERS ONLY)

Level 4 Obstructed Scenario - Payload Functionality:

• NIST sUAS Obstructed Scenario - Stickers for Linear Rails - WHITE and BLACK - 8in Diam (v2020B Answer Key)
• NIST sUAS Obstructed Scenario - Stickers for Linear Rails - WHITE and BLACK - 8in Diam (v2020B Answer Key) (ROUND STICKERS ONLY)
• NIST sUAS Obstructed Scenario - Stickers for Linear Rails - WHITE and BLACK - 8in Diam (v2020B Answer Key) (RECTANGULAR STICKERS ONLY)

 

[TOP]