Augmented Reality Wildland Urban Interface (WUI) Mitigation Framework and Application

JANUS Research Group

In June 2022, JANUS Research Group was awarded nearly $1.2M for the Public Safety Innovation Accelerator Program 2022 (PSIAP-2022) funding opportunity. 

Man-made and natural disasters are increasing in size and frequency in the United States, creating significant threats to life and property. The need for increased safety through better data management and employment has never been more evident. The theme for Public Safety Organizations (PSOs) is always doing more with less and this is amplified due to COVID-19 effects. PSOs need new, cost-effective, widely employable and deployable tools to facilitate better mitigation before incidents. Team JANUS (JANUS Research Group, Western Fire Chiefs Association (WFCA), Team DSGN, and the Boise Fire Department) will develop WUI-MF (Wildland Urban Interface – Mitigation Framework), and the Boise WUI Fire Mitigation application. The WUI-MF will be used to facilitate building PSO specific mitigation applications. The Boise WUI Fire Mitigation application will be the first proof of concept built on WUI-MF.

Meet the team

JANUS’ team is made of the following key personnel:

• Radhakishan Shetty, Principal Investigator, has experience leading projects of this size having completed work on the ARgo Training project for the Defense Threat Reduction Agency. The project involved conducting SAM, UCD, tailoring computer vision algorithms for object recognition and Agile Scrum development to create an AR training package for military first responders on the use of a Polymerase Chain Reaction device.
• Scott Byers, Scrum Master, will ensure the project stays on track by mitigating any cost and schedule risks for the team. He is a proven leader with a proven process improvement and operational excellence-oriented focus. He has experience as an accomplished program management professional working in a dynamic team environment.
• Scott Halvorson, Lead Artist, has 18 years of professional experience creating graphics, 3D models, animations, environments, leading teams of multiple artists, and quality control of art assets. Scott has recent experience with Microsoft HoloLens and HoloLens 2, Magic Leap 1, and mobile phone and tablet devices creating AR training with delivered products for Polymerase Chain Reaction device training for the Defense Threat Reduction Agency, the PSCR CHARIoT Challenge, and the NIH MERRTT.
• Nick Sorrell, Senior Software Developer, has 15 years of professional experience working with learning frameworks and applications. His recent experience developing and deploying web application software to the Azure government cloud and working with DevOps platforms makes him an ideal teammate for development of the framework and administering the IT cloud infrastructure the project will depend on.
• Parker Old, Staff Software Developer, has recent experience working with PI Radhakishan Shetty on an AR/VR project to create a 3D sand table making use of Unity3D and c#. Parker will be focused on mobile application development based on the Framework.
• Western Fire Chiefs Association (WFCA) – Representing the western half of the United States including Alaska, Hawaii, Guam, and Saipan, the WFCA is unquestionably one of the leaders in data integration into the fire service. Their experience, data sources (CAD/AVL, RMS), and sensors integrated into the Fire Data Lab along with access to various fire departments in the western half of the United States presents an ideal opportunity to incorporate live data feeds directly into applications.
• Team DSGN – Created at the beginning of 2019, Team DSGN exists to enhance public safety designs. Team DSGN has competed in the previous NIST PSCR 2019 AR/VR Haptic Interfaces for Public Safety Challenge, partnering with JANUS to place fourth in the competition. In addition, Team DSGN was a JANUS’ partner for the PSCR CHARIoT Challenge. Ryan Poltermann is the Principal of Team DSGN and has over a decade of experience in public safety communications, designing radio systems around the world.

Project overview

Limitations with current and near-future head mounted AR technology creates challenges to using AR in the field. Current headsets are not ruggedized, lack features such as GPS, do not work well outdoors, and are expensive to deploy. Mobile devices require hands to operate, interfering with actual tasks first responders need to perform. Network connectivity may be intermittent or completely unavailable.

JANUS’ approach is to use the Successive Approximation Model, User Centered Design, and Scrum Agile Software Development processes and MLOps along with the iPhone, iPad, HoloLens2 and Unity3D to build AR interfaces that convey information to users. Using the already identified PSCR UI/UX Summit Report and other PSCR feedback, after action reports, and on-going feedback from Boise Fire Department and other PSOs, this provides a tailored solution guided directly by public safety for public safety, ensuring usable solutions the first time.

Janus’ primary objectives include:

• Create a framework for integration of the data and system visuals into new and existing PSO applications. Create infrastructure that allows independent modules to be developed for visualization of mitigation per PSO. Setup MLOps infrastructure to improve AI recommendations as application is used. Setup Web modules to provide reference implementation and to be integrated into external applications (prevention organizations, insurance companies, mitigation providers). 
• Display mitigation recommendations in a widely available and user-friendly commercial-off-the-shelf (COTS) AR interfaces. Convey mitigation recommendations from Boise FD to users. Display relevant open source and Boise FD data such as from FirewiseUSA, state forest service, ALERTWildfire, GIS based Ada Situational Awareness Tool, overlaid on a map visualization (Chariot Challenge IRAH interface). 
• Researching the integration of modeling tools that can provide predictive user interfaces in AR, optimizing the information based on the tasks and situations at hand. 
• Evaluate effectiveness of application’s AR interfaces (visuals, audio, gestures) for training, operations, and After-Action Review (AAR) in terms of usability by collecting user interaction data, conducting observational surveys, and gathering user feedback data.

Potential impacts of the project

WUI-MF will improve public safety operations by conveying actionable real time information to wildfire mitigation staff and will be able to be used in an educational capacity with property owners. The impact of WUI-MF and AR Interfaces will be to help the PSOs with their objective measures of performance, allowing for more effective allocation and use of resources, budgeting for the correct amount of resources, and reporting PSO accomplishments to managers, customers, and policy makers. As AR hardware matures and becomes more rugged and usable for everyday operations, WUI-MF will position a PSO to make more effective use of those hardware technologies with investments they have already made. Ultimately, Team JANUS will reduce barriers to AR adoption by familiarizing the Public Safety community with AR interfaces, increasing adoption and thus enabling new capabilities for public safety agencies.