Extreme Reality Telemetry for First Responders

Imagine a limited visibility scenario: smoke fills the air, making it difficult to see more than a few feet in front of you. The roar of the flames is deafening, and you can feel the heat of the fire on your skin even through your protective gear. You know that time is of the essence, but you can't rush blindly into the inferno. Under these conditions, how would you get a sense of the situation around you or the obstacles in your path? 

This is the question researchers from Carnegie Mellon University (CMU) and University of California, San Diego (UCSD) seek to answer. So far, the solution they have landed on is: extreme reality telemetry. 

Telemetry is the process of collecting, transmitting, and analyzing data from remote sources. In the context of public safety, telemetry refers to the use of technology to gather and transmit information about the location, status, conditions of first responders, and the physical environment in the field. When the environment is particularly challenging, such as those that are hazardous, remote, or difficult to access, extreme reality telemetry could be used to collect data about air quality, temperature, humidity, and other factors that could impact the safety of first responders. 

In his work on Extreme Reality (EXR) Telemetry Interface for Real-Time Operation and Training, PSIAP awardee and Principal Investigator Yang Cai aims to leverage recent hardware and software advancements to both expand augmented reality (AR) interfaces for extreme environments and also develop virtual content like 3D immersive scenarios for extreme reality training.

Most existing AR interfaces are not built for smoky, dark, or noisy environments. Therefore, Cai and his team are developing an extreme reality helmet with interfaces that allow users to both perceive temperature changes and tactile sensations in the environment, as well as real-time breathing pattern detection to support public safety training and operations. These advancements can help improve situational awareness, enhance communication, reduce risk, and increase efficiency – all through more mindful and deliberate application of AR to first responder operations.

So far, the team has developed a prototype for live streaming ground temperature and GPS data from a firefighter’s helmet to a mobile phone map using a network connection. The map can be shared between teammates and commanders. This system has far-reaching applications for wildfire fighting since, currently, firefighters use their bare hands to touch the ground to survey the temperature, a technique called “cold trailing.” CMU’s system can automate the process and improve the efficiency of data acquisition and mapping.

In addition to the physical helmet, Cai’s team has also developed a number of AR and VR demonstrations, including scenarios involving helicopter launchpad fire escapes and extinguishers, helicopter preflight checkups, video medical procedures with mixed reality, and the “cold trailing” sensing and mapping for wildland firefighting mentioned above. In terms of the next steps, the team is using cutting-edge scan systems to create training avatars and photo realistic backgrounds that are even more lifelike with higher resolution and speed. These digital scenarios will go a long way to enhance training and help first responders gain valuable experience and skills without the need for costly, time-consuming, and dangerous real-world simulations.

While there is still work to be done, Cai and his team are making valuable progress toward developing ruggedized AR interfaces that adapt to extreme environments. By focusing their research on EXR for public safety, their work could result in increased real-time situational awareness in extreme conditions and real-time medical telemetry data for first responders. This can have major downstream impacts, like ad-hoc mobile command and control centers that can be quickly set up and deployed in response to an emergency or incident into standardized and shareable 3D building data, and more. 

To learn more about how PSCR supports innovation, be on the lookout in spring 2023 for PSCR’s Impact Assessment report that will dive into the details of PSCR program impacts from 2016 to 2022. For more information about upcoming program opportunities, be sure to sign up for the PSCR newsletter.