Extreme Reality (EXR) Telemetry Interface for Real-Time Operation and Training

Carnegie Mellon University

In April 2021, Carnegie Mellon University (CMU) was awarded $1.27M for the Public Safety Innovation Accelerator Program: Augmented Reality (AR) cooperative agreement. 

Public safety operations involve intensive interactions among first responders, extreme conditions, equipment, and communication, as well as incident command posts. Telemetry is a critical component to provide emergency response teams with situational-awareness about the responders, victims, targets, and environment of any given situation. However, existing hardware configurations are not built for tough environments such as, dark, smoke-filled and noisy conditions with poor wireless connectivity. 

That is why CMU is proposing a modular, wearable AR platform for public safety telemetry in extreme conditions. This project supports CMU’s prior work in AR headsets for location-based services (LBS) and expands their efforts in simultaneous localization and mapping (SLAM) through work in extreme environment overlays, gesture recognition, and thermal recognition.

Meet the team

• Principal Investigator Dr. Yang Cai is the Director of Visual Intelligence Studio (VIS), a Senior Systems Scientist, and a Cylab Institute and Associate Research Professor at Biomedical Engineering by Courtesy at CMU.
• Co-Investigator Dr. Lenny Weiss is an Assistant Professor in the Department of Emergency Medicine at the University of Pittsburgh School of Medicine. 
• Co-Investigator Dr. Mel Siegel is Professor Emeritus in Robotics Institute at CMU, an IEEE Fellow, and sensor and instrumentation expert.
• Subject matter expert Thomas Carr is a senior member of IEEE and Certified Emergency Manager (CEM).
• Co-Investigator Karen Lightman is the Executive Director of Metro21: Smart Cities Institute.
• Haocheng Zheng is Research Assistant from ECE Dept., CMU, who specializes in AR hardware and software development.
• Tomas Vancura is Research Assistant from ECE Dept., CMU, who specializes in thermal imaging algorithms.
• Dr. Lei Shen has a research background in AI and he is currently an AI Algorithm Engineer with publications in natural language processing and machine learning algorithms.
• Ronald Romano is the Chief of the City of Pittsburgh and is a subject matter expert in Emergency Medical Services.
• Brian Kikkola is the Assistant Chief of the Bureau of Fire in the City of Pittsburgh’s Department of Public Safety. 
• Daniel Stack is a Fire Marshal of McCandless Township, Pennsylvania. 
• Stan Caldwell is the Executive Director of Traffic21 Center.

Project overview

CMU’s proposed work has two objectives. First, the team aims to develop expanded and ruggedized AR interfaces for real-time operations that adapt to extreme environments, enhance communication, and provide the “sixth sense” to first responders. This will include adaptive Head-Up Display (HUD), haptic interfaces, intelligent thermography, and thermography-based gesture control and localization. 

Second, the team will develop virtual content for Mixed Reality training with real-time IoT data streams. Artificial Intelligence (AI) technologies will be applied to create Extreme Reality (EXR) scenarios. The AR training system will provide a feedback loop for assessing and improving performance and decision-making processes.

The team plans to integrate a number of innovations, including:

• Adaptive and haptic interface for extreme reality
• Thermal-based gesture and vital data sensing
• Virtual extreme environment overlays
• Virtual dynamic human overlays
• Ad-hoc Mobile Incident Command Post

The team will work with Public Safety partners in Pittsburgh Bureaus of Fire and EMS, STAT Medevac, Pittsburgh SWAT, Allegheny County Fire and Police Academy, McCandless Township Fire Department, and CMU Police Department.

Potential impacts of the project

CMU’s work on developing an extreme reality telemetry interface has a number of potential impacts for first responders, including:

• Real-time situation-awareness in extreme conditions 
• Real-time medical telemetry data
• Ad-hoc mobile Incident Command Post (ICP)
• Standardized and shareable 3D building data
• Mixed reality training for extreme reality scenarios
• Dual-use for real-time operation and training playback