Augmented Reality for Public Safety

Public safety operations involve intensive interactions among first responders, extreme conditions, equipment, and communication, as well as incident command posts. Telemetry is a critical component to provide emergency response teams with situational-awareness about the responders, victims, targets, and environment of any given situation. However, existing hardware configurations are not built for tough environments such as, dark, smoke-filled and noisy conditions with poor wireless connectivity.

This project supports CMU’s prior work in AR headsets for location-based services (LBS) and expands their efforts in simultaneous localization and mapping (SLAM) through work in extreme environment overlays, gesture recognition, and thermal recognition.

CMU’s proposed work has two objectives. First, the team aims to develop expanded and ruggedized AR interfaces for real-time operations that adapt to extreme environments, enhance communication, and provide the “sixth sense” to first responders. This will include adaptive Head-Up Display (HUD), haptic interfaces, intelligent thermography, and thermography-based gesture control and localization. 

Second, the team will develop virtual content for Mixed Reality training with real-time IoT data streams. Artificial Intelligence (AI) technologies will be applied to create Extreme Reality (EXR) scenarios. The AR training system will provide a feedback loop for assessing and improving performance and decision-making processes.

The team plans to integrate a number of innovations, including:

• Adaptive and haptic interface for extreme reality
• Thermal-based gesture and vital data sensing
• Virtual extreme environment overlays
• Virtual dynamic human overlays
• Ad-hoc Mobile Incident Command Post

The technology is expected to assist first responders in emergency environments of fire, flood, shooting, cyber attack, and medical distress. It would have impacts on public safety planning, training, and situation-awareness.

Read more about the full project here.