An embedded deep learning system for augmented reality in firefighting applications

Firefighting is an inherently dangerous and potentially fatal activity. Even the most experienced firefighters can be affected by stress and anxiety during a fire, leading to disorientation and the corresponding impairment of situational awareness. Furthermore, these stressors are only exacerbated by the smoke, high temperatures, and low visibility at the scene, further compromising a firefighter’s ability to effectively respond to the situation. Situational awareness, or the real-time knowledge of continuously changing conditions at the scene, is essential to accurate decision-making and the ability of a firefighter to maintain situational awareness in these conditions is likewise critical. The firefighter’s life and the lives of those needing rescue are reliant on the firefighter’s ability to make accurate decisions during scene navigation, and recent advancements in deep learning [1], data processing, and embedded systems technologies now offer a new way to assist and improve this decision-making process.

Indeed, such technologies are already being implemented in firefighting applications. Just within the last decade, these applications have grown to include the use of wireless sensor networks (WSN) for early fire detection and response [2], virtual reality platforms for firefighter training [3, 4, 5], and autonomous or semi-autonomous firefighting robots [6, 7, 8]. But fighting fires poses many challenges to the firefighters themselves that are not adequately addressed by these solutions. Continuously changing life threatening situations can engender severe stress and anxiety, and induce disorientation that can lead to inaccurate decision making and decreased situational awareness. Recent research [9] has demonstrated a discrepancy between the information available during fire emergencies and the information required for critical situational awareness, as well as the importance of information accuracy to error-free decision making.

In fact, inaccurate judgment calls are consistently among the top causes of firefighter fireground injuries each year, and firefighter personal protective equipment (PPE), contrary to its purpose, is often a contributing factor [10]. Although firefighter PPE is designed to protect the wearer during extreme fire conditions, it is not without its drawbacks, including reduced visibility and range of motion [10, 11]. These drawbacks have led to the integration of various sensors, such as infrared cameras, gas sensors, and microphones, into firefighter PPE to assist and monitor firefighters remotely [11]. The data acquired by these sensors can also be leveraged for an additional purpose: to provide firefighters with object recognition and tracking and 3D scene reconstruction in real-time, creating a next generation smart and connected firefighting system.

The procedure for object and human detection can be used to describe the scene by using human language, prioritize the presentation of images to the commander depending on the contents of the image, or to find different scenarios on demand (by presenting those scenes that contain fire, procumbent humans, or other situations of interest). Another interesting application is to produce augmented reality for the fire fighter in the scene. The situations of interest where this implementation can be applied are diverse. For instance, fire fighters may find themselves in a scenario where their vision is impaired due to dense smoke or dust particles, or where there are no visible light, so the scene has to be illuminated with a handheld or helmet flashlight. In general, in these situations, the thermal cameras can assist in improving the visual clarity of the scenario. Nevertheless, thermal images are more difficult to interpret than visible light-based images. A way to improve the perceptual experience of the fire fighters is to use the detected images to reconstruct the scenario using augmented reality glasses. We have developed preliminary prototypes of an augmented reality system that projects the images detected and segmented by a neural network to the Hololens augmented reality glasses, commercialized by Microsoft. They project the image from a computer over a semitransparent screen, so the user can see the projected images, and also through them. The images are projected so their virtual focal point is about 10 inches away from the user’s eyes, thus producing the illusion of seeing a semitransparent screen floating over the user. Their see-through effect allows the user to interact with the environment without being impeded or disoriented by a virtual experience. The headset has an incorporated inertial measurement unit (IMU) that is used to determine the position of the head. This information is useful to maintain the position of the image when the user rotates their head.

The aim of this paper is to demonstrate the hardware deployment of the firefighter aid system. This proposed system integrates FLIR One G2 and Intel RealSense D435i thermal and depth cameras with an NVIDIA Jetson embedded Graphics Processing Unit (GPU) platform that deploys already developed machine and deep learning models [12] to process captured datasets and then wirelessly stream the augmented images in real time to a secure local network router based on a mesh node communication topology [13]. The augmented images are then relayed to a Microsoft HoloLens augmented reality platform incorporated into a firefighter’s PPE as part of a system that is insusceptible to environmental stressors while also being able to detect objects that can affect safe navigation through a fire environment with a greater than 98% test accuracy rate [12]. This relay, interpretation process and return of interpreted and projected data is real-time.

This research deploys a trained deep Convolutional Neural Network (CNN) based autonomous system to identify and track objects of interest from thermal images captured at the fireground. While most applications of CNN systems to date are found in the related fields of surveillance and defense, some recent work has explored applications in unmanned aerial vehicles (UAVs) to detect pedestrians [14] and avalanche victims [15] from the air. Other recent work has demonstrated the use of CNN systems on embedded NVIDIA Jetson platforms, such as the TX1 and TX2, for traffic sign identification (with applications for autonomous vehicles) [16] and autonomous image enhancement [17]. Other work has deployed fully-Convolutional Neural Networks (FCNNs) on the TX2 to provide single-camera real-time, vision-based depth reconstruction [18] and demonstrated an attention-based CNN model that can learn to identify and describe image content that can then be generated as image captions [19].

The embedded system described in this paper deploys a deep CNN system, but is a novel approach that, to the best of our knowledge, is the first of its kind [12]. In order to implement the augmented reality system, we constructed a light weight Mask RCNN based neural network as shown in Figure 1.This framework is built on top of a VGG-16- like model. To accomplish the object segmentation and tracking tasks, we extended the VGG model framework with proposal networks to construct a region-based CNNs (RCNNs). These models are able to localize each one of the objects in the scene and continuously track them. The model was further modified with the addition of a 1x1 Convolutional layer constructing a Mask RCNN to achieve better detection and to produce, at the same time, a semantic segmentation of the scene. The system was trained with a large quantity of images recorded in real fire training situations by firefighters to identify and track objects in real time [12].

Objects are identified and tracked in images from the thermal imaging dataset using Faster RCNN [20], and then instances of significant objects are masked and shaded using Mask RCNN [21]. While the Faster RCNN uses a fully connected neural network for classification and regression for bounding box coordinates estimation, the Mask RCNN uses a set of $1\times 1$ convolutional layers to estimate the segmentation of each one of the objects. The previous layers in both architectures are identical in structure. Therefore, in our approach we construct a hybrid network containing the outputs of both networks in a single structure. The combination of both outputs produces a sequence with the objects in the scene segmented and classified. These algorithms effectively exploit the images gathered from the infrared camera by using a trained deep CNN system to identify, track, and segment objects of interest. The tasks of object recognition and tracking have the structure depicted in Figure 1.

Furthermore, this system is capable of performing human recognition and posture detection to deduce a victim’s condition and guide firefighters accordingly, which can assist in prioritizing victims by urgency. Processed information can be relayed back to the firefighter via an augmented reality platform that enables them to visualize the results of the analyzed inputs and draws their attention to objects of interest, such as doors and windows otherwise invisible through smoke and flames. This visualization also provides localized information related to those objects.

The resulting live stream output of flirap.py (both raw and processed thermal images) is illustrated in Figure 6. Similarly, the live stream output of rscap.py (the RGB color and color mapped depth images captured by the RealSense camera) is illustrated in Figure 7. The results of the algorithm of the thermal captured feeds in firefighting scenarios is shown in Figure 8(a) and Figure 8(b).

Figure 8(a) shows the result of the object detection and tracking, where each object is confined in a bounding box with a detection probability. This is performed by the Faster RCNN described in section II. By using our modified R-CNNs, the objects are segmented in different colors and tracked in real time as shown in Figure 8(b). The sequences of segmented images are represented in the Hololens. The raw depth maps and RGB feeds are presented in 7. Currently, these two different feeds are combined to construct the 3D map of the scene via Intel Realsense built in a 3D reconstruction algorithm. We are currently working to combine these feeds with the thermal feed to construct a better 3D map in low light conditions.

The work presented here lays a foundation for the development of a real-time situational map of the structural configuration of a building that is actively built and updated via the live thermal imagery being recorded by firefighters moving through the scene. An initial demonstration of indoor positioning and path planning is presented in [22] which is based on the estimation of camera movement through estimation of the relative orientation with SIFT and Optical flow. This map, which is updated in real-time, could be used by firefighters to assist them in safely navigating the burning structure and improve the situational awareness necessary in decision making by tracking exits that may become blocked and finding alternatives. Utilizing the features detected via the approach presented in this paper, a robust localization and tracking system to track objects of interest in sequences of frames can be built. The visual features from this framework have also been coupled with a Natural Language Processing (NLP) system for scene description and allow the framework to autonomously make human-understandable descriptions of the environment to aid firefighters to improve their understanding of the immediate surroundings and assist them when anxiety levels are heightened. Our future work seeks to join these two components with a reinforcement learning (Q-learning) algorithm that utilizes the continuously updated state map and reinforcement learning techniques that assist in path planning and can be vocalized through the NLP system. The deep Q-learning based approach provides a navigation system that actively avoids hazardous paths. These three components, built on the backbone of the research presented here, can be fused to accomplish the ultimate goal of providing an artificially intelligent solution capable of guiding firefighters to safety in worst-case scenarios. Further to compress the neural net parameters into an embedded platform for a real time performance, we would like to explore the tensor train networks[23] and distributed implementation [24] for large scale data. In moments of panic, our system may assist fire fighters to regain a sense of their surroundings and the best path they can take to exit the burning structure.

This work was funded by NSF grant 1637092. We would like to thank the UNM Center for Advanced Research Computing, supported in part by the National Science Foundation, for providing the high performance computing, large-scale storage, and visualization resources used in this work. We would also like to thank Sophia Thompson for her valuable suggestions and contributions to the edits of the final drafts.