Transformer-based Flood Scene Segmentation for Developing Countries

The Center for Research on the Epidemiology of Disasters, in affiliation with the World Health Organization (WHO), reported that natural disasters accounted for 1.3 million deaths and over USD 2 trillion in economic damage — all between 1998 and 2017 [19]. Flooding related damage is a factor in most of them [4] and frequent the list of most expensive disasters [17]. Developing economies of Asia are disproportionately affected and are the worst-hit by floods, accounting for 44% of all flood disasters from 1987-1997 [18]. India alone registers 1/5th of global deaths from floods [11]. Rapid urbanization, global climate change, and rising sea water levels will expose 1.47 billion more people to flood risk, with 89% of them living in low-middle income countries [5]. Flood Segmentation technology is instrumental for Disaster Prediction and Response is critical to save lives and livelihoods.

Flood Response: Typically, disaster management teams complete a Post Disaster Needs Assessment (PDSA) and rapidly develop infrastructure based on this report on the collected data [6]. Unmanned Aerial Vehicles (UAVs) are deployed to collect large volumes of image and video data in affected regions. PDSA is essential for identification of submerged regions, sanity check of large building structures, debris identification, and search-and-rescue (S&R) operations.

Flood Forecasting: Flood segmentation techniques can be critically important for flooding-related Early Warning Systems (EWS). According to research in [12], Indians given a flood warning are twice as likely to evacuate safely than Indians without any notice. which require constant monitoring of river or sea water levels. Comparison of current levels with historical evidence of flood-prone water levels can help understand when to trigger warnings appropriately.

Constraints: Developing countries are plagued by resource and economic constraints. Failure of macro- and micro- infrastructure planning in Nicaragua led to re-construction on top of an earthquake faultline [7]. Weak social safety and insurance policies inflate recovery time [27]. Economic vulnerability renders countries like Haiti, Ethiopia, Nepal, El Salvador in a near-permanent state of emergency alert [7]. In these countries, processing and analysis of large-scale visual data from UAVs for PDSA in Flood response is a manual process that requires multi-team intervention, which poses a serious bottleneck in search-and-response speed. Deployment of EWSs is infeasible because human monitoring of video feeds is too cumbersome and expensive.

AI Technology: To reduce the burden of manual analysis on crisis responders, Deep learning is well-suited to scale, automate and expedite these operations. The last few years have witnessed a tremendous rise in CNN-based image classification and segmentation research [21]. However, CNNs suffer from a well-known problem — large inductive biases. Conceretely, CNNs assume locality and translation equivariance, which hurt the interpretability of pure CNN-based algorithms. Recently, visual transformers have garnered attention for image classification, segmentation and object detection tasks [2, 20, 26, 3] for challenging these assumptions with comparable accuracy.

Contributions: In this work, we propose a hybrid fused CNN-Transformer: FloodTransformer to tackle flood water segmentation on the Water Segmentation Open Collection (WSOC) dataset [22]. First, we achieve state-of-the-art results and are the first work (to the best of our knowledge) to apply new transformer-driven research to the flood data domain. Second, our approach is extendable — we demonstrate the ability of our model to generalize well on unseen data sources. Further local calibration, if required at all, simply requires weight fine-tuning with previous, region-specific, flood scene data. Third, our model does not suffer from data scarcity - it only requires image data input and not complex sensor data which is hard to collect [13]. Last, the transformer-based encoder applies recent DL innovations to the flood data domain. Although the hybrid method still uses CNNs in the decoder network, the aforementioned spatial inductive biases no longer occur throughout the entire network. Dependencies between patch embeddings are learnt from scratch. This improves the robustness of our approach.

To achieve the Flood Scene Understanding, we introduce a Deep Learning model for Flood image segmentation and quantify the impact of flooding with a custom metric called Flooding Capacity.

Water Segmentation Open Collection (WSOC) dataset [22] consists of 11900 flood or water body related images, composed from multiple open-source source datasets such as COCO [9], Semantic Drone Dataset [10] and other datasets containing flood-related images [22]. These images were shot from a variety of sources such as fixed surveillance cameras, drones (UAVs), crowdsourced in-field observations (on foot, boat, vehicle) and from social media streams. Some sample images are shown in Figure 2.

Image resolutions vary from 147x150 pixels for low-quality social media streams to 2448x3264 pixels for sophisticated aerial imagery. Sampling from disparate sources is the reason for significant variations in visual appearance, illumination, water body, and geography; this makes water segmentation challenging. Newly-added social media images undergo a three-step manual annotation process to generate binary ground truth masks for pixel-wise classification between water or no-water.

We follow the WSOC method [22] and perform a 90-10 train-test split of the data. Qualitative results of output segmentation masks are shown in Figure 2. We evaluate the performance of our hybrid Flood Transformer model with mean intersection-over-union (mIOU) and Percentage Accuracy (PA) scores, on WSOC, as shown in Table 1. We compare our result to that of other benchmark models on WSOC [22], in Table 1.

As shown, FloodTransformer comfortably achieves state-of-the-art segmentation mIOU of 0.93. It is based on Zheng et al [23]’s TransFuse architecture, which comfortably beats traditional benchmarks such as UNet [14]. The Flood Capacity (FC) from Equation 1 of predicted mask in Fig 2(i)(c) is 0.53 and of mask in Fig 2(ii)(c) 0.47.

To demonstrate the robustness of our model with further qualitative results, we perform inference on out-of-dataset images, as shown in Figure 3. The Flood Capacity (FC) from Equation 1 for the predicted mask in Fig 3(i)(b) is 0.18 and for mask in Fig 3(ii)(b) is 0.26.

In this work we propose, to the best of our knowledge, the first hybrid transformer-CNN model for flood water segmentation: FloodTransformer. We achieve state-of-the-art segmentation results on the WSOC dataset and empirically demonstrate the power and robustness of visual transformers, deviating from pure CNN-based approaches.

We further validate our model’s performance by demonstrating competitive qualitiative results on out-of-dataset aerial images, indicating competitive performance on images in the wild. With the increasing number and variability of flooding-related disasters in developing countries (varying weather conditions, water flow, geography etc.) and variability of visual data collected from UAVs (varying mode of capture, camera technology etc.), this generalization capacity is crucial.

The flood water segmentation methodology adapted in this paper is useful for Flood Prediction and Response. Given the high incidence of flooding-related disasters, our method is an important step towards improving flood warning systems and covering billions of people in high-risk developing countries. It can also aid in the first step of disaster management: Post Disaster Needs Assessment (PDSA). In particular, in both cases it reduces the need for manual analysis; this is crucial in developing countries with constrained resources and high populations, where speed and efficiency of response is of paramount importance.