Physics-informed GANs for coastal flood visualization

As our climate changes, natural disasters become more intense [2]. Floods are the most frequent weather-related disaster [3] and already cost the U.S. $3.7\,\mathrm{B\;USD}$ per year; this damage will only grow over the next several decades [4, 2]. Today, emergency managers and local decision-makers rely on visualizations to understand and communicate flood risks (e.g., building damage) [5]. Shortly after a coastal flood, however, clouds cover the affected area and before a coastal flood no RGB satellite imagery exists to plan flood response or resilience strategies. Existing visualizations are limited to informative overviews (e.g., color-coded maps [5, 1, 6]) or intuitive (i.e., photorealistic) street-view imagery [7, 8]. Expert interviews, however, suggest that color-coded maps (displayed in Fig.˜2a) are non-intuitive and can complicate communication among decision makers [9]. Street-view imagery (displayed in Fig.˜2b), on the other hand, offers intuitive understanding of flood damage, but remains too local for city-wide planning [9]. To assist with both climate resilience and disaster response planning, we propose the first deep learning pipeline that generates satellite imagery of coastal floods, creating visualizations that are both intuitive and informative.

Recent advances in generative adversarial networks (GANs) generated photorealistic imagery of faces [12, 13], animals [14, 15], or even satellite [16, 17], and street-level flood imagery [7]. Disaster planners and responders, however, need imagery that is not only photorealistic, but also physically-consistent. In our implementation, we consider both GANs and variational autoencoders (VAEs), where GANs generate more photorealistic imagery ([18, 19], Fig.˜3) and VAEs capture system uncertainties more accurately [20, 21]. Because our use case requires photorealism to provide intuition, we extend a state-of-the-art, high-resolution GAN, pix2pixHD [13], to take in physical constraints and produce imagery that is both photorealistic and physically-consistent. We leave ensemble predictions to account for system uncertainties for future work.

There are multiple approaches to generating physically-consistent imagery with GANs, where we define physically-consistent to assess: Does the generated imagery depict the same flood extent as the storm surge model? One approach is conditioning the GAN on the outputs of physics-based models [22]; another approach is using a physics-based loss during evaluation [23]; and yet another is embedding the neural network in a differential equation [24] (e.g., as parameters, dynamics [25], residual [26], differential operator [27, 28], or solution [29]). Our work focuses on the first two methods, leveraging years of scientific domain knowledge by incorporating a physics-based storm surge model in the image generation and evaluation pipeline.

This work makes three contributions: 1) the first physically-consistent visualization of coastal flood model outputs as high-resolution, photorealistic satellite imagery; 2) a novel metric, the Flood Visualization Plausibility Score (FVPS), to evaluate the photorealism and physical-consistency of generated imagery; and 3) an extensible framework to generate physically-consistent visualizations of climate extremes.

The proposed pipeline uses a generative vision model to generate post-flooding images from pre-flooding images and a flood extent map, as shown in Fig.˜1.

Data Overview. Obtaining post-flood images that display standing water is challenging due to cloud-cover, time of standing flood, satellite revisit rate, and cost of high-resolution imagery. This work leverages the xBD dataset [11], a collection of pre- and post-disaster images from events like Hurricane Harvey or Florence, from which we obtained ${\sim}3\,\mathrm{k}$ pre- and post-flooding image pairs with the following characteristics: ${\sim}.5\,\mathrm{m/px}$, RGB, $1024{\times 1024}\,\mathrm{px/img}$ (post-processing details in Section˜5.2). We also downloaded flood hazard maps (at $30\,\mathrm{m/px}$), which are outputs of NOAA’s widely used storm surge model, SLOSH, that models the dynamics of hurricane winds pushing water on land (Section˜5.2). We then aligned the flood hazard map with the flood images and reduced it into a binary flood extent mask (flooded vs. non-flooded).

Model architecture. The central model of our pipeline is a generative vision model that learns the physically-conditioned image-to-image transformation from pre-flood image to post-flood image. We leveraged the existing implementation of the GAN pix2pixHD [13] and extended the input dimensions to $1024{\times}1024{\times}4$ to incorporate the flood extent map. Note that the pipeline is modular, such that it can be repurposed for visualizing other climate impacts.

The Evaluation Metric Flood Visualization Plausibility Score (FVPS). Evaluating imagery generated by a GAN is difficult [30, 31]. Most evaluation metrics measure photorealism or sample diversity [31], but not physical consistency [32] (see, e.g., SSIM [33], MMD [34], IS [35], MS [36] , FID [37, 38], or LPIPS [39]).

To evaluate physical consistency, as defined in Section˜1, we propose using the intersection over union (IoU) between water in the generated imagery and water in the flood extent map. This method relies on flood masks, but because there are no publicly available flood segmentation models for RGB imagery, we trained our own model on ${\sim}100$ hand-labeled flooding images (Section˜5.2). This segmentation model produced flood masks of the generated and ground-truth flood image which allowed us to measure the overlap of water in between both. When the flood masks overlap perfectly, the IoU is 1; when they are completely disjoint, the IoU is 0.

To evaluate photorealism, we used the state-of-the-art perceptual similarity metric Learned Perceptual Image Patch Similarity (LPIPS) [39]. LPIPS computes the feature vectors (of an ImageNet-pretrained deep CNN, AlexNet) of the generated and ground-truth tile and returns the mean-squared error between the feature vectors (best LPIPS is $0$, worst is $1$).

Because the joint optimization over two metrics poses a challenging hyperparameter optimization problem, we propose to combine the evaluation of physical consistency (IoU) and photorealism (LPIPS) in a new metric (FVPS), called Flood Visualization Plausibility Score (FVPS). The FVPS is the harmonic mean over the submetrics, IoU and $(1-\text{LPIPS})$, that are both $[0,1]$-bounded. Due to the properties of the harmonic mean, the FVPS is $0$ if any of the submetrics is $0$; the best FVPS is $1$.

In terms of both physical-consistency and photorealism, our physics-informed GAN outperforms an unconditioned GAN that does not use physics, as well as a handcrafted baseline model (Fig.˜3).

A GAN without physics information generates photorealistic but non physically-consistent imagery. The inaccurately modeled flood extent in Fig.˜3e illustrates the physical-inconsistency and a low IoU of $0.226$ in Table˜1 over the test set further confirms it (see Section˜5.2 for test set details). Despite the photorealism ($\,\mathrm{LPIPS}=0.293$), the physical-inconsistency renders the model non-trustworthy for critical decision making, as confirmed by the low FVPS of $0.275$. The model is the default pix2pixHD [13], which only uses the pre-flood image and no flood mask as input.

A handcrafted baseline model generates physically-consistent but not photorealistic imagery. Similar to common flood visualization tools [6], the handcrafted model overlays the flood mask input as a hand-picked flood brown (#998d6f) onto the pre-flood image, as shown in Fig.˜3g. Because typical storm surge models output flood masks at low resolution ($30m/px$ [1]), the handcrafted baseline generates pixelated, non-photorealistic imagery. Combining the high IoU of $0.361$ and the poor LPIPS of $0.415$, yields a low FVPS score of $0.359$, highlighting the difference to the physics-informed GAN in a single metric.

The proposed physics-informed GAN generates physically-consistent and photorealistic imagery. To create the physics-informed GAN, we trained pix2pixHD [13] from scratch on our dataset (${\sim}7\,\mathrm{hrs}$ on $8{\times}\,\mathrm{V100}$ Google Cloud GPUs). This model successfully learned how to convert a pre-flood image and a flood mask into a photorealistic post-flood image, as shown in Fig.˜5. The model outperformed all other models in IoU ($0.553$), LPIPS ($0.263$), and FVPS ($0.532$) (Table˜1). The learned image transformation “in-paints“ the flood mask in the correct flood colors and displays an average flood height that does not cover structures (e.g., buildings, trees), as shown in $64$ randomly sampled test images in Fig.˜4. While our model also outperforms the VAEGAN (BicyleGAN), the latter has the potential to create ensemble forecasts over the unmodeled flood impacts, such as the probability of destroyed buildings.

Although our pipeline outperformed all baselines in the generation of physically-consistent and photorealistic imagery of coastal floods, there are areas for improvement in future works. For example, our dataset only contained $3k$ samples and is biased towards vegetation-filled satellite imagery; this data limitation likely contributes to our model rendering human-built structures, such as streets and out-of-distribution skyscrapers in Fig.˜4 top-left, as smeared. In addition, the dataset was generated by Maxar imagery and preliminary results suggest that our model does not generalize well to other data sources such as NAIP imagery [10]. Although we attempted to overcome our data limitations using several state-of-the-art augmentation techniques, this work would benefit from more public sources of high-resolution satellite imagery (experiment details in Section˜5.3). Finally, the computational intensity of training GANs made it difficult to fine-tune models on new data; improved transfer learning techniques could address this challenge. Lastly, satellite imagery is an internationally trusted source for analyses in deforestation, development, or military [40, 41], and with the rise of “deep-fake“ models, more work is needed in the identification of and education around misinformation and ethical AI [42]. Given our pipeline’s results, however, we hope to deploy with NOAA by integrating flood forecasts with aerial imagery along the entire U.S. East Coast.

Vision for the future. We envision a global visualization tool for climate impacts. Our proposed pipeline can generalize in time, space, and type of event. By changing the input data, future work can visualize impacts of other well-modeled, climate-attributed events, including arctic sea ice melt, wildfires, or droughts. Non-binary climate impacts, such as inundation height, or drought strength can be generated by replacing the binary flood mask with continuous model predictions. Opportunities are abundant for further work in visualizing our changing Earth, and given its potential impact for both climate mitigation and adaptation, we encourage the ML community to take up this challenge.