Learning Restoration is Not Enough: Transfering Identical Mapping for Single-Image Shadow Removal

To restore the shadow-free image from the degraded shadow counterpart, traditional methods (Mohan et al., 2007; Finlayson et al., 2005; Guo et al., 2012; Zhang et al., 2015; Finlayson et al., 2009, 2005) rely on the priors information, e.g. gradients, illumination, and patch similarity. For example, (Mohan et al., 2007) proposes a gradient-domain processing technique to adjust the softness of the shadows without introducing any artifacts. (Guo et al., 2012) uses a region-based approach that predicts relative illumination conditions between segmented regions to distinguish shadow regions and relight each pixel. (Zhang et al., 2015) extends this region-based approach and constructs a novel illumination recovering operator to effectively remove the shadows and restore the detailed texture information based on the texture similarity between the shadow and non-shadow patches. Although well-designed, these methods have been surpassed by deep learning-based shadow removal methods recently (Wan et al., 2022; Fu et al., 2021c; Zhu et al., 2022a; Inoue and Yamasaki, 2020; Qu et al., 2017; Le and Samaras, 2019, 2020; Hu et al., 2019a; Wei et al., 2019; Zhang et al., 2020; Zhu et al., 2022b; Li et al., 2023; Gao et al., 2022; Abiko and Ikehara, 2022). Specifically, (Qu et al., 2017) is the pioneering work that uses an end-to-end network to tackle shadow detection and shadow removal problems by extracting multi-context features from the global and local regions. Since then, a large number of interesting deep learning-based methods have been proposed by focusing on different problem aspects. (Fu et al., 2021c) reformulates the shadow removal task as an exposure problem and employs a neural network to predict the exposure parameters to get shadow-free images. (Zhu et al., 2022a) takes into account the auxiliary supervision of shadow generation in the shadow removal procedure and proposes a unified network to perform shadow removal and shadow generation. (Wan et al., 2022) explicitly considers the style consistency between shadow and non-shadow regions after shadow removal and proposes a style-guided shadow removal network. Although these methods achieve promising results, the problem of lacking large-scale paired training data becomes a bottleneck that limits their performance. To alleviate this problem, (Inoue and Yamasaki, 2020) designs a pipeline to generate a large-scale synthetic shadow dataset to improve the shadow removal performance. While the unsupervised or weekly supervised methods are also introduced (Liu et al., 2021b, a; Hu et al., 2019b; Jin et al., 2021; He et al., 2021) to train the deep neural network with unpaired data.

Iterative networks are extensively employed in machine learning-based image processing tasks (Yan et al., 2022; Zhou et al., 2022; Li et al., 2022b; Hang et al., 2019; Saharia et al., 2022; Wang and Wang, 2022; Zhan and Lu, 2019; Yu et al., 2019; Li et al., 2020) to recursively and gradually improve the quality of predictions. For example, (Yan et al., 2022) uses a recurrent image-guided network to address challenges in depth prediction, where the recurrent is applied to both the image guidance branch and the depth generation branch to gradually and sufficiently recover depth values. (Zhou et al., 2022) introduces an edge-guided recurrent positioning network for predicting salient objects in optical remote sensing images. The proposed approach can sharpen the predicted positions by utilizing the effective edge information and recurrently calibrating them during the prediction process. (Hang et al., 2019) introduces a cascaded recurrent neural network that utilizes gated recurrent units to effectively explore the redundant and complementary information present in hyperspectral images. (Saharia et al., 2022) performs image Super-Resolution via Repeated Refinement, which employs a stochastic iterative denoising process to improve image super-resolution performance. (Zhan and Lu, 2019) introduces an end-to-end trainable scene text recognition system that utilizes an iterative rectification framework to address the problem of perspective distortion and text line curvature. (Yu et al., 2019) introduces a deep iterative down-up convolutional neural network for image denoising that employs a resolution-adaptive approach by iteratively reducing and increasing the resolution at the feature level. (Li et al., 2020) presents a recurrent feature reasoning (RFR) network for single-image inpainting that iteratively predicts hole boundaries at the feature level of a convolutional neural network, which then serves as cues for subsequent inference. Inspired by the success of these iterative networks, in this paper, we employ an iterative de-shadow branch (IDB) to gradually improve the performance of shadow removal.

To begin with, we define mutual interference as a phenomenon in which the performance of a shadow removal network increases in the shadow restoration task but decreases in the identical mapping task or vice versa. To uncover this phenomenon, we build a baseline encoder-decoder shadow removal network and evaluate its performance on the ISTD+ dataset(Le and Samaras, 2019) via root mean squared error (RMSE) every 10000 iterations during training. Specifically, given the shadow removal network trained at $t$th iteration, we can calculate the RMSEs at both shadow and non-shadow regions of testing images. Then, we can draw plots along the iteration indexes for both shadow and non-shadow regions (i.e., the red plot and green plot in Fig. 3 (a)). Then, we say that mutual interference occurs if the RMSE variation between two neighboring iterations at the shadow region is different from the one at the non-shadow region. We define the mutual interference ratio as the number of times mutual interference occurs during the training procedure. As depicted in Fig. 3(c), we see that the baseline encoder-decoder network has a high mutual interference ratio (i.e., over 30 times within 100 counted iterations), which illustrates that a shared deep neural network can hardly cover the two distinct tasks at the same time.

To analyze the potential advantages of identical mapping, we conduct two additional experiments using the same encoder-decoder network. For Exp1, the input is the shadow image, and the network is trained to remove shadows from that. While for Exp2, we aim to have the reconstructed result identical to the input shadow images (i.e., identical mapping). For both experiments, we evaluate the restoration quality in the non-shadow regions. Surprisingly, we observed that Exp2 had a significant advantage over Exp1. As shown in Fig. 4 (a), in the non-shadow regions, Exp2 has a much lower RMSE than Exp1 throughout the entire training procedure. To further explore the potential functionality of identical mapping, we used the Information in the Weight (IIW) (Wang et al., 2021) technique to analyze the training procedure of Exp1 and Exp2. A lower IIW means the network has a higher generalization to different non-shadow scenes. The results are displayed in Fig. 4 (b). We observe that Exp2 substantially improved the model’s generalization (i.e., lower IIW) in the non-shadow regions compared to Exp1, which demonstrates the efficacy of identical mapping in reconstructing non-shadow regions.

To investigate the effectiveness of the iterative network, we conduct a series of experiments using the same encoder-decoder structure in Sec. 3.1. Specifically, we feed the decoder’s feature back into the encoder at various times and evaluate the restoration quality in both shadow and non-shadow regions. As depicted in Fig. 4(c), both the encoder-decoder structure and our proposed method show significant improvement in the restoration quality of shadow regions with more iterations (see the red line). However, we notice that for the encoder-decoder structure, the restoration quality in non-shadow regions decreases as the number of iterations increases (see the square green line). In contrast, with our method, the restoration quality in non-shadow regions remains nearly unchanged even with increasing iterations (see the dotted green line). In addition, we also calculate and visualize the $\mathbf{L}_{1}$ differences between the ground truth and reconstructed results obtained by the encoder-decoder structure at different iterations. The results are shown in Fig. 5, where green arrows highlight the differences in shadow regions, and red arrows highlight the differences in non-shadow regions. With a single iteration, the restoration quality is favorable in non-shadow regions but subpar in shadow regions, while the opposite trend is observed with two iterations.

Overall, the above experiments present the necessity of decoupling the shadow removal task into two distinct tasks and the effectiveness of the combination of identical mapping and iterative shadow removal.

The proposed method consists of three components: an identical mapping branch (IMB) (see Sec. 4.2), an iterative de-shadow branch (IDB) (see Sec. 4.3), and a smart aggregation block (SAB) (see Sec. 4.4). Given a shadow image, we decouple the shadow removal into two distinct tasks: restoring shadow regions to their shadow-free counterparts and identical mapping for non-shadow regions. We use IMB to handle the identical mapping task and use IDB to handle the shadow restoration task. The SAB is designed to adaptive integrate features from both IMB and IDB. To prove the advantage of our method, we conduct the same experiment as discussed in Sec. 3.1 by using our method. As shown in Fig. 3(b), for both tasks, our method exhibits less fluctuation in terms of RMSE compared to the encoder-decoder structure during training. Besides, our method also demonstrates a lower mutual interference ratio, as shown in Fig. 3(c).

We propose the identical mapping branch (IMB) to reconstruct the input image via an encoder-decoder network, which can benefit the restoration of non-shadow regions. Given a shadow image $\mathbf{I}$, the objective of IMB is to reconstruct $\hat{\mathbf{I}}$, where $\hat{\mathbf{I}}$ should be identical to $\mathbf{I}$. This procedure can be represented by

where $\phi(\cdot)$ denote the IMB. Let $\mathbf{F}_{l}$ denote the feature extracted from the $l$th convolution layer of $\phi(\cdot)$. $\mathbf{F}_{l}$ can be formalized as

where $\phi_{l}(\cdot)$ denotes the $l$th convolution layer of $\phi(\cdot)$. After training the IMB, we freeze its parameters and rely solely on the multi-scale features, i.e. $\mathbf{F}_{l}$, to guide the iterative de-shadow branch (IDB).

The iterative de-shadow branch (IDB) is responsible for progressively transferring the information from the non-shadow regions to the shadow regions in an iterative manner to facilitate the process of shadow removal by utilizing the multi-scale features provided by IMB. Let $\psi(\cdot)$ denotes the IDB and $\hat{\mathbf{F}}_{l}$ denote the feature extracted from the $l$th convolution layer of $\psi(\cdot)$. $\hat{\mathbf{F}}_{l}$ can be formalized as

where $\mathbf{Cat}$ means channel-wise concatenation, $\psi_{l}(\cdot)$ denotes the $l$th convolution layer of $\psi(\cdot)$ and $\mathbf{M}$ denotes the corresponding binary shadow mask of the shadow image $\mathbf{I}$. The shadow and non-shadow regions are annotated by $\mathbf{1}$ and $\mathbf{0}$ respectively in $\mathbf{M}$. Then we aggregate $\mathbf{F}_{l}$ and $\hat{\mathbf{F}}_{l}$ in an adaptive manner at the multi-scale features level (i.e. after the first, third, and second-to-last convolution layers of $\mathbf{\psi(\cdot)}$ as illustrated in Fig. 6). The procedure of the aggregation can be represented as

where $\hat{\mathbf{F}}_{l}^{{}^{\prime}}$ denotes the adaptive aggregated feature which will be used as the input of the next convolution layer of $\psi(\cdot)$, i.e. $\psi_{l+1}(\cdot)$, and $\mathbf{SAB}$ denotes the smart aggregation block (see Sec. 4.4).

Instead of directly concatenating the features extracted from the IMB and IDB, we propose to aggregate them in an adaptive manner. Specifically, we utilize a convolutional layer with a kernel size of 3x3 followed by a sigmoid activation function to estimate the adaptive aggregation weights, which can be represented as

where $\mathbf{W}_{l},\hat{\mathbf{W}}_{l}$ denote the corresponding aggregation weights of $\mathbf{F}_{l}$, $\hat{\mathbf{F}}_{l}$ respectively. Since the IMB is frozen, $\mathbf{F}_{l}$ remains constant throughout the iterative procedure of IDB. Therefore, we utilize only $\hat{\mathbf{F}}_{l}$ to predict the aggregation weights. The whole aggregation procedure can be formalized as

where $\odot$ denotes the element-wise multiplication, $\text{Conv}_{\text{agg.}}$ denotes a convolution layer with a kernel size of 3x3, $\hat{\mathbf{M}}_{l}$ denotes the generated soft shadow mask which is obtained by applying average pooling along the channels of $[\mathbf{W}_{l},\hat{\mathbf{W}}_{l}]$. As shown in Fig. 7, the soft shadow masks (see (c)-(e)) obtained from the procedure of aggregation are capable of accurately capturing the shadow regions, in contrast to the binary shadow mask (see (b)) provided by the shadow removal dataset, i.e. SRD dataset(Qu et al., 2017), which can not capture the shadow details, especially for the regions along the shadow boundary.

Network architectures. Following (Li et al., 2022a; Guo et al., 2021) the identical mapping branch $\mathbf{\phi(\cdot)}$ and the iterative de-shadow branch $\mathbf{\psi(\cdot)}$ employ a similar encoder-decoder architecture with different input and output, as shown in Table 1. Theoretically, the smart aggregation block can be added after each convolution layer of $\mathbf{\psi(\cdot)}$ to maximize its potential impact. However, to optimize computation efficiency, in our experiments, we selectively add the smart aggregation block after the first, third, and second-to-last layers of $\mathbf{\psi(\cdot)}$ to balance the computation and performance.

Loss functions. Following the previous shadow removal method (Fu et al., 2021c), we only employ the $\mathbf{L}_{1}$ loss during the training process. Specifically, we first train the identical Mapping branch $\phi(\cdot)$ with the objective function

Then we freeze $\phi(\cdot)$ and train the iterative de-shadow branch $\psi(\cdot)$ with the same objective function

where $\overline{\mathbf{I}}$ and $\mathbf{I}^{*}$ denote the de-shadowed result and the corresponding ground truth shadow-free image, respectively.

Training details. We adopt a two-step training strategy. Firstly, we exclusively utilize shadow images to train the identical mapping branch $\phi(\cdot)$ for 500,000 iterations, employing a batch size of 8. Subsequently, we freeze $\phi(\cdot)$ and utilize paired shadow & shadow-free images to train the iterative de-shadow branch $\psi(\cdot)$ for 150,000 iterations with the same batch size. Following (Fu et al., 2021c), we resize the input shadow image to 256x256 resolution. Both branches are optimized using the Adam optimizer with a learning rate of 0.00005. All experiments are conducted on the Linux server equipped with two NVIDIA Tesla V100 GPUs.

Datasets. Following the previous shadow removal method (Wan et al., 2022), we conduct experiments on two widely used shadow removal datasets i.e. ISTD+ (Le and Samaras, 2019) and SRD (Qu et al., 2017). The ISTD+ dataset consists of 1330 triplets for training and 540 triplets for testing. We use the provided ground truth masks directly during the training procedure. While in the evaluation step, we follow the previous method (Fu et al., 2021c) and use Ostu’s algorithm to detect the corresponding shadow masks. The SRD dataset contains 2680 paired shadow and shadow-free images for training and 408 paired shadow and shadow-free images for testing. Because the SRD does not provide the corresponding shadow masks, we use the shadow masks provided by DHAN(Cun et al., 2020) for both training and evaluation steps.

Metrics. We adopt a comprehensive evaluation approach for assessing the performance of our proposed method. Firstly, we calculate the root mean square error (RMSE) in the LAB color space. Furthermore, following the previous approaches (Zhu et al., 2022a; Wan et al., 2022), we employ the commonly used image quality evaluation metrics, i.e. peak signal-to-noise ratio (PSNR) (Johnson et al., 2016), structural similarity index (SSIM), and learned perceptual image patch similarity (LPIPs) (Zhang et al., 2018a). This allows us to thoroughly evaluate the restoration quality of our proposed method from multiple perspectives.

Baselines. We conduct comprehensive comparisons with previous state-of-the-art shadow removal algorithms, including SP+M-Net (Le and Samaras, 2019), Param+M+D-Net (Le and Samaras, 2020), Fu et al. (Fu et al., 2021c), LG-ShadowNet (Liu et al., 2021a), DC-ShadowNet (Jin et al., 2021), G2R-ShadowNet (Liu et al., 2021b), BMNet (Zhu et al., 2022a), and SG-ShadowNet (Wan et al., 2022) on the ISTD+ dataset. Additionally, we compare with DSC (Hu et al., 2019a), DHAN (Cun et al., 2020), Fu et al. (Fu et al., 2021c), DC-ShadowNet (Jin et al., 2021), BMNet (Zhu et al., 2022a), and SG-ShadowNet (Wan et al., 2022) on the SRD dataset.

Quantitative comparison. To validate the effectiveness of our proposed method, we first conduct a comprehensive comparison with recent state-of-the-art methods on the ISTD+ datasets. The results, as depicted in Table 2, clearly demonstrate our method’s superiority in terms of reconstruction quality, as evaluated by multiple metrics, including RMSE, PSNR, SSIM, and LPIPS. Specifically, for the comparison at the whole image level, our method outperforms all the competitors. Compared to Fu et al.(Fu et al., 2021c), our method achieves a reduction of 20.50% in RMSE and 68.38% in LPIPS, as well as an increase of 15.32% in PSNR and 14.21% in SSIM. Similarly, compared to Param+M+D-Net(Le and Samaras, 2020), our method demonstrates superior performance with a reduction of 15.92% in RMSE and 30.30% in LPIPS, as well as an increase of 12.68% in PSNR and 1.89% in SSIM. For the comparison in the shadow regions, our method also outperforms other methods. Compared to Fu et al.(Fu et al., 2021c), our method achieves a reduction of 9.80% in RMSE, as well as an increase of 5.14% in PSNR and 1.30% in SSIM. When compared to Param+M+D-Net(Le and Samaras, 2020), our method achieves a reduction of 38.87% in RMSE, as well as an increase of 13.96% in PSNR and 0.47% in SSIM. For the comparison in the non-shadow regions, our method continues to outperform other methods. Compared to Fu et al.(Fu et al., 2021c), our method achieves a reduction of 24.12% in RMSE, as well as an increase of 19.45% in PSNR and 11.54% in SSIM. Additionally, when compared to Param+M+D-Net(Le and Samaras, 2020), our method achieves a reduction of 1.06% in RMSE, as well as an increase of 7.89% in PSNR and 0.43% in SSIM.

To further substantiate the effectiveness of our proposed method, we conducted additional comparison experiments on the SRD dataset. The results, as presented in Table 3, demonstrate the superiority of our method over other state-of-the-art shadow removal approaches. Our method exhibits a significant margin of improvement across all evaluation metrics. Specifically, for the comparison at the whole image level, our method outperforms all the competitors. Compared to DHAN(Cun et al., 2020), our method achieves a reduction of 22.20% in RMSE and 9.09% in LPIPS, as well as an increase of 9.46% in PSNR and 1.47% in SSIM. Compared to BMNet(Zhu et al., 2022a), our method demonstrates superior performance with a reduction of 14.39% in RMSE and 11.87% in LPIPS, as well as an increase of 5.30% in PSNR and 0.41% in SSIM. For the comparison in the shadow regions, our method also outperforms other methods. Compared to DHAN(Cun et al., 2020), our method achieves a reduction of 24.44% in RMSE, as well as an increase of 7.15% in PSNR and 0.41% in SSIM. When compared to BMNet(Zhu et al., 2022a), our method achieves a reduction of 15.90% in RMSE, as well as an increase of 6.12% in PSNR and 0.43% in SSIM. For the comparison in the non-shadow regions, our method continues to outperform other methods. Compared to DHAN(Cun et al., 2020), our method achieves a reduction of 20.31% in RMSE, as well as an increase of 9.22% in PSNR and 0.93% in SSIM. Additionally, when compared to BMNet(Zhu et al., 2022a), our method achieves a reduction of 13.13% in RMSE, as well as an increase of 2.65% in PSNR and 0.04% in SSIM. These comparison results unequivocally support the effectiveness of our proposed method and its superiority over the state-of-the-art methods in relation to reconstruction quality in both the shadow regions and non-shadow regions.

Qualitative comparison. We compared our visualized results with other state-of-the-art shadow removal methods on both ISTD+ and SRD datasets. As shown in Fig. 8, our method consistently outperforms the competitors in two aspects: ❶ Our reconstructed results exhibit superior color consistency. In particular, for case 2 and case 4, our method produces color-consistent results where the shadow regions and non-shadow regions are nearly indistinguishable from the human eye. In contrast, the competitors’ results show obvious color inconsistency. ❷ The mask boundary in our reconstructed results is smoother and more seamless. For case 1, the mask boundary in the competitors’ results is clearly visible, whereas it is imperceptible in our method. For case 3, the competitors’ results exhibit ghosting artifacts around the mask boundary, while our result does not show any artifacts around the mask boundary.

In this section, we conduct comprehensive ablation experiments to validate each part of the proposed method, and the results are displayed in Table 4.