Joint Depth Estimation and Mixture of Rain Removal From a Single Image

Rain streak removal. Conventional rain streak removal efforts resort to exploiting hand-crafted priors based on image statistics to restore the rainy images [4, 5, 6, 23, 24]. Kang et al. [4] employ dictionary learning and sparse coding to decompose the rainy images into different components and then reconstruct the clean images. Chen et al. [5] develop a generalized low-rank appearance model to remove rain streaks from images. Similarly, Chang et al. [23] exploit a low-rank image decomposition model to restore the rain-free images from their rainy versions. Zhu et al. [24] propose a joint optimization framework with three image priors for rain streak removal from rainy images.

Recently, learning-based approaches have demonstrated their superiority for rain streak removal [9, 10, 25, 26, 27]. Yang et al. [9] develop a multi-task framework to detect and remove rain streaks jointly. Zhang et al. [10] create a density-aware image deraining network to simultaneously predict rain density and remove rain streaks. Li et al. [25] develop a recurrent squeeze-and-excitation contextual dilated network for single image deraining and achieve very promising results. Zhang et al. [26] exploit a conditional GAN-based model with additional regularization for rain streaks removal. Ahn et al. [27] develop a two-step rain removal method that first estimates the rain density and rain streak intensity and then remove them.

Raindrop removal. Since most rain streak removal algorithms cannot be directly applied to raindrop removal, many approaches have been proposed for raindrop detection and removal [28, 29, 12, 30, 13, 14]. Kurihata et al. [28] employ the well-known PCA to learn the shape of raindrops and attempt to match the rainy regions. Roser et al. [29] propose an algorithm for monocular raindrop detection in single images. Eigen et al. [31] develop a convolutional network to remove raindrops from a single image. Latter, You et al. [32] exploit Spatio-temporal information for video raindrop removal. More recently, Qian et al. [12] propose an attentive GAN-based network for single-image raindrop removal. Quan et al. [30] create a CNN-based network to remove raindrops by using shape-driven attention and channel re-calibration. Yan et al. [14] propose a two-stage video-based raindrop removal approach that first employs a single image module to produce the initial clean results and then refine them by using temporal constraints.

Very recently, several image deraining approaches [15, 16, 33, 17, 18] have noticed the presence of more than one type of rainwater artifact in rainy images and attempted to remove them in one go. Hu et al. [15] first establish a new dataset with rain streaks and rainy haze, and then develop an end-to-end model to remove them simultaneously. Guo et al. [16] propose an integrated multi-task framework to handle the joint raindrop and haze removal problem by combining the atmospheric scattering model and deep neural network. Zhang et al. [33] create a dual branch neural network for removing both rain streaks and raindrops. Quan et al. [17] propose an effective cascaded network for removing raindrops and rain streaks simultaneously via using the neural architecture search algorithm. Shen et al. [18] develop a multi-branch attention GAN-based framework to simultaneously remove rain streaks, raindrops, and rainy haze, which is the only study that considers the MOR effect among existing rain removal algorithms. However, these methods either ignore the MOR effect or do not exploit the depth information of the scene, which can be regarded as important prior knowledge to guide the network in removing different types of rainwater artifacts, thus limiting their rain removal capacity in these complex scenarios.

Rain Streak Model. We adopt the widely used rain model [34] to define a rain streak degraded image $R_{s}$ as a superposition of the clean background $B$ and the accumulated rain streak layer $S$, and the observed $R_{s}(x)$ at pixel $x$ can be expressed as follows:

where

In the above equations, $S_{\text{pattern }}(x)\in[0,1]$ denotes the intensity of uniformly-distributed rain streaks in the image; $\odot$ refers to the pixel-wise multiplication; and $t_{r}(x)$ denotes the intensity map of rain streak, which depends on the value of the scene depth $d(x)$ and can be expressed as:

where $\alpha$ denotes the attenuation coefficient, which controls the intensity of the rain streaks. Specifically, objects close to the camera lens are mainly affected by the raindrops and the rain streak intensity is smaller here. As the objects are far from the camera, rain streak intensity gradually increases to its maximum intensity and then falls to zero with the increase of $d(x)$. At this point, the image is mainly degraded by the rainy haze.

Raindrop Model. A raindrop degraded image $R_{d}$ can be modeled as a combination of the clean background $B$ as well as the impact of the raindrops $D$, and the observed $R_{d}(x)$ can be expressed as [12]:

where $M(x)\in{\{1,0\}}$ denotes whether the pixel $x$ is corrupted by raindrops or not, and $D(x)\in[0,1]$ refers to the raindrop layer.

Rainy Haze Model. Unlike rain streaks, the visual intensity of rainy haze increases exponentially with the scene depth $d(x)$, we adopt the standard atmosphere scattering model [35] to simulate the image degradation process caused by rainy haze, and the observed hazy image $R_{h}(x)$ at pixel $x$ can be formulated as:

where

In the above equations, $A$ represents the global atmosphere light, $t(x)$ denotes the atmospheric transmission, and $\beta$ denotes the atmosphere scattering parameter. Note that a larger $\beta$ suggests a thicker rainy haze and vice versa.

MOR Model. In rainy weather, especially under heavy rain conditions, raindrops, rain streaks, and rainy haze often appear simultaneously during outdoor image capture. Therefore, a MOR image $R_{mor}(x)$ can be formulated as follows:

where $\hat{A}$ refers to the light conditions and color cast from active light sources. Unlike the fixed global atmosphere light $A$, $\hat{A}$ enables us to model the nighttime MOR images, which expands the application scenarios of the MOR imaging model. Note that rain streaks and rainy haze may also change the illumination conditions, which in turn affects the transparency of raindrops during image capture. That is, these three rainwater artifacts interact with each other to form the final MOR effect, which is a quite complex and entangled process.

As observed in Eq. 7, scene depth plays a crucial role in the degradation process of MOR images. To fully exploit the informative guidance provided by the scene depth, we present a highly effective and unified deraining paradigm called DEMore-Net. As illustrated in Fig. 2, DEMore-Net consists of two principal modules: the MOR removal sub-network and the depth estimation sub-network. In this way, the joint learning framework enables the MOR removal and depth estimation tasks to collaborate and benefit from each other. DEMore-Net operates in an end-to-end fashion, taking a MOR photo as input and producing the corresponding MOR-free image and depth map as outputs. To produce clearer and more realistic MOR-free images, we introduce the adversarial training strategy in our MOR removal sub-network.

The pipeline of DEMore-Net can be described in detail as follows: First, the input MOR image undergoes preliminary image restoration and depth estimation tasks through both the MOR removal sub-network and the depth estimation sub-network. Then, the predicted depth map is utilized by the MOR removal sub-network to provide guidance for better restoring the MOR image. To achieve this, a depth-guided non-local (DGNL) [36] module is employed to fuse the depth information with the coarsely restored features, resulting in a refined MOR-free image. Finally, a discriminator is employed to evaluate the authenticity of the restored MOR-free image, determining whether it is a real clean image or a fake image generated by the MOR removal network. This allows for the generation of more realistic images.

Overall, DEMore-Net comprises three components: 1) a depth estimation sub-network that employs an encoder-decoder network with multiple feature enhancement modules to produce the depth map; 2) a MOR removal sub-network that produces the MOR-free image using the estimated depth map as additional guidance; and 3) a discriminator that utilizes adversarial training strategies to improve the quality of the restored MOR-free images.

Depth Estimation Network. Inspired by the success of encoder-decoder architecture and ResNet in image restoration field [12, 37, 38], we exploit an encoder-decoder network as our depth estimation network and introduce an up-to-date multi-scale feature extraction network (self-calibrated convolutions [22]) to further improve the prediction accuracy of depth maps. Self-calibrated convolution module is an improved convolutional network, which can construct long-range spatial and inter-channel dependencies for each spatial location in the feature space, thereby enriching the output features and helping CNNs generate more discriminative representations. Herein, we employ self-calibrated convolutions as our feature enhancement module to improve the depth estimation network’s performance.

As depicted in Fig. 2, the depth estimation network adopts 11 convolutional blocks to extract features from the input MOR image and then output the depth map in an end-to-end manner. For the first 10 convolution blocks, each block contains a convolutional operation, a residual block, a hybrid normalization, a Scaled Exponential Linear Unit (SELU), and a self-calibrated convolution module. Note that all the normalization methods used in this work are the proposed hybrid normalization block, which will be described in the next section. After extracting the image features, we adopt a convolutional operation, and a sigmoid activation function (i.e., $11^{th}$ convolutional block) to output the final depth map in a supervised manner. Here, we adopt the simple $L_{1}$ loss to train the depth estimation network, which can be expressed as:

where $D(x)$ is the predicted depth map, $\hat{y}$ is the ground truth depth map; $W$ and $H$ denote the width and height of the depth map, respectively.

MOR Removal Network. Considering that adversarial training strategies can encourage the generated images close to the clean images (i.e., ground truth) and make them more realistic, we built the MOR removal sub-network based on a standard generative adversarial network. As demonstrated in Fig. 2, the proposed generator consists of a convolution head to extract image features, a DenseNet-shape body to extract deeper features and restore the clean features simultaneously, and a 1$\times$1 convolution tail to convert channel numbers and output the restored clean image features. The DenseNet-shape body is a stack of dilated residual blocks (DRB) with skip-connections, which contains two 3$\times$3 dilated convolutions and a non-linear ReLU layer. Although applying dilated convolutions can enlarge the fields of view of the model without adding additional computation, it may easily cause the well-known gridding problem. To address this issue, we follow [36] to combine dilated convolutions with conventional convolutions and set the dilation rates of these 11 DRBs to $\left\{1,1,2,2,4,8,4,2,2,1,1\right\}$. Finally, the depth-guided non-local (DGNL) module is employed to fuse the predicted clean image features with the depth map, thus helping the model to better remove different types of rainwater according to the distance.

The function of the discriminator is to determine whether an input image is a real MOR-free image or a fake image produced by the restoration network, thereby encouraging the generator to generate high-quality images. For adversarial training, we adopt 3 convolutional layers and a linear layer to construct our discriminator. Considering that the Least-Squares GAN (LSGAN) loss is more effective in improving the training stability than vanilla GAN loss, we employ the LSGAN to train our MOR removal network. The definition of adversarial loss can be formulated as:

where $y$ refers to the ground truth images and $G(x)$ represents the restored MOR-free images.

Depth-Guided Non-Local Module. The depth-guided non-local (DGNL) module [36] is an effective feature fusion network, which is employed to build the relationship between each pair of spatial locations based on the depth map and hence enriches the output features. In addition, as mentioned previously, both the type and visual intensity of rainwater artifacts in MOR images depend on scene depth, as does the process of MOR removal. Therefore, we adopt the DGNL module to fuse the depth information with the restored clean image features to help our DEMore-Net better recover the MOR-free images.

As depicted in Fig. 3, the DGNL module is extended based on the Non-local neural network [39] and aggregated the depth information as additional guidance to enrich the final output features. In light of this, we regard the depth information as a piece of prior knowledge and leverage the DGNL module to fuse the depth map with restored image features, thus helping DEMore-Net to remove different types of rainwater in a depth-guided manner. After fusing the depth and image features, we employ two convolutional operations to enlarge the feature map and output the final MOR-free images.

Normalization has become a crucial component in various high-level vision tasks but is rarely used in low-level vision tasks, especially Batch Normalization (BN). In fact, the work [40] has demonstrated that Instance Normalization (IN) can learn features that are not affected by appearance changes, such as brightness, colors, and styles, whilst BN is important to preserve content-related features. Both BN and IN are essential as they can promote each other to benefit the vision tasks. Inspired by this, we carefully integrate BN and IN as building blocks and develop a Hybrid Normalization Block (HNB) to advance the network performance in MOR removal tasks. Specifically, HNB leverages both BN and IN in one normalization operation, thus improving its learning and generalization capacities. Also, the additional parameters and computational costs introduced by HNB can be ignored.

As depicted in Fig. 4, given a feature map $F_{in}\in\mathbb{R}^{C_{in}\times H\times W}$, a 3$\times$3 convolution is first adopted to produce the intermediate feature map $F_{mid}\in\mathbb{R}^{C_{out}\times H\times W}$. Then, HNB divides the input $F_{mid}$ into two parts, namely, $F_{mid_{1}}$ and $F_{mid_{2}}$ ($F_{\operatorname{mid}_{1}}\&F_{\text{mid }_{2}}\in\mathbb{R}^{C_{\text{out }}/2\times H\times W}$). The first part $F_{mid_{1}}$ is normalized by BN with learnable parameters and the second part $F_{mid_{2}}$ is normalized by IN, and then they are concatenated in the channel dimension. HNB employs BN on half of the channels and IN on the other half, so that content-related features can be preserved while learning features that are invariant to appearance changes. After concatenating the two intermediate features, the output feature map $F_{out}\in\mathbb{R}^{C_{out}\times H\times W}$ is obtained by passing features to a SELU layer.

To train the proposed DEMore-Net more effectively, we take into account all the positive factors that can improve the quality of the restored MOR-free images. Besides depth estimation loss and adversarial loss, we also adopt the reconstruction loss, Dark Channel (DC) loss [41] and Total Variation (TV) loss [42] to train our DEMore-Net. The total loss function can be expressed as:

where $\lambda_{i},i=1,2,\cdots 3$, are loss weights, and we set $\lambda_{1}=0.1$, $\lambda_{2}=0.01$, and $\lambda_{3}=0.01$ in our experiments.

Reconstruction Loss. The reconstruction loss in this work is a combination of multi-scale structural similarity (MS-SSIM) loss [43] and $L_{1}$ loss, which is formulated as follows:

where

In the above equations, $\alpha$ is the hyperparameter, and we empirically set $\alpha=0.1$ in our experiments. $M$ is the total number of the scales and is set to 5 according to [43]; $\mu_{G(x)}$, $\mu_{y}$ and $\sigma_{G(x)}$, $\sigma_{y}$ represent the mean and standard deviations of $G(x)$ and $y$, $\sigma_{G(x)y}$ refers to their covariance. $\beta_{m}$ and $\gamma_{m}$ denote the relative importance of the two components, both are simply set to 1 in our experiments. Two small constants $C_{1}=0.0001$ and $C_{2}=0.0009$ are added to avoid the unstable case of division by zero.

Dark Channel Loss. In light of the effectiveness of the dark channel prior [44] in the image dehazing field, we apply the Dark Channel loss to help the DEMore-Net deal with the rainy haze in the MOR images, which is formulated as:

where $\mathbf{D}_{G\left(x\right)}$ represents the vector form of the dark channel of the restored MOR-free images $G\left(x\right)$. Since dark channel operation is a highly non-convex and non-linear term, it cannot be directly embedded into the learning networks. We adopt the look-up table scheme (following [41]) to implement the forward and backward steps of the dark channel operation.

Total Variation Loss. We adopt the total variation loss [42] to reduce the difference between adjacent pixel values to make the restored images more natural, which can be expressed as:

where $\nabla_{h}$ and $\nabla_{v}$ denote the horizontal and vertical differential operation matrices, respectively.

Dataset. To the best of our knowledge, the RainCityscapes++ dataset [18] stands as the sole synthetic MOR benchmark to encompass raindrops, rain streaks, and rainy haze in one consolidated dataset. It comprises a total of 8580 synthetic MOR images, out of which 7580 images serve for training and 1000 images for testing. Henceforth, we adopt the RainCityscapes++ dataset to train and evaluate the performance of our proposed DEMore-Net.

Training Details. DEMore-Net is implemented by PyTorch 1.7 on a system with an Intel Core i9-12900KF CPU, 32 GB RAM, and an NVIDIA GeForce RTX 3090 GPU. We adopt the Adam optimizer with a batch size of 2 to train our model, where the momentum parameters $\beta_{1}$ and $\beta_{2}$ are set to 0.9 and 0.999. The total number of iterations and the initial learning rate are empirically set to $300,000$ and $5\times 10^{-4}$, respectively. Considering that the original image size of the RainCityscapes++ dataset is too large ($2048\times 1024$ pixels), we resize the images to $720\times 480$ pixels for both training and inference.

Evaluation Settings. To quantitatively assess the performance of DEMore-Net, we employ the average Peak Signal to Noise Ratio (PSNR) and Structural Similarity index (SSIM) as the evaluation metrics, which are the most widely used image objective evaluation indexes in the field of image restoration. The proposed DEMore-Net is compared with various SOTA image deraining approaches. These algorithms can be divided into two categories, namely, single-type rain removal methods and multi-type rain removal methods. For single-type rain removal methods, we compare with both raindrop, rain streak, and rainy haze removal approaches, including AttentiveGAN [12], CMFNet [19], MSPFN [45], Syn2Real [20], DGCN [46], MPRNet [37], LD-Net [47], and SGID [21]. For multi-type rain removal methods, we compare with DAF-Net [15], DGNL-Net [36], CCN [17], MBA-RainGAN [18], and the cascade of different types rain removal models (i.e., raindrop+rain streak+rainy haze removal).

Comparison on Synthetic Dataset. For quantitative evaluation, we report the averaged PSNR and SSIM metrics of 14 state-of-the-art dehazing algorithms on the RainCityscapes++ test set in Table I. For a fair comparison, all compared methods are retrained on the RainCityscapes++ training set. As can be seen, since single-type rain removal algorithms can only remove one type of rainwater artifact, they cannot achieve satisfactory results. Surprisingly, cascading combinations of different rain removal methods (raindrop+rain streak+rainy haze) also usually yielded poor results. We argue this is because different restoration methods may introduce additional artifacts or lose details in the image processing process (see Fig. 5), and this process is constantly accumulating, degrading the quality of the final restored images. Compared with these SOTA approaches, the proposed DEMore-Net achieves the best performance with 35.46 PSNR and 0.959 SSIM.

Moreover, Fig. 5 exhibits a visual comparison of synthetic MOR images from the RainCityscapes++ dataset. As observed, Syn2Real cannot completely remove the rainy haze in the image, and some haze remains in the sky area. Although CMFNet and CCN succeed in removing the rain to some extent, they cannot preserve the details of the image well, making the image look a little blurry. CCN+SGID (cascading combination method) seems unable to cope with the intractable MOR problem by combining it with another dehazing algorithm. Compared with the original CCN approach, it introduces additional artifacts and tends to darken the restored images. The deraining results of MBA-RainGAN are pretty good but still cannot completely remove the rainy haze in sky regions. In contrast, our DEMore-Net can produce much clearer and more natural MOR-free images.

Comparison on Real-World MOR Images. To evaluate the effectiveness of DEMore-Net in real-world rainy scenarios, we compare our method with SOTA rain removal approaches on real-world MOR images. Fig. 6 displays 4 real-world rainy images and the deraining results by different algorithms. As observed, Syn2Real cannot produce satisfactory deraining results due to color distortion. For CMFNet and CCN, although they have largely overcome the issue of color distortion, they introduce additional artifacts into the image, making the deraining results look unreal and unnatural. The results of CCN+SGID look very dark, and there is still some remaining rain. MBA-RainGAN can remove most of the rain streaks and raindrops in the image, but still cannot effectively address the rainy haze in some regions. Compared with these SOTA methods, our DEMore-Net produces the most realistic MOR-free with perceptually pleasing and high quality.

To better understand the deraining capacity of DEMore-Net in real-world rainy scenarios, a user study is performed to quantitatively evaluate the deraining results. To be more specific, we first prepare 50 real-world rainy images from an existing real-world rainy dataset (DDN-SIRR [11]) or from the Internet. Then, we apply 5 representative SOTA deraining algorithms and our DEMore-Net to restore these 50 images. Next, 10 participants (5 males/5 females) are recruited and asked to score the deraining results on a scale from 1 (worst) to 5 (best). We exhibit to each participant these 300 derained images in a random order, without telling them the corresponding deraining algorithms. The experimental results are depicted in Table II, demonstrating that our DEMore-Net can cope well with real-world MOR removal tasks.

For quantitative comparison, 4 well-known no-reference image quality assessment indexes are employed to evaluate the performance of real-world MOR removal, including NIQE, SSEQ, BRISQUE, and PI [48]. All these indexes are evaluated on the above-mentioned 50 real-world rainy images. As tabulated in Table III, our DEMore-Net achieves the best performance in terms of NIQE, SSEQ, and PI, showing that the restored MOR-free images produced by our approach are much clearer and more realistic. Furthermore, DEMore-Net also achieves an impressive performance in BRISQUE. In a nutshell, our DEMore-Net wins three of the four indexes, fully validating the superiority of DEMore-Net on real-world rain removal tasks.

The proposed DEMore-Net exhibits superior MOR removal capacity compared to 14 SOTA deraining algorithms. To evaluate the effectiveness of DEMore-Net, an ablation study is performed to validate different components, including the DenseNet structure, self-calibrated convolutions, depth estimation branch, adversarial training strategy, and Hybrid Normalization Block (HNB).

We first adopt 11 dilated residual blocks (DRB) with 2 convolutional operations to construct the base model of the MOR removal network (without depth estimation branch) and then train this model via the aforementioned implementation details. The normalization approach used here is group normalization [49]. Next, different components are gradually added to our base model:

1. 1.

   base model + DenseNet structure $\rightarrow$ $V_{1}$,

2. 2.

   $V_{1}$ + self-calibrated convolutions $\rightarrow$ $V_{2}$,

3. 3.

   $V_{2}$ + depth estimation branch (including DGNL) $\rightarrow$ $V_{3}$,

4. 4.

   $V_{3}$ + adversarial training strategy $\rightarrow$ $V_{4}$,

5. 5.

   $V_{4}$ + Hybrid Normalization Block $\rightarrow$ $V_{5}$ (full model).

All these models are retrained in the same training strategy as before and evaluated on the RainCityscapes++ dataset. The experimental results of these models are depicted in Table IV.

As exhibited in Table IV, each component in our DEMore-Net contributes to MOR removal, especially the well-designed depth estimation branch, which achieves a 4.96dB PSNR and 0.057 SSIM improvement over variant $V_{2}$. Additionally, the proposed HNB has significantly advanced the performance of our network in terms of PSNR, and the use of adversarial training strategy and self-calibrated convolutions improve the model performance in SSIM. The adoption of the DenseNet structure has also improved the deraining performance of the model. As observed, the committee with these five components produced the best MOR removal performance, indicating that the five components can complement each other.

Typically, the accuracy of object detectors drops significantly in rainy conditions due to obvious visibility degradation. To prove that our DEMore-Net can benefit vision-based applications, we adopt a pre-trained YOLOXs [50] detector to detect objects on real-world rainy images and the corresponding deraining results by different algorithms. As depicted in Fig. 7, after removing the rain, the confidence in recognizing objects is greatly improved. Clearly, DEMore-Net outperforms the other image deraining approaches, fully demonstrating the effectiveness of our approach in real-world deraining tasks.

Considering that efficiency is quite crucial for computer vision systems, we evaluate the computational performance of some typical deraining algorithms and report their average running times in Table V. All the methods are implemented on an NVIDIA GeForce RTX 3090 GPU. It can be observed that DEMore-Net takes on average about 0.07$s$ to process a rainy image from the RainCityscapes++ dataset (720$\times$480 pixels). Our DEMore-Net has the second-fastest inference speed among the 13 representative deraining approaches.