A Categorized Reflection Removal Dataset with Diverse Real-world Scenes

Most single image reflection removal methods [5, 40, 37, 38] rely on various assumptions. Considering image gradients, Arvanitopoulos et al. [3] propose the idea of suppressing the reflection, and Yang et al. [38] propose a faster method based on convex optimization. These methods fail to remove sharp reflection. Under the assumption that transmission is in focus, Punnappurath et al. [23] design a method based on dual-pixel camera input. CEILNet [5], Zhang et al. [40], and BDN [37] assume that reflection is out of focus and synthesize images to train their neural networks. CEILNet [5] estimates target edges first and uses it as guidance to predict the transmission layer. Zhang et al. [40] use perceptual and adversarial losses to capture the difference between reflection and transmission. BDN [37] estimates the reflection images, which is then used to estimate the transmission layer. These methods [40, 5, 37] work well when reflection is more defocused than transmission but fail otherwise. For these deep learning based approaches, training data is critical for good performance. To bridge the gap between synthetic and real-world data, Zhang et al. [40] and Wei et al. [32] collected some real-world images for training. However, their images have misalignment issues between transmission and input images, and the dataset size is small. Wei et al. [32] propose to use high-level features that are less sensitive to small misalignment to calculate losses. To obtain more realistic and diverse data, Wen et al. [33] and Ma et al. [20] propose to synthesize data using a deep neural network and achieve better performance and generalization. Kim et al. [11] propose a physics based method to render the reflection and achieve better performance than using synthetic images.

Utilizing multiple images as input provides additional information, which makes it possible to relax some strict assumptions used in prior work. A number of approaches [28, 25, 24, 16, 8, 9, 36, 27, 2, 18] exploit the relative motion between reflection and transmission with multiple images captured with camera movement to remove reflection.

Some other methods may take a sequence of images under specific conditions or camera settings. For example, pairs of flash and no-flash images [1, 13], near-infrared cameras [10], light field cameras  [31], dual-pixel cameras [23], and polarization cameras  [19, 22, 26, 12, 6] can be used.

Different from supervised learning models, Double-DIP [7] separates a mixed image into reflection and transmission layers based on internal self-similarities in multiple superpositions, in an unsupervised fashion.

Although the methods based on multiple images do not rely on strict assumptions for appearances of reflections (e.g., blurry reflection, ghosting effects), they need additional requirements on data [36] or special devices [14], which may prevent them from broader applications. Therefore, in this work, we focus on building a dataset and evaluating single image reflection removal methods.

Wan et al. [29] propose the SIR² benchmark dataset for single image reflection removal. This dataset contains images taken in controlled scenes and in the wild. To solve the misalignment problem, they calibrate the alignment between the mixed image $M$ and the background $B$. The dataset was captured with three glasses of different thicknesses, various combinations of aperture sizes, and different exposure times to improve the diversity in this dataset.

As described by Wan et al. [29], most of the objects in their controlled scene contain only flat objects (postcard) or objects with similar scales (solid objects). However, real-world scenes contains objects at different depths, and the natural environment illumination also varies greatly, while the controlled scenes are mostly captured in an indoor office environment. To address this limitation, 55 pairs of images with ground-truth reflection and transmission are captured in the wild, but 55 pairs are far from large scale. Also, this dataset does not provide a standard split among the training set, the validation set, and the test set.

In order to collect diverse data with perfect alignment in the wild, we adopt the M-R pipeline proposed by Lei et al. [14] where $T$ is obtained by $M-R$. Different from Lei et al. [14] which implements $M-R$ on raw data obtained by a polarization sensor, we use normal cameras that provide the raw data to construct the CDR dataset.

Fig. 4 shows our data collection pipeline. The first step is to have an appropriate glass. Since we do not need to remove the glass to obtain background $B$ as other methods do [29, 32, 40], we can utilize immovable glasses in the real world. As the second step, we use a piece of black cloth behind the glass to block transmission to obtain the reflection $R$. In the end, we remove the cloth to collect the mixed image $M$. Please refer to Fig. 3 for some captured example iamges.

There are some details in real capturing progress:

• •

  To ensure perfect alignment between $M$ and $R$, we use a tripod to fix the camera to take images.

• •

  To ensure the exposure is consistent between $M$ and $R$, all the cameras are set to the manual mode with a fixed camera setting including ISO and exposure time.

• •

  To reduce the noise level in $M$ and $R$ as much as possible, we capture data with a long exposure time and a small ISO.

• •

  The objects in the reflection (not transmission) need be static in both $M$ and $R$ to ensure the perfect alignment in $M-R$.

Fig. 5 shows the overall pipeline of our post-processing step. With raw $M$ and raw $R$, we calculate raw $T$ by $T=M-R$ in the raw data space. Note that $M-R$ should be implemented in the RGB space because the linearity between light intensities and RGB values does not hold, as shown in Fig. 6. In $T=M-R$, negative values may appear due to noise, and they are set to zeros directly. The black level of a camera is added back to ensure its ISP can be applied to $T$. So far, the raw data format of $T$ is the same as $M$ and $R$.

Since most existing reflection removal algorithms adopt RGB images as input, we need to convert raw images from the raw data space to RGB space. However, the camera default ISP is not public. Therefore, we implement our own image signal processing (ISP) pipeline to generate RGB images for $M$, $T$, and $R$ using the same ISP. As for the metadata of $T$, we simply apply the metadata of $M$ to $T$ directly.

After obtaining RGB $T$ images, we need to crop the area of interest for $T$. In the dataset, we eliminate the triplets in the following two cases: (1) If multiple glasses exist behind the covered glass (i.e., glasses exist in the background), the obtained $T$ still has glasses. (2) If there is no glass in $M$, then no black cloth exists to cover the transmission. In this case, the obtained $T$ will equal 0, which contradicts the ground truth.

Different from SIR² that classify images as bright or dark scenes according to absolute intensity, focus, or thickness of glass, we categorize images according to three criteria: relative intensity, smoothness between $R$ and $T$ and the ghosting effect. First, we observe that the impact of reflection is decided by the relative intensity instead of absolute intensity. We calculate the mean intensity ratio for each pair of $R$ and $T$ and categorize them as weak reflection, moderate reflection or strong reflection. Second, we sort the data based on the smoothness of reflection and transmission: BRST (blurry reflection and sharp transmission), SRST (sharp reflection and sharp transmission), BRBT (blurry reflection and blurry transmission). In fact, the BRST type has been generally used in previous work. At last, we pick up the data which has ghosting effect as a class.

One interesting observation is that the performance of most existing single image reflection removal methods is highly related to the type of reflection.

Quantitative results. Table 2 shows the performance of the evaluated methods in terms of PSNR, SSIM, and NCC. We also split data according to smoothness, relative intensity, and ghosting effect of reflection and report separate results to analyze the impact of different image patterns. We find that learning free methods [3, 38] achieve good performance in cases that follow their model assumptions. Although these methods usually rank poorly on average, they can perform quite well when the reflection is blurry, weak, or has ghosting effect.

In the following, we analyze the impact of different factors on the results.

1) Impact of real data. The methods [32, 30] trained on real data achieve better performance on general real-scene data. Since Kim et al. [11] synthesized physically-based data for training, they also achieve good performance on real-world data. These data include SRST data, reflection with moderate intensity or strong intensity. As we can see, these methods [11, 32, 30] are often the first, second, and third best-performing methods.

2) The smoothness of reflection/transmission. The results on different smoothness of reflection and transmission are consistent with previously adopted assumptions. All methods perform relatively well the BRST (blurry reflection and sharp transmission) set. However, when the assumption does not hold (e.g., on SRST set), the performance degrades heavily. This phenomenon occurs to all evaluated algorithms. Note that in our dataset, almost 50% percents of $M$ are SRST.

3) Ghosting effect. Another assumption about reflection is the ghosting effect. Although most deep learning methods do not synthesize ghosting reflection data, they still achieve better performance on such kind of data.

4) Reflection Intensity. The difficulty of reflection removal increases significantly when the intensity of the reflection increases. However, the importance of reflection removal also increases in this case. When reflection is too weak, we even do not need to remove it because it is almost invisible. When the reflection is moderate or strong, the image quality suffers heavily.

Qualitative results. We present qualitative results in Fig. 8 to analyze the results further. The perceptual performance is consistent with the quantitative results on different reflection types.

The performance on SRST is not satisfactory. For SRST, most methods cannot remove most reflection. For BRST, most benchmarked methods can remove the reflection when the reflection is weak. Learning free methods achieve good performance in this case [38, 3]. However, it is quite rare when the reflection is both blurry and weak.

Another problem that occurs commonly is the degradation of image quality as shown in Fig. 8. In some cases, a method may remove the reflection nearly completely, but transmission is modified at the same time.

From the quantitative evaluation and perceptual results, we find that state-of-the-art single image reflection removal methods are still far from perfect, although they have achieved great performance on synthetic data or real-world data in a controlled environment.

From our experiment, we find that evaluation on synthetic data is flawed. The improvement of results on synthetic data or real-world data in a controlled environment cannot represent real improvement. If we want to apply the reflection removal method in our daily life, we should aim to achieve excellent performance on real-world data collected in the wild.

We believe we should relax strong assumptions in reflection removal methods. Strong assumptions may make the method perform well on a certain type of reflection but fail on other types. As analyzed above, most methods can achieve satisfactory results on blurry reflection but have poor performance on sharp reflection. It is definitely a difficult task to distinguish between the sharp reflection and transmission.

It is unclear whether single image reflection removal can benefit from raw images. However, we keep the raw data in our dataset since applying raw images has achieved amazing results on low-level computer vision tasks, including low-light image enhancement  [4], super-resolution [35], image denoising [39] and ISP [21, 34]. We leave the study for raw images on single image reflection removal to future work. To the best of our knowledge, our dataset is the first dataset containing raw images for single image reflection removal.