A Benchmark for Edge-Preserving Image Smoothing

We chose seven state-of-the-art and representative edge-preserving algorithms to construct our dataset, including SD filter [6], $L_{0}$ smoothing [7], Fast Global Smoother (FGS) [8], Tree Filtering [9], Weighted Median Filter (WMF) [10], $L_{1}$ smoothing [12] and Local Laplacian filter (LLF) [11]. The selection considerations of these filters are twofold. The first is the representativeness and impact of the work (e.g., high citations). This consideration ensures that the selected filters are state-of-the-arts. The second consideration is the diversity to leverage the merits of different types of filters. Based on these considerations, we selected 4 global methods [6, 7, 8, 12] and 3 local methods [9, 10, 11]. The global filters explicitly formulate the edge-preserving smoothing process as a global optimization problem, while the local filters apply a weighted function depending on the similarity of features within a local window.

According to the fact that the best smoothing results of different images may come from different algorithms with different parameter settings, we have developed an interactive interface where one can choose the proper edge-preserving smoothing result in two steps:

$\bullet$ Step 1: Given a source image, choose a parameter setting for each algorithm which generates the best smoothing result for that algorithm .

$\bullet$ Step 2: Choose the best one from the best results of the seven algorithms.

Figure 3 shows snapshots of the selection tool. The source image is shown in the top left corner. We pre-defined eight parameter settings for each algorithm, as shown in Table I. These eight settings are selected to cover a wide range of coarse-to-fine smoothing levels. For each algorithm, the smoothing results corresponding to these eight parameter settings are presented on the same screen (Figure 3(a)). A user first chooses the best result produced by different parameters for each method. Afterwards, the seven smoothing results chosen from the previous step, one for each of the seven algorithms, are shown side by side on a pop-out window (Figure 3(b)). The user then chooses the best result from the seven as the final selection for the current source image.

Since the above two steps heavily rely on visual comparisons and multiple images need to be shown on the same screen, we use a 32-inch Truecolor IPS monitor with a high resolution ($3840\times 2160$) for dataset construction.

As is widely acknowledged, the general criterion for basic edge-preserving smoothing is that the visually salient edges of major structures should be preserved while the trivial details should be removed. Subjects are instructed to select good edge-preserving smoothing results from different edge-preserving smoothing (EPS) filters with different parameter settings.

Since humans could have different perceptual comprehension of trivial details and major structures, they might select different smoothed images as their preferred edge-preserving smoothing results. To model such perceptual differences among human users, we formed a subject group of 26 volunteers, all of whom are graduate students in The University of Hong Kong and The Hong Kong Polytechnic University. Each source image in our dataset and its associated smoothing results were randomly assigned to 14 volunteers. That is, on average one volunteer was asked to select proper smoothing images for 267 source images. Most of the volunteers do not have prior experiences on edge-preserving image smoothing. Simple but key instructions were given to the volunteers in order to familiarize them with the nature of edge-preserving smoothing.

$\bullet$ Instruction 1: Strong edges should be preserved and blurry effects at significant edges are extremely undesired.

$\bullet$ Instruction 2: The color of a smoothed image should be as close to the original image as possible.

$\bullet$ Instruction 3: Under instructions 1 and 2, the smoother, the better.

The volunteers were further presented with a few unambiguous examples of edge-preserving smoothing for training. It typically takes a volunteer around 2 minutes to select one final result. In order to minimize the negative effects of visual fatigue, only a single work session up to 60 minutes is allowed in any single day.

In addition, many types of EPS algorithms have been proposed or tailored to meet specific criterion of their targeted applications (e.g., tone mapping). In our proposed benchmark construction, we carefully selected 7 representative EPS algorithms, which are designed for various applications, to create the ground truths. The constructed dataset implicitly captures various good edge-preserving smoothing properties. As a consequence, the benchmark is applicable to a wide range of applications. We will demonstrate the effectiveness of our benchmark in applications of tone mapping and contrast enhancement in Section VI.

The constructed dataset for edge-preserving smoothing contains 500 natural images (400 for training and 100 for testing). As mentioned above, to reduce the bias of subject preference during manual selection, we collected 14 human-selected smoothing results for each image. The entire process lasted for one and half months.

Since each image is finally associated with 14 human-selected smoothing results, there are 7000 choices in total. As shown in Figure 4(a), a large proportion of the choices (3999 choices) are generated by the $L_{1}$ smoothing algorithm. Nevertheless, there are still a considerable number of choices (3001) distributed among the other 6 smoothing algorithms. Figure 4(b) shows the distribution of the parameter choices for the $L_{1}$ smoothing method. This distribution confirms the observation that the proper smoothing result of different images may come from different algorithms with different parameter settings.

To show the consistency of choices of different volunteers on the same source image, we compute the maximum number of repeated choices for each image. Let ${count_{t}(m,p)}$ denote the number of volunteers who chose method $m$ with parameter setting $p$ to compute the proper smoothing result of the $t$-th source image. Note that $\sum_{m=1}^{7}\sum_{p=1}^{8}count_{t}(m,p)=14$. The maximum number of repeated choices for image $t$ is defined as $\max_{m}\max_{p}count_{t}(m,p)$. The distribution of the maximum number of repeated choices across all images is shown in Figure 5. We can see that there are 420 (out of 500) images whose maximum number of repeated choices is greater than or equal to 3. In other words, for 84% of the source images, at least 3 volunteers chose the result from the same algorithm with the same parameter.

The parameter setting of a smoothing algorithm affects its performance. We measure WRMSE and WMAE across the entire testing set of our dataset with different parameter settings for each of the seven chosen algorithms. The minimum WRMSE and WMAE are denoted by $\mbox{WRMSE}^{*}$ and $\mbox{WMAE}^{*}$, respectively, and the optimal parameter setting of each method was determined by greedy search. The process is as follows: set a group of parameter settings for each algorithm; apply them to all testing images; record WRMSE and WMAE. The parameter settings that make the seven chosen methods achieve their $\mbox{WRMSE}^{*}$ are given as follows: SD filter ($\lambda=5$), $L_{0}$ smoothing ($\lambda$=0.02), FGS ($\sigma_{c}=0.025,\lambda=600$), Tree Filtering ($\sigma=0.05,\sigma_{s}=8$), WMF ($\sigma$=30), $L_{1}$ smoothing ($\alpha=20,\theta=50$), LLF ($\sigma_{r}=0.4,\alpha=2$). The parameter settings that make the seven methods achieve their $\mbox{WMAE}^{*}$ are given as follows: SD filter ($\lambda=5$), $L_{0}$ smoothing ($\lambda$=0.01), FGS ($\sigma_{c}=0.025,\lambda=600$), Tree Filtering ($\sigma=0.05,\sigma_{s}=8$), WMF ($\sigma$=50), $L_{1}$ smoothing ($\alpha=20,\theta=50$), LLF ($\sigma_{r}=0.2,\alpha=4$). Additional parameters are set to default values suggested by original authors.

From Table II, we can see that the $L_{1}$ smoothing algorithm has lower $\mbox{WRMSE}^{*}$ and $\mbox{WMAE}^{*}$ than other smoothing algorithms because the results generated by the $L_{1}$ smoothing algorithm were most frequently chosen by the volunteers as their preferred results when our dataset was constructed, as shown in Figure 4.

Since there exist multiple “groundtruth” smoothed images for each source image in our dataset, we define a weighted $L_{2}$ loss (Equation 6) and a weighted $L_{1}$ loss (Equation 7) in a manner similar to the weighted RMSE in Equation 3 and the weighted MAE in Equation 4:

where $M_{\theta}(x^{t})$ represents the output from a deep network and $x^{t}$ is the input image.

In addition to the weighted $L_{2}$ loss and weighted $L_{1}$ loss which enforce the consistency between the predicted smoothed images and their corresponding groundtruth smoothed images, we propose a neighborhood loss (Equation 8) to explicitly encourage a network to learn the local variations of the groundtruth images:

where $N_{i,j}$ denotes the $5\times 5$ neightborhood centered at pixel $(i,j)$. The neighborhood term $||(M_{\theta}(x^{t})_{i,j}-M_{\theta}(x^{t})_{p,q})-(Y^{t,k}_{i,j}-Y^{t,k}_{p,q})||_{1}$ explicitly penalizes deviations in the gradient domain. We add this neighborhood term to the weighted $L_{1}$ loss since edge-preserving smoothing involves evident gradient changes.

Quantitative results are presented in Table III, where we can see that ResNet achieves better performance than VDCNN when the same loss is used. $loss_{l_{1}}+loss_{nb}$ achieves better performance than $loss_{l_{1}}$ or $loss_{l_{2}}$ alone when the same network architecture is used, which validates the effectiveness of $loss_{nb}$. Figure 8 shows an example where we can see that the VDCNN trained with $loss_{l_{1}}+loss_{nb}$ produces smoother result at sky, roof and grass regions than $loss_{l_{2}}$ or $loss_{l_{1}}$ alone.

We thus take the VDCNN model and ResNet model trained with $loss_{l_{1}}+loss_{nb}$ as our baseline algorithms for our edge-preserving smoothing benchmark.

We augment the training data with horizontal flips. RGB training patches are randomly sampled from source images and the corresponding smoothed images. We set different training patch sizes and mini-batch sizes for VDCNN and ResNet since they have different receptive fields and model complexity. For VDCNN, patch size is set to $41\times 41$, and mini-batch size is set to 64. For ResNet, patch size is set to $96\times 96$, and mini-batch size is set to 16.

Our deep models are trained using the ADAM optimizer [32] with $\beta_{1}=0.9,\beta_{2}=0.999,\epsilon=10^{-8}$. The initial learning rate is set to $10^{-3}$. After the initial model converges, the learning rate is decreased by a factor of 10. Training is terminated once the model converges again.

We implemented the VDCNN and ResNet models in the Tensorflow framework and trained them using NVIDIA GeForce GTX 1080TI GPU. It takes one day to train VDCNN and two days to train ResNet, respectively.

The VDCNN model and ResNet model trained with $loss_{l_{1}}+loss_{nb}$ are taken as the baseline algorithms for our edge-preserving smoothing benchmark. In this section, we report their performance both quantitatively and qualitatively.

As shown in Table II, the ResNet model achieves the lowest WRMSE and WMAE when compared to existing smoothing algorithms. The VDCNN model also achieves favorable performance, with slightly bigger errors than the ResNet model. An example is presented in Figure 9, where our two baseline models are compared with the existing edge-preserving smoothing algorithms. Please note that these smoothing algorithms use their optimal parameter settings for achieving $\mbox{WMAE}^{*}$. It can be seen that some algorithms, such as SD filter, $L_{0}$ smoothing, Tree Filtering and LLF, cannot effectively remove trivial details at the floor regions. Some algorithms, such as FGS and WMF, blur the area around the foot regions with salient edges. $L_{1}$ smoothing and the deep models achieve overall better quality of edge-preserving smoothing than other algorithms. More results can be found in the supplemental materials.

A more detailed visual comparison between the deep models and the $L_{1}$ smoothing algorithm [12] is given in Figure 11 considering the fact that the $L_{1}$ smoothing algorithm is the most frequently chosen algorithm and it achieves the lowest WRMSE and WMAE among existing state-of-the-art algorithms. From Figure 11 we can see that the $L_{1}$ smoothing algorithm wrongly increases the color contrast between two flattened regions on the airplane. The results from VDCNN and ResNet models do not have such artifacts.

As mentioned earlier, we do not aim to reproduce individual filters like [13, 14, 15, 27]. By utilizing the constructed dataset, our baseline algorithm aims to train a deep CNN model that can produce reasonable edge-preserving smoothing results for a wide range of image contents without further tuning parameters. To the best of our knowledge, existing smoothing algorithms cannot perform consistently well on a wide range of image contents using a single parameter setting. As an example, a comparison between $L_{0}$ smoothing [7] and our ResNet model is shown in Figure 10. We can see that the $L_{0}$ smoothing algorithm needs to set different parameters for the ‘Racing car’ and the ‘Gloves’ images. If we set $\lambda$=0.03 for the ‘Racing Car’ image, the edge between grass and road will blur. $\lambda$=0.01 is the proper setting for the ‘Racing Car’ image. However, if we set $\lambda$=0.01 for the ‘Gloves’ image, there still remain undesirable noises. In contrast, our ResNet model produces robust visual results on different images without tuning parameters. More results can be found in the supplementary file.

In addition to visual quality, testing speed is also an important aspect for image smoothing methods. We report the running time of existing smoothing algorithms using the author-provided Matlab code on a 3.4GHz Intel i7 processor. The average running time over 100 testing images is shown in Table IV. The $L_{1}$ smoothing algorithm has lower $\mbox{WRMSE}^{*}$ and $\mbox{WMAE}^{*}$ than other smoothing algorithms, but spends hundreds of seconds on solving a series of large-scale sparse linear systems. The slow processing speed of $L_{1}$ smoothing algorithm prevents it from being an ideal pre-processing tool for other image processing applications, e.g., edge detection. In contrast, our ResNet-based model achieves the lowest WRMSE and WMAE while its GPU implementation runs faster than most existing state-of-the-art smoothing algorithms.

Recently, CNN-based approaches [13, 14, 15, 27] have been proposed to reproduce individual smoothing filters. We also compare our method with those approaches on our proposed dataset. Note that [13, 14, 15, 27] are originally designed to mimic smoothing filters where the groundtruth smoothed images are produced by the target filter. However, each source image in our dataset is associated with five groundtruth smoothed images and the quantitative measures are defined as weighted RMSE and weighted MAE (Equations 3-4). For fair comparison, we modify the loss function of previous CNN-based methods to weighted loss function like what we define in Equations 6-7.

Table V presents the quantitative results of our methods and previous CNN-based methods on the 100 test images. It can be seen that our ResNet-based model achieves the lowest WRMSE and WMAE thanks to the deeper structure and novel neighborhood loss function. There are totally 37 convolutional layers in our ResNet model. In comparison, Xu et al. [13] proposed a 3-layer convolutional neural networks to learn the gradient map. Liu et al. incorporated convolutional neural networks in U-net style and recurrent neural networks together while the deep CNN consists of 9 layers. Li et al. [15] proposed a joint network architecture of three components. Each component is a three-layer network. Fan et al. [27] presented a Cascaded Edge and Image Learning Network (CEILNet). Both E-CNN and I-CNN consist of 32 convolutional layers and residual unit is implemented for the middle layers.

Figure 12 shows some visual results of previous CNN-based approaches. It can be seen that [13] produces smoothed regions but it suffers from color transitions since this method requires a reconstruction step from gradient domain to final image output. The method of [14] generates obvious artifacts at ’sky’ and ’grass’ regions. Methods in [15] and [27] produce generally visually pleasing results but there still exist unwanted details if we inspect closely. In contrast, our ResNet result is closer to the groundtruth.

Tone mapping is a popular technique to map one set of colors to another to reproduce the appearance of a high dynamic range (HDR) image on a low dynamic range (LDR) displayer. The state-of-the-art tone mappers commonly adopt a layer decomposition scheme to decompose the HDR image into low- and high-frequency layers and then process them separately. In particular, the low frequency layer is estimated by applying an edge-preserving filter to the original HDR image. The edge-preserving property is very important for avoiding halo artifact and achieving naturalness in the tone-mapped images.Thus, a stable and effective edge-preserving filter is highly desirable to improve the tone mapping performance.

To avoid halo artifact, an edge-preserving filter should be able to preserve the strong edge regions and flatten other regions in the image, regardless of the image contents and types. Our ResNet baseline model can handle this task well, because it is trained on our dataset which is constructed with such criteria. We use the tone mapping framework in [33] by replacing the original bilateral filter by our ResNet model. We compare the tone mapped results with several state-of-the-art tone mappers, including bilateral filter method (BF) [33], visual adaptation (VAD) [34], and local edge-preserving filter (LEP) [35]. BF-based tone mapper [33] may not be as effective as the recently proposed approaches, but BF is widely adopted in different image processing tasks. On the other hand, VAD [34] and LEP [35] are selected because they obtain state-of-the-art performance. We do not compare with [41] because saliency is beyond the scope of this work. Fig .13 shows our tone mapping results compared with these tone mappers. We can see that our tone mapper with ResNet model reaches an excellent balance between halo removal and naturalness preservation. Other tone mappers suffers from either halo artifact or over-enhancement problems.

To better investigate the performance of the competitive tone mappers, we collected 100 HDR images online for objective evaluation. The TMQI metric [39] is used to score each tone mapped image by each method. Table VI shows the average TMQI score of each tone mapper. We can see that our tone mapper with ResNet model achieves the highest TMQI score. This demonstrates that our edge-preserving benchmark can facilitate the tone mapper to gain robust performance over different image types and contents.

Contrast enhancement aims to enhance the local contrast of an image that suffers from large illumination variation. Similar to tone mapping, a proper contrast enhancement framework decomposes an image into two components, illumination and reflectance. The estimation of illumination requires a high-performance edge-preserving filter. In [37], the authors proposed a criterion for optimal contrast enhancement, based on which the edge-preserving filter should preserve the boundary regions of an image and flatten the texture regions as much as possible. This coincides with the criterion adopted in our benchmark.

To test our benchmark in the application of contrast enhancement, we adopt the algorithm framework in [37] and only replace the Diffusion-based filtering component by our ResNet model. We also compare with bilateral filter [1] and guided filter (GF) [38] since they have been widely used in different image processing tasks. The enhancement results together with the results of the state-of-the-art contrast enhancement methods including nonlinear diffusion method (ND) [37] and weighted variational method (WV) [36] are shown in Fig. 14. We can see that our enhancement results exhibit clearer structures and higher local contrast.

To measure the contrast enhancement results quantitatively, we report the Image Enhancement Metric (IEM) [40], a full reference IQA metric, to assess the contrast of the enhanced images. We used the 18 test images denoted as ‘a’ to ‘r’ in [42] for evaluation. Some results are shown in Fig. 14, and all results can be found in supplementary. As Table VII shows, the improved performance validates that the ResNet model trained on our benchmark can be adopted as an efficient edge-preserving smoothing filter for image contrast enhancement.