MC-Blur: A Comprehensive Benchmark for Image Deblurring

Several datasets have been developed for advances in image deblurring [6, 7, 3, 8, 9, 10, 11, 12, 13, 14, 15]. For example, motion blur is simulated by convolving images with uniform blur kernels by Levin et al.  [6] and Sun et al.  [7], or with non-uniform kernels by Köhler et al.  [3]. In [8], Lai et al. introduce a dataset including images of both real-world and synthetic blur (uniform blur kernels). However, the size of this dataset is still relatively small. Even when synthesizing blurry images via realistic blur kernels, the scale of the datasets mentioned above is small, making them difficult for deep learning-based deblurring methods. Furthermore, the blur kernels are relatively small, making them less effective for deblurring Ultra-High-Definition (UHD) images.

Considering that images are captured within the duration of camera exposure, blurry images can be modeled by the integration of neighboring frames [16],

where $T$ is the exposure time period and $g(\cdot)$ is the Camera Response Function (CRF). To model this process [16] and alleviate the problem of alignment [4], several deblurring datasets have been created based on the discrete formulation,

where $M$ is the number of frames.

In particular, the GoPro dataset [9] has been widely used for training deep models. Its sharp images are captured by a GoPro Hero4Black camera with a shutter speed of $240$ fps. Blurry images are generated by averaging continuous sharp frames over a time window. Similarly, based on this method, the HIDE [10] and REDS [2] datasets are created.

Rim et al.  [4] develop a dataset containing real blurry images and the corresponding sharp images. Two different cameras take image pairs with varying times of exposure. While the blur is realistic, this work requires an additional image alignment step to generate image pairs, which causes the problem of imprecise alignment. In addition, this dataset also lacks defocus blurry images or UHD images, which is of great interest for real-world scenarios. On the other hand, Abuolaim and Brown [5] capture $500$ images with defocus blur, but this number is small compared to the recent large-scale deblurring datasets. Moreover, the extent of defocus blur on their blurry images is relatively slight. The details of the existing representative datasets and the proposed MC-Blur dataset are listed in Table I.

In the computer vision community, the problem of image deblurring has garnered significant attention owing to its inherently ill-posed nature. Consequently, numerous image deblurring methods have emerged in the literature. Broadly speaking, these methods can be categorized into two main groups: 1) conventional deblurring methods and 2) deep learning-based deblurring methods. We introduce these methods in the following.

Conventional Deblurring Methods. Early deblurring methods refer to traditional approaches that address the image deblurring problem by incorporating constraints on blur kernels or latent images [17, 18, 19]. Consequently, lots of effective priors have been proposed, including the sparse gradients distribution model [20], dark channel prior [21], Local Maximum Gradient prior [22], structure prior [17], superpixel segmentation prior [19], and more. However, most image deblurring methods primarily focus on mitigating blur caused by camera movement. At the same time, real dynamic scenes involve additional complexities, such as camera movement, object movement (rigid or non-rigid), and variations in scene depth. Consequently, these methods often encounter challenges when handling blur within dynamic scenes.

Deep Deblurring Methods. In recent years, numerous deep learning methods have been proposed to address various computer vision tasks [23, 24, 25], which also include single image deblurring [26, 27, 28, 29, 30], and video deblurring [31, 32, 33, 34]. Deep deblurring methods typically train neural networks in an end-to-end manner, using blurry images as inputs and updating network parameters by comparing the outputs and the ground truth sharp images [35, 36, 37, 11]. The idea of multi-scale processing and an adversarial loss is used in [9] for image deblurring. Kupyn et al.  [38] adopt a conditional GAN for the deblurring task, resulting in the DeblurGAN model. An enhancement in terms of accuracy and speed is introduced with DeblurGAN-v2 [39]. In [27], patch-level deblurring is carried out to obtain an initial global estimation. The blur kernel is estimated, and the final result is obtained via deconvolution. Motivated by Spatial Pyramid Matching, a multi-patch scheme is applied to learn hierarchical representations in [40]. A CNN is combined with an RNN in [41] to deblur images of spatial-variant blur in dynamic scenes. Similarly, an LSTM with a CNN is employed in [42]. More recently, Zamir et al. [43] introduce a multi-stage architecture that gradually learns restoration functions for the degraded inputs, thus decomposing the overarching recovery process into a series of more manageable steps. They [44] also develop an efficient Transformer-based model designed to address the limitations of convolutional neural networks (CNNs) and enable effective image restoration. In addition, new architectures including skip connections [45, 46, 47, 48, 49, 50, 51], reblurring networks [52, 53, 54], unsupervised/self-supervised learning [55, 56, 57], attention blocks [58, 59, 60, 61, 62, 63, 64, 65, 66], NAS [67] and Nerf [68], are developed to improve the image deblurring performance.

We evaluate nine state-of-the-art deblurring methods on the proposed MC-Blur dataset, including the multi-scale architectures (DeepDeblur [9], SRN [42] and MIMO-UNet [70]), GAN based frameworks (DeblurGAN [38], DeblurGAN-v2 [39], and DBGAN [52]), multi-patch networks (DMPHN [40] and MPRNet [43]), and attention-based networks (Restormer [44]). We employ PSNR and SSIM as quantitative metrics to evaluate the deblurring methods while also comparing their performance qualitatively on synthesized and real blurry images. In addition, we use NIQE and SSEQ to evaluate the deblurring methods’ performance on real-world blur images.

RHM Subset. To evaluate image deblurring methods’ performance, we first conduct experiments on RHM. During the training stage, samples from all the training subsets of RHM are used together. We test them separately on the 250fps, 500fps, and 1000fps sets in the test stage. Table II shows that DeepDeblur, SRN, DMPHN, MPRNet, Restormer, and MIMO-UNet perform well in terms of PSNR and SSIM. One contributing factor is the utilization of pixel-level loss functions in these methods, which significantly contributes to achieving elevated values in full-reference pixel-based metrics. The DeblurGAN, DeblurGAN-v2, and DBGAN models use discriminators to help synthesize more realistic deblurred images. These models are not only enforced to focus on pixel-wise measures (L1 or L2) but also pay attention to the whole image. Deblurring networks focusing on pixel-wise measures are expected to achieve higher values of PSNR and SSIM. MIMO-UNet and Restormer consistently outperform other methods, demonstrating the remarkable efficacy of their multi-scale schemes and transformer blocks. We also show a visual comparison of RHM in Fig. 3.

UHDM Subset. In this part, the same deblurring methods are evaluated on the UHDM images. The training and testing samples are all from UHDM. Table III shows that all methods’ PSNR and SSIM values are significantly lower than those in Table II. It could be attributed to several reasons. One reason is that we use large-size blur kernels to synthesize blurry images, making the UHDM set more difficult. Another reason is that compared with non-UHD image deblurring, deblurring UHD images require recovering more details. Therefore, UHD image deblurring is a more challenging task. In addition, all the benchmarking deblurring networks are proposed for non-UHD image deblurring, which is also a reason causing their performance drop on UHD image deblurring. Among these methods, MPRNet stands out as the top performer, underscoring the effectiveness of its multi-stage architecture. Furthermore, MIMO-UNet and Restormer also demonstrate strong performance in the UHD image deblurring setting. We show qualitative results corresponding to the large-kernel blur caused in Fig. 4.

LSD Subset. To investigate the performance of the SOTA deblurring methods in the case of defocus blur, we conduct a benchmark study on the LSD subset. All the training and test samples are from LSD. Quantitative results are reported in Table IV, and qualitative results are shown in Fig. 5. We note that defocus image deblurring is a more complex problem compared with deblurring of motion-blurred images. This is because defocus blur happens when the camera lens cannot converge all incoming light onto one sensor point, causing loss of vital image info, making it difficult to observe details and sharpen edges. Unlike motion blur, defocus severity is closely tied to object depth. Objects at varying distances experience different blurriness. This depth-dependent blur requires precise spatially varying kernel estimation for accurate deblurring. While the SOTA deep deblurring methods can restore high-quality motion-deblurred images synthesized by averaging neighboring frames, the performance of defocus deblurring is significantly lower. Among these methods, MIMO-UNet, as a multi-scale deblurring approach, excels in performance, providing further evidence of the effectiveness of its multi-scale deblurring scheme for enhancing defocused image quality.

RMBQ Subset. In addition, we evaluate the performance of deblurring methods in real-world scenarios. The deblurred images by the evaluated methods (trained on RHM and tested on RMBQ) are shown in Fig. 6 and Fig. 7. Overall, the deblurring methods trained on the proposed MC-Blur dataset generate sharp images given the input, recovering sharp text or image structure. Among these methods, MIMO-UNet exhibits superior qualitative performance in real-world blurry images. For instance, in Fig. 6, the word ‘SPECIAL,’ and in Fig. 7, ‘WET PAINT,’ appear sharper than all other methods. These results suggest that MIMO-UNet can outperform the benchmarking methods in real-world scenarios.

Efficiency should be considered when the image resolution is high, especially for UHD images. Table V shows the efficiency evaluation results on UHD images. These experiments are carried out using a standard platform with a P40 GPU. The DeepDeblur [9], SRN [42], DMPHN [40], DBGAN [52], MPRNet [43] and Restormer [44] models require more than ten seconds to process one UHD image. On the other hand, it takes $2.46$, $3.63$, $2.45$ seconds for DeblurGAN [38], DeblurGAN-v2 [39] and MIMO-UNet [70] models, respectively, to process one UHD image. We evaluate the speed and parameter efficiency of state-of-the-art deblurring methods. As such, we offer valuable insights into the efficiency of current methods in handling blurry images. Furthermore, this evaluation serves as a benchmark for the research community and underscores the practical applicability of our dataset.

To illustrate the advantages of our RHM over the existing ones, we conduct an analysis with a cross-dataset learning protocol in this section. Specifically, we select a main-stream deblurring method [43] and train this identical deblurring network using five combinations of different training sets. Then we evaluate the trained different versions of [43] on various testing sets. Note that all parameter settings are set as the original paper, except that the epoch is set as $200$. Table VI shows the results.

Based on the top three rows in Table VI, the deblurring network is expected to achieve the best performance when the training and testing datasets come from the same source. It is also observed that our RHM provides better generalization potential to the deblurring network when trained with a single source than its counterparts. For instance, when tested on different test datasets, the network trained with RHM consistently achieves the best performance (see the italic values in the table). If the test set is not constrained to be different from the training source, the best performance will be achieved when the training and the test data sources are identical.

When trained with multiple sources (the bottom two rows in Table VI), the performance is significantly improved if our RHM is taken as an additional training source. Fig. 8 demonstrates exemplar testing results corresponding to Test-1 in Table VI by the deblurring network [43] trained with the five settings of training source. These qualitative results coincide with our analysis from Table VI.

In addition, we also evaluate on other real blurry datasets (i.e., the proposed RMBQ and RWBI datasets [52]), which do not provide ground-truth sharp images. The results in Table VI demonstrate that the network trained with our RHM performs better in real-world scenery.

As mentioned above, there exists a high-quality real defocus blurry DPDD dataset. Compared to the DPDD dataset, the proposed LSD set consists of heavier defocus blurry images; thus, there is a significant difference between them. In this section, to verify the effectiveness of the proposed LSD, we conduct another experimental study.

Similarly, we train an identical deblurring method [43], using our proposed LSD and DPDD respectively as training sets and then evaluate the trained different versions on different testing sets, including LSD, DPDD and the joint of LSD and DPDD. Table VII shows that, though both LSD and DPDD consist of real defocus blurry images, a model trained on one dataset cannot perform well on the other. Specifically, the model trained on LSD performs better for heavy-defocus deblurring, while the model trained on DPDD performs better on non-heavy-defocus blurry images. It is also important to observe that, for the testing set consisting of the joint of LSD and DPDD, the model trained with LSD performs better than the one prepared with DPDD. This suggests the utility advance of LSD over DPDD. We attribute this to the heavy defocus blurry images in the LSD set.

To further study how different blur factors influence the behavior of SOTA methods, we examine how methods trained on the motion blur subset (i.e., RHM) behave on subsets of other factors (i.e., LSD and UHDM). We train MPRNet, Restormer, and MIMO-UNet on the RHM and test these methods respectively on UHDM and LSD.

Experimental results in Table LABEL:table:cross-sets_2 show that models trained on one type of blurry images perform poorly on recovering sharp images from other types of blur. Therefore, proposing a method that can recover sharp images from various kinds of blurry images is still an open topic.

The benchmarking results on different sets of the proposed MC-Blur dataset, reveal several interesting findings. First, GAN-based networks achieve lower values of PSNR & SSIM for motion-blurred images than methods without using the GAN framework. However, the two networks (with and without using the GAN framework) show fewer differences for defocus images. This indicates that paying attention to the whole images (e.g., the adversarial loss function), rather than just considering the pixel level (e.g., L1 and L2 loss functions), may be a direction for defocus deblurring. The results in Tables II, IV support this finding. Second, current deep deblurring networks can generate high-quality images for motion-blurred images. However, it is difficult for them to achieve similar performance on large kernel-based Ultra-High-Definition blurry images (Tables II, III). Since increasing numbers of modern mobile devices allow capturing UHD images, it may be a meaningful direction for researchers to study UHD image deblurring. Third, current deep methods can deblur a non-UHD image in two seconds [40]. However, as shown in Table V, handling a UHD image takes significantly longer. Therefore, generating deblurred UHD images at a high rate while maintaining deblurring performance is still an open problem. In addition, this work has limitations. First, the proposed dataset is mainly for benchmarking single-image deblurring; thus, we do not evaluate video deblurring methods. Second, the MC-Blur dataset does not include human faces, so it is unsuitable for face restoration. Third, except the RMBQ subset, all the other subsets focus on a single blur factor.