Toward Real-World Single Image Super-Resolution:
A New Benchmark and A New Model

The DSLR camera imaging system can be approximated as a thin lens [53]. An illustration of the image formation process by thin lens is shown in Fig. 2. We denote the object distance, image distance and focal length by $u,v,f$, and denote the size of object and image by $h_{1}$ and $h_{2}$, respectively. The lens equation is defined as follows [53]:

$$\frac{1}{f}=\frac{1}{u}+\frac{1}{v}.$$

(1)

The magnification factor $M$ is defined as the ratio of the image size to the object size:

$$M=\frac{h_{2}}{h_{1}}=\frac{v}{u}.$$

(2)

In our case, the static images are taken at a distance (i.e., $u$) larger than 3.0m. Both $h_{1}$ and $u$ are fixed and $u$ is much larger than $f$ (the largest $f$ is 105mm). Combining Eq. (1) and Eq. (2), and considering the fact that $u\gg f$, we have:

$$h_{2}=\frac{f}{u-f}h_{1}\approx\frac{f}{u}h_{1}.$$

(3)

Therefore, $h_{2}$ is approximately linear to $f$. By increasing the focal length $f$, larger images with finer details will be recorded in the camera sensor. The scaling factor can also be controlled (in theory) by choosing specific values of $f$.

We used two full frame DSLR cameras (Canon 5D3 and Nikon D810) to capture images for data collection. The resolution of Canon 5D3 is $5760\times 3840$, and that of Nikon D810 is $7360\times 4912$. To cover the common scaling factors (e.g., $\times 2,\times 3,\times 4$) used in most previous SISR datasets, both cameras were equipped with one 24$\sim$105mm, $f$/4.0 zoom lens. For each scene, we took photos using four focal lengths: 105mm, 50mm, 35mm, and 28mm. Images taken by the largest focal length are used to generate the ground-truth HR images, and images taken by the other three focal lengths are used to generate the LR versions. We choose 28mm rather than 24mm because lens distortion at 24mm is more difficult to correct in post-processing, which results in less satisfied quality in image pair registration.

The camera was set to aperture priority mode and the aperture was adjusted according to the depth-of-field (DoF) [52]. Basically, the selected aperture value should make the DoF large enough to cover the scene and avoid severe diffraction. Small ISO is preferred to alleviate noise. The focus, white balance, and exposure were set to automatic mode. The center-weighted metering option was selected since only the center region of captured images were used in our final dataset. For stabilization, the camera was fixed on a tripod and a bluetooth remote controller was used to control the shutter. Besides, lens stabilization was turned off and the reflector was pre-rised when taking photos.

To ensure the generality of our dataset, we took photos in both indoor and outdoor environment. Scenes with abundant texture are preferred considering that the main purpose of super-resolution is to recover or enhance image details. For each scene, we first captured the image at 105mm focal length and then manually decreased the focal length to take three down-scaled versions. $234$ scenes were captured, and there are no overlapped scenes between the two cameras. After discarding images having moving objects, inappropriate exposure, and blur, we have $595$ HR and LR image pairs in total. The numbers of image pairs for each camera at each scaling factor are listed in Table 1.

Although it is easy to collect images on different scales by zooming the lens of a DSLR camera, it is difficult to obtain pixel-wise aligned image pairs because the zooming of lens brings many uncontrollable changes. Specifically, images taken at different focal lengths suffer from different lens distortions and usually have different exposures. Moreover, the optical center will also shift when zooming the focal length because of the inherent defect of lens. Even the scaling factors are varying slightly because the lens equation (Eq. (1)) cannot be precisely satisfied in practical focusing process. With the above factors, none of the existing image registration algorithms can be directly used to obtain accurate pixel-wise registration of two images captured under different focal length. We thus develop an image registration algorithm to progressively align such image pairs to build our RealSR dataset.

The registration process is illustrated in Fig. 3. We first import the images with meta information into PhotoShop to correct the lens distortion. However, this step cannot perfectly correct the lens distortion especially for the region distant from the optical center. We thus further crop the interested region around the center of the image, where distortion is not severe and can be well corrected. The cropped region from the image taken at 105mm focal length is used as the ground-truth HR image, whose LR counterparts are to be registered from the original images taken at 50mm, 35mm, or 28mm focal length. Since there is certain luminance and scale difference between images taken at different focal lengths, those popular keypoint based image registration algorithms such as SURF [3] and SIFT [31] cannot always achieve pixel-wise registration, which is necessary for our dataset. To obtain accurate image pair registration, we develop a pixel-wise registration algorithm which simultaneously considers luminance adjustment. Denote by $\mathbf{I}_{H}$ and $\mathbf{I}_{L}$ the HR image and the LR image to be registered, our algorithm minimizes the following objective function:

$$\min_{\bm{\tau}}||\alpha C(\bm{\tau}\circ\mathbf{I}_{L})+\beta-\mathbf{I}_{H}||_{p}^{p},$$

(4)

where $\bm{\tau}$ is an affine transformation matrix, $C$ is a cropping operation which makes the transformed $\mathbf{I}_{L}$ have the same size as $\mathbf{I}_{H}$, $\alpha$ and $\beta$ are luminance adjustment parameters, $||\cdot||_{p}$ is a robust $L_{p}$-norm $(p\leq 1)$, e.g., $L_{1}$-norm.

The above objective function is solved in an iterative manner. At the beginning, according to Eq. (3), the $\bm{\tau}$ is initialized as a scaling transformation with scaling factor calculated as the ratio of two focal lengths. Let $\mathbf{I}_{L}^{\prime}=C(\bm{\tau}\circ\mathbf{I}_{L})$. With $\mathbf{I}_{L}^{\prime}$ and $\mathbf{I}_{H}$ fixed, the parameters for luminance adjustment can be obtained by $\alpha=\text{std}(\mathbf{I}_{H})/\text{std}(\mathbf{I}_{L}^{\prime})$ and $\beta=\text{mean}(\mathbf{I}_{H})-\alpha\text{mean}(\mathbf{I}_{L}^{\prime})$, which can ensure $\mathbf{I}_{L}^{\prime}$ having the same pixel mean and variance as $\mathbf{I}_{H}$ after luminance adjustment. Then we solve the affine transformation matrix $\bm{\tau}$ with $\alpha$ and $\beta$ fixed. According to [37, 58], the objective function w.r.t. $\bm{\tau}$ is nonlinear, which can be iteratively solved by a locally linear approximation:

$$\min_{\Delta\bm{\tau}}||\alpha C(\bm{\tau}\circ\mathbf{I}_{L})+\beta+\alpha\mathbf{J}\Delta\bm{\tau}-\mathbf{I}_{H}||_{p}^{p},$$

(5)

where $\mathbf{J}$ is the Jacobian matrix of $C(\bm{\tau}\circ\mathbf{I}_{L})$ w.r.t. $\bm{\tau}$, and this objective function can be solved by an iteratively reweighted least square problem (IRLS) as follows [8]:

$$\min_{\Delta\bm{\tau}}||\mathbf{w}\odot(\mathbf{A}\Delta\bm{\tau}-\mathbf{b})||_{2}^{2},$$

(6)

where $\mathbf{A}=\alpha\mathbf{J}$, $\mathbf{b}=\mathbf{I}_{H}-(\alpha C(\bm{\tau}\circ\mathbf{I}_{L})+\beta)$, $\mathbf{w}$ is the weight matrix and $\odot$ denotes element-wise multiplication. Then we can obtain:

$$\Delta\bm{\tau}=(\mathbf{A}^{\prime}\text{diag}(\mathbf{w})^{2}\mathbf{A})^{-1}\mathbf{A}^{\prime}\text{diag}(\mathbf{w})^{2}\mathbf{b},$$

(7)

and $\bm{\tau}$ can be updated by: $\bm{\tau}=\bm{\tau}+\Delta\bm{\tau}$.

We iteratively estimate the luminance adjustment parameters and the affine transformation matrix. The optimization process converges within $5$ iterations since our prior information of the scaling factor provides a good initialization of $\bm{\tau}$. After convergence, we can obtain the aligned LR image as $\textbf{I}_{L}^{A}=\alpha C(\bm{\tau}\circ\mathbf{I}_{L})+\beta$.

To demonstrate the advantages of our RealSR dataset, we conduct experiments to compare the real-world super-resolution performance of SISR models trained on simulated datasets and RealSR dataset. Considering that most state-of-the-art SISR models were trained on DIV2K [45] dataset, we employed the DIV2K to generate simulated image pairs with bicubic degradation (BD) and multiple degradations (MD) [60]. We selected three representative and state-of-the-art SISR networks, i.e., VDSR [23], SRResNet [29] and RCAN [61], and trained them on the BD, MD and RealSR training datasets for each of the three scaling factors ($\times 2$, $\times 3$, $\times 4$), leading to a total of $27$ SISR models. To keep the network structures of SRResNet and RCAN unchanged, the input images were shuffled with factor $\frac{1}{2},\frac{1}{3},\frac{1}{4}$ for the three scaling factors $\times 2$, $\times 3$, $\times 4$, respectively.

We applied the $27$ trained SISR models to the RealSR testing set, and the average PSNR and SSIM indices are listed in Table 2. The baseline bicubic interpolator is also included for comparison. One can see that, on our RealSR testing set, the VDSR and SRResNet models trained on the simulated BD dataset can only achieve comparable performance to the simple bicubic interpolator. Training on the MD dataset brings marginal improvements over BD, which indicates that the authentic degradation in real-world images is difficult to simulate. Employing a deeper architecture, the RCAN ($>$400 layers) can improve (0.2dB$\sim$0.3dB) the performance over VDSR and SRResNet on all cases.

Using the same network architecture, SISR models trained on our RealSR dataset obtain significantly better performance than those trained on BD and MD datasets for all the three scaling factors. Specifically, for scaling factor $\times 2$, the models trained on our RealSR dataset have about $1.0$dB improvement on average for all the three network architectures. The advantage is also significant for scaling factors $\times 3$ and $\times 4$. In Fig. 5, we visualize the super-resolved images obtained by different models. As can be seen, the SISR results generated by models trained on simulated BD and MD datasets tend to have blurring edges with obvious artifacts. On the contrary, models trained on our RealSR dataset recover clearer and more natural image details. More visual examples can be found in the supplementary file.

To demonstrate the efficiency and effectiveness of the proposed LP-KPN, we then compare it with $8$ SISR models, including VDSR, SRResNet, RCAN, a baseline direct pixel synthesis (DPS) network and four KPN models with kernel size $k=5,7,13,19$. The DPS and the four KPN models share the same backbone as our LP-KPN. All models are trained and tested on our RealSR dataset. The PSNR and SSIM indices of all the competing models as well as the bicubic baseline are listed in Table 3.

One can notice that among the four direct pixel synthesis networks (i.e., VDSR, SRResNet, RCAN and DPS), RCAN obtains the best performance because of its very deep architecture (over 400 layers). Using the same backbone with less than $50$ layers, the KPN with $5\times 5$ kernel size already outperforms the DPS. Using larger kernel size consistently brings better results for the KPN architecture, and it obtains comparable performance to the RCAN when the kernel size increases to $19$. Benefitting from the Laplacian pyramid decomposition strategy, our LP-KPN using three different $5\times 5$ kernels achieves even better results than the KPN with $19\times 19$ kernel. The proposed LP-KPN obtains the best performance but with the lowest computational cost for all the three scaling factors. The detailed complexity analysis and visual examples of the SISR results by the competing models can be found in the supplementary file.

To evaluate the generalization capability of SISR models trained on our RealSR dataset, we conduct a cross-camera testing. Images taken by two cameras are divided into training and testing sets, separately, with $15$ testing images for each camera at each scaling factor. The three scales of images are combined for training, and models trained on one camera are tested on the testing sets of both cameras. The LP-KPN and RCAN models are compared in this evaluation, and the PSNR indexes are reported in Table 4.

It can be seen that for both RCAN and LP-KPN, the cross-camera testing results are comparable to the in-camera setting with only about $0.32$dB and $0.30$dB gap, respectively, while both are much better than bicubic interpolator. This indicates that the SISR models trained on one camera can generalize well to the other camera. This is possibly because our RealSR dataset contains various degradations produced by the camera lens and image formation process, which share similar properties across cameras. Between RCAN and LP-KPN models, the former has more parameters and thus is easier to overfit to the training set, delivering slightly worse generalization capability than LP-KPN. Similar observation has been found in [2, 48, 34].

To further validate the generalization capability of our RealSR dataset and LP-KPN model, we evaluate our trained model as well as several competitors on images outside our dataset, including images taken by one Sony a7II DSLR camera and two mobile cameras (i.e., iPhone X and Google Pixel 2). Since there are no ground-truth HR versions of these images, we visualize the super-resolved results in Fig. 1 and Fig. 6. In all these cases, the LP-KPN trained on our RealSR dataset obtains better visual quality than the competitors, recovering more natural and clearer details. More examples can be found in the supplementary file.

Currently, the proposed RealSR dataset contains $595$ HR-LR image pairs covering a variety of image contents. To ensure the diversity of our RealSR dataset, images are captured in indoor, outdoor and laboratory environments. Several examples of our RealSR dataset are shown in Fig. 7. It provides, to the best of our knowledge, the first general purpose benchmark for real-world SISR model training and evaluation. The RealSR dataset will be made publicly available.

The network architecture of our proposed Laplacian pyramid based kernel prediction network (LP-KPN) is shown in Table 5. In this table, “$H\times W\times C$ conv” denotes a convolutional layer with $C$ filters of size $H\times W$ which is immediately followed by a ReLU nonlinearity. Each residual block contains two $3\times 3$ convolutional layers with the same number of filters on both layers. The stride size for all convolution layers is set to $1$ and the number of filters $C$ in each layer is set to $64$, except for the last layer where $C$ is set to $25$. The structure of the residual block is shown in Fig. 8, which is same as [30]. We use the shuffle operation [42] to downsample and upsample the the image.

In this subsection, we provide more visual results by SISR models trained on simulated SISR datasets (BD and MD [60]) and our proposed RealSR dataset. Two images captured by Canon 5D3, two images captured by Nikon D810 and their super-resolved results are shown in Fig. 9. Again, the models trained on our RealSR dataset consistently obtain better visual quality compared to their counterparts trained on simulated datasets.

The running time and the number of parameters of the competing models are listed in Table 6. One can see that although larger kernel size can consistently bring better results for the KPN architecture, the number of parameters will also greatly increase. Benefitting from the Laplacian pyramid decomposition strategy, our LP-KPN using $5\times 5$ kernel can achieve better results than the KPN using $19\times 19$ kernel, and it uses much less parameters. Specifically, our LP-KPN model contains less than $\frac{1}{5}$ parameters of the RCAN model [61] and it runs about 3 times faster than RCAN. The visual examples of the SISR results by the competing models are shown in Fig. 10. Though all the SISR models in Fig. 10 are trained on our RealSR dataset and they all achieve good results, our LP-KPN still obtains the best visual quality among the competitors.

In this subsection, we provide more super-resolved results on images outside our dataset, including images taken by one Sony a7II DSLR camera and two mobile cameras (i.e., iPhone X and Google Pixel 2). The visual examples are shown in Fig. 11.