Descanning: From Scanned to the Original Images with
a Color Correction Diffusion Model

Most image restoration methods that handle single CD (e.g., color fading or saturation (Wang et al. 2018; Xu et al. 2022; Zhu et al. 2017; Wang et al. 2022a)) have been developed based on the convolutional neural network (CNN) and Vision Transformer (Dosovitskiy et al. 2020) (Wang et al. 2022b; Zamir et al. 2022; Liang et al. 2021). For example, (Zhu et al. 2017) and (Wang et al. 2018) are popular image-to-image translation generative adversarial network (GAN) (Goodfellow et al. 2014) methods. For single NCD, many image restoration methods have been proposed for a single task such as denoising (Lefkimmiatis 2018; Chang et al. 2020), super-resolution (SR) (Zhang et al. 2018b; Niu et al. 2020), and deblurring (Nah, Hyun Kim, and Mu Lee 2017; Sun et al. 2015).

These models show notable performance when only a single type (blur, noise, etc.) of degradation is present. But it is unclear if they can handle many CDs and NCDs simultaneously. In our descanning problem, scanned images have complex CDs and NCDs with high uncertainty and diversity due to digital processing stages, e.g., scanning, printing, etc. Thus, directly restoring scanned images using the above methods may lead to poor performance, and a more dedicated model should be developed for descanning. In this paper, we propose a novel image restoration model with components designed to adequately handle both CD and NCD.

Many studies (Wan et al. 2020; Ho and Zhou 2022; Luo et al. 2021; Kim and Park 2018; Yu et al. 2022; Chen et al. 2021) have been proposed for real-world photo restoration. (Wan et al. 2020) uses translation networks for image and latent space, respectively, to restore real-world old photos with various degradations such as scratches, dust spots, and multiple noises. (Ho and Zhou 2022) removes degradations from smartphone-scanned photos in a semi-supervised way, with smartphone-scanned DIV2K (Timofte et al. 2018) images as inputs and the original digital versions as targets. (Yu et al. 2022) proposes ESDNet for demoiréing, which is a similar task to descanning in that both tasks aim to remove visually awkward color transitions and patterns simultaneously.

However, real-world scanned images still cannot be appropriately restored due to the more complex special NCDs such as the halftone pattern and bleed-through effect. There are a few classic image processing-based methods for restoring scanned documents (Verma and Malik 2015; Bhasharan, Konstantinides, and Beretta 1997). However, they mainly focus on eliminating dark borders and scanning shading which are dedicated to document-related degradations. These degradations typically arise from the geometric misalignment of books (e.g., curled pages and book spines), which is different from our focus of comprehensively restoring scanned images containing a variety of color photos and texts to clean original (digital) images.

Hence, to holistically address the descanning problem, we build a huge dataset with real scanned images from multiple scanners and their originals. Also, we propose a descanning model that is tailored to the properties of scanned images.

Recently, due to the impressive generation performance of diffusion models, they have been actively applied to various fields such as text-to-image generation (Ramesh et al. 2021; Saharia et al. 2022a), natural language processing (Li et al. 2022), and vision applications (Lugmayr et al. 2022; Baranchuk et al. 2021). Several diffusion models have also been developed for image restoration. (Kawar et al. 2022) introduces a diffusion model for various image restoration tasks such as SR, deblurring, and inpainting. (Saharia et al. 2022c) adapts DDPM in a conditional manner and achieves strong SR performance with iterative refinement processes.

In this paper, we propose DescanDiffusion which exploits the restoration power and generalization ability of diffusion models, especially DDPM. We observed that naively applying vanilla DDPMs for descanning can result in shifting away from the color distribution of the original image. To tackle this issue, we design a color encoder that predicts the color distribution of the original image given the scanned image and computes the color-corrected image, thereby offering a superior starting point for DDPM. The estimated color distribution is also used as a condition for the diffusion model to explicitly guide the model with color information during the diffusion process.

We manually scanned each page of the magazines with different popular scanners: Plustek OpticBook 4800, Canon imageRUNNER ADVANCE 6265, Fuji Xerox ApeosPort C2060, and Canon imagepress C650. The scanned images are digitized in the format of RGB TIFF and calibrated by the IT 8.7 (ISO 12641) standard. Since most scanners follow this standard for color calibration, it reduces the variance across scanner models, making our model more generalizable to different scanner types. After obtaining scanned images, we gather their corresponding original PDF copies online and convert them to the same RGB TIFF format.

As the scanned and original versions of magazine pages are misaligned due to margin settings and crumpled pages, etc., we take the following steps to align them: we first perform image registration with AKAZE (Alcantarilla and Solutions 2011) for each page. The page pairs are then manually inspected, filtering out images that are unmatched on a significant scale. Finally, we randomly crop each image into $1024\times 1024$ sizes and register them again with AKAZE, securing 18,360 pairs of aligned scanned and original images.

Among 18,000 images scanned using Plustek OpticBook 4800 and Cannon imageRUNNER ADVANCE 6265, 17,640 are used for training and 360 are used for validation. That is, the validation set is different from the training set. We leave 360 images scanned by Fuji Xerox ApeosPort C2060 and Canon imagepress C650 as the testing set. Note that the scanners used for the testing set are different from those for training and validation, which allows us to evaluate the generalization ability for unseen-type scanners.

By analyzing the complete dataset, we classify the degradations in scanned images into six types. Note that although we discuss each type of degradation separately, degradations themselves are often a combination of multiple degradation types. In Fig. 1, both (a) and (e) are scanned examples of DESCAN-18K. From Fig. 1 (b) to (h), except for (e), patches in the upper row with orange dotted lines are from the original images, and patches in the lower row with blue dotted lines are from their scanned counterparts.

As Fig. 1 shows, we categorize degradations as follows:

• •

  External noise is caused by the inflow of foreign substances during printing, scanning, and preserving. It appears in the form of dots or localized stains.

• •

  Internal noise is the visual degradation generated by the scanning process. It usually occurs as crumpled, curved and/or linear laser patterns.

• •

  Bleed-through effect is a degradation in which the contents of the back page are transmitted through and scanned together. Note that it solely appears in scanned images, not in ordinary real-world images.

• •

  Texture distortion consists of physical textures or wrinkles that occur during scanning. Note that this tends to appear globally, whereas external noise tends to appear locally in a specific region.

• •

  Halftone pattern is generated due to the printing process where many dots of different colors (e.g., cyan, magenta, yellow and black), sizes and spacings are imprinted to represent continuous shapes.

• •

  Color transition is the chromatic distortion of an image being globally altered during scanning and preserving. There are degradations such as color fading or saturation.

Detailed statistical analysis on DESCAN-18K can be found in the supplementary material.

Based on our analysis of the dataset, we simulate some of the degradations found in scanned images: (i) for color transition, we modify the HSV color space of the original image, (ii) for the bleed-through effect, we alpha-blend two original images, (iii) for halftone pattern and texture distortion, we apply Gaussian noise, (iv) for external and internal noise, we synthesize the form of dots and linear laser patterns, respectively. By doing so, we aim to improve the generalization performance of DescanDiffusion, enabling it to effectively restore images even if they are scanned from new scanners.

The degradation strength and probability of synthesizing them are determined randomly for each sample, with a uniform distribution, and original images from a subset of the DESCAN-18K training dataset are utilized to generate synthetic data. Specifically, we train DescanDiffusion+ using 25% synthetic-original pairs and 75% scanned-original pairs out of the total 17,640 image pairs in the training set, while the original DescanDiffusion exclusively utilizes scanned-original pairs from the same set. This ratio was determined empirically and its ablation study is provided in the supplementary material. Note that our synthetic data generation scheme can be applied to any original document image to augment the training set further.

In the global color correction module shown in Fig. 2 (b), we utilize the color encoder $\Phi$ to predict the color distribution of the original image $I_{o}$. The output of $\Phi$ is then used to correct the color distribution of the scanned image $I_{s}$ such that the color distribution of $I_{s}$ is approximated to that of $I_{o}$ thus removing most CDs from $I_{s}$. This results in the color-corrected image $I_{c}$, which can be exploited as a good condition in the following local generative refinement module.

We adopt ResNet-34 (He et al. 2016) as $\Phi$ because it is computationally efficient while having a large receptive field. With $I_{s}$ as input and the color distribution of the original image ($v_{o}\in\mathbb{R}^{1\times 6}$) as the target, $\Phi$ predicts $v_{c}=\Phi(I_{s})$, where $v_{c}\in\mathbb{R}^{1\times 6}$. Here, $v_{o}$ and $v_{c}$ are vectors composed of means ($\mu^{k}_{o}$, $\mu^{k}_{s}$) and standard deviations ($\sigma^{k}_{o}$, $\sigma^{k}_{s}$) of color channels $k$ in $I_{o}$ and $I_{c}$, respectively, where $k\in\{R,G,B\}$. This process is optimized by the L2 loss which can be written as

where $\Theta$ denotes the learnable parameters of $\Phi$.

Employing the estimated color statistics, i.e., $\mu^{k}_{c}$ and $\sigma^{k}_{c}$, we re-normalize the color distribution of $I_{s}$ to mimic the color distribution of $I_{o}$. This re-normalization process can be formulated as

where $I^{k}_{c}$ and $I^{k}_{s}$ are the $k$-th channels in $I_{c}$ and $I_{s}$, respectively. $\varepsilon$ in Eq. 7 is to secure numerical stability when $\sigma^{k}_{s}$ is close to zero, and set to $2^{-16}$. We perform this re-normalization for each R, G, B channel and concatenate them to be $I_{c}$.

It is noted that image-to-image translation methods (Isola et al. 2017; Zhu et al. 2017) that are able to mimic histogram matching can also be used to restore $I_{c}$. However, we found that the proposed color correction method yields competitive performance with much lower computational complexity.

Our proposed Local Generative Refinement Diffusion Model (LGRDM) mainly aims at removing NCDs from the color-corrected image $I_{c}$. In addition, LGRDM allows shifting the local color distributions of $I_{c}$ further toward $I_{o}$.

LGRDM involves a conditional denoising network $\epsilon_{\theta}$ based on UNet (Ronneberger, Fischer, and Brox 2015). As shown in Fig. 2 (c), $\epsilon_{\theta}$ is conditioned on two factors from the previous global color correction module: the color-corrected image $I_{c}$, and the color correction vector $v_{c}$.

The first condition $I_{c}$ guides the restoration process toward $I_{o}$ resulting in faster and better convergence. For $I_{c}$ conditioning, we concatenate $I_{c}$ with the latent variables $x_{t}$ at each time step $t$, where $t\in\{T,\ldots,1\}$.

The second condition $v_{c}$ aims to constrain color distribution shifts of the generated image. Note that DDPM tends to generate different color distributions from the target image due to its high generation ability. Color conditioning with $v_{c}$ serves as color guidance, allowing to preserve consistent color distribution. For conditioning with $v_{c}$, we project $v_{c}$ to a higher dimensional embedding space with a single-layer color projection network. The resulting color embedding is then added to the timestep embedding for conditioning. (Nichol and Dhariwal 2021)

Finally, $\epsilon_{\theta}$ is trained to estimate the added noise in $x_{t}$, where $x_{t}=\sqrt{\bar{\alpha}_{t}}I_{o}+\epsilon\sqrt{1-\bar{\alpha}_{t}}$. This process is optimized with the following loss:

Algorithm 1 and 2 describe the pseudo-codes of the training and inference processes of LGRDM, respectively. Note that $T_{o}$ in Algorithm 2 is the optimal number of sampling steps and is determined empirically. (Detailed explanation can be found in the supplementary material)

We trained our model from scratch with our DESCAN-18K. The ResNet is trained separately so that it can serve the purpose of global color correction by aligning the color distribution of the scanned image with the original one. If our full framework is trained jointly from the start, premature outputs of the ResNet may confuse the training of the DDPM leading to sub-optimal results, since the DDPM is conditioned on outputs of the ResNet. It is similar to the common training strategy of freezing the text encoder during training text-to-image diffusion models (Saharia et al. 2022b).

Given that descanning is a novel problem that has yet been previously explored, comparing to existing work is highly challenging. To address this problem, we extensively evaluate our method with models performing related tasks, which can be classified into: (i) image-to-image translation models (Pix2PixHD (Wang et al. 2018) and CycleGAN (Zhu et al. 2017)), (ii) recent image restoration models that conduct similar tasks as descanning (HDRUNet (Chen et al. 2021), Restormer (Zamir et al. 2022), ESDNet (Yu et al. 2022), and NAFNet (Chen et al. 2022)), (iii) real-world photo restoration models (OPR (Wan et al. 2020) and DPS (Ho and Zhou 2022)), (iv) commercial products (Clear Scan¹¹1Accessed 20 July. 2023, https://play.google.com/store/apps/details?id=com.indymobileapp.document.scanner (IndyMobileApp 2016), Adobe Scan²²2Accessed 20 July. 2023, https://play.google.com/store/apps/details?id=com.adobe.scan.android (Adobe 2017), and Microsoft Lens³³3Accessed 20 July. 2023, https://play.google.com/store/apps/details?id=com.microsoft.office.officelens (Microsoft 2015)), (v) a recent diffusion-based image restoration model (DDRM (Kawar et al. 2022)).

We re-train all compared methods using DESCAN-18K training set except OPR and DPS, for which official pre-trained models are used as we expect that they are optimized for restoring damaged real-world photos.

We employ the following four metrics to quantitatively evaluate the descanning performance. PSNR is adopted to calculate pixel-wise fidelity between the restored and original image. To measure perceptual quality, we use SSIM (Wang et al. 2004) and LPIPS (Zhang et al. 2018a). We also calculate Fréchet Inception Distance (FID) (Heusel et al. 2017) to assess generation performance.

Quantitative results are reported in Table. 1. Our DescanDiffusion and DescanDiffusion+ outperform other methods including commercial products on all metrics. As the testing set only contains images scanned from different scanners that were not used in the training set, the results suggest that our proposed method has good generalization performance and practicality for unseen-type scanners. In other words, regardless of which scanner is used, our method is able to restore scanned images robustly, which is important for the descanning task as various scanners exist in the real-world.

Table. 2 shows quantitative results after applying the global color correction through histogram matching on compared models. Compared to Table. 1, most models show notable improvements in most metrics after applying a global color correction. This suggests that CDs are dominant in scanned images, emphasizing the importance of addressing them with global color correction.

It also can be interpreted that the proposed color encoder and the color-conditioned DDPM contribute to the high descanning performance by estimating low dimensional color statistics and guiding the model with the color distribution.

Compared to DescanDiffusion, DescanDiffusion+ provides slightly better performance in PSNR and SSIM. For LPIPS and FID, DescanDiffusion and DescanDiffusion+ result in comparable performance. We found that DescanDiffusion+ tends to better eliminate high-frequency degradations which are similar to the synthesized ones when analyzed visually (See the supplementary material).

In addition, Table. 3 demonstrates that our DescanDiffusion surpasses the DDRM (Kawar et al. 2022), a recent diffusion-based image restoration model. This observation implies that diffusion-based image restoration models necessitate additional functions, such as our proposed global color correction module, to effectively eliminate multiple degradations in scanned images.

Fig. 3 shows visual results of both deep-learning-based methods and commercial products. DescanDiffusion almost resolves NCD and CD problems in scanned images, while the others leave these issues inadequately resolved or even worsen them. For instance, in the example in $3^{rd}$ row, NAFNet and ESDNet cannot completely eliminate internal noises. Moreover, the commercial product and real-world photo restoration models are not able to remove degradation well or even generate additional artifacts in some cases.

We conduct an ablation study to analyze the effect of three components in our proposed model: (i) color-corrected image condition for DDPM (denoted as CIC), (ii) color-correction vector condition for DDPM (denoted as CVC), and (iii) synthetic data generation scheme (denoted as SDG).

Table. 4 (a) and (b) shows that using the color-corrected image ($I_{c}$) obtained through the global color correction module as a condition for DDPM leads to a significant performance boost compared to the vanilla DDPM conditioned by the scanned image. Additionally providing the color-correction vector ($v_{c}$) from the global color correction module as a condition to DDPM further improves the descanning performance. (Table. 4 (c)) The color-correction vector is composed of the mean and standard deviation from each R, G, B channel in the color-corrected image. Hence, it can explicitly guide DDPM to consistently maintain the color distribution of the color-corrected image. Finally, we mix synthetic data rather than using only the original DESCAN-18K to enhance the generalization ability of DescanDiffusion when handling input images scanned by unseen-type scanners. Table. 4 (d) demonstrates that our model with SDG shows superior performance compared to the other versions. Meanwhile, when applying only SDG to the vanilla DDPM (Table. 4 (e)), it outperformed the vanilla DDPM. However, compared to our final model (Table. 4 (d)) including CIC, CVC, and SDG, a performance drop is observed. Therefore, it is verified that global color correction is still important.

Table. 5 provides a comparison of inference times for representative competitor models, conducted on an NVIDIA TESLA V100 GPU. As our method is a diffusion-based model, the inference time is slower compared to others based on CNNs or Transformers. However, as illustrated in Table. 1, our model exhibits superior performance compared to other methods. Moreover, in our training process, sampling begins from the scanned image instead of pure noise. Consequently, our method’s inference time can be reduced by 92% with just 10 reverse steps. (Detailed explanation can be found in the Algorithm 2 and the supplementary material)

To evaluate the performance of our proposed method on various image degradations, we further compared our model on additional datasets: 100 smartphone-scanned images from DPS (Ho and Zhou 2022) and 7 old photo images from OPR (Wan et al. 2020). Table. 6 shows the quantitative results of each dataset (separated by a slash) for comparison models, validating that our DescanDiffusion generalizes well to smartphone-scanned images and old photos containing multiple degradations. Nevertheless, the specialization of our DescanDiffusion lies in removing mixtures of complex NCDs and CDs unique to scanned images from scanners. Furthermore, due to such characteristics of scanned images, it is noted that our DESCAN-18K is the most suitable dataset for evaluating the descanning performance.