Image Super-Resolution with
Text Prompt Diffusion

We aim to reconstruct HR images from LR images with complex and unknown degradation. To encompass the typical degradations while maintaining design simplicity, we develop the degradation model, as depicted in Fig. 2a. Note that while the degradation process in the illustration is applied sequentially, our degradation pipeline supports the more flexible format, e.g., random degradation sequences and the omission of certain components. We describe each component in detail.

Blur. We employ two kinds of blur: isotropic and anisotropic Gaussian blur. The blur is controlled by the kernel with two parameters: kernel width $\eta$ and standard deviation $\sigma$.

Resize. We upsample/downsample images using two resize with scale factors $\gamma_{1}$ and $\gamma_{2}$, respectively. We employ area, bilinear, and bicubic interpolation. The two-step resizing can broaden the degradation range and enhance the generality of the model. We demonstrate it in Sec. 4.2.2.

Noise. We apply Gaussian and Poisson noise, with noise levels controlled by $\varphi_{1}$ and $\varphi_{2}$, respectively. Meanwhile, noise is randomly applied in either RGB or gray format.

Compression. We adopt JPEG compression, a widely used compression standard, for image compression. The quality factor $q$ controls the image compression quality.

Given an HR image $\mathbf{y}$, we determine the degradation by randomly selecting the degradation method (e.g., Gaussian noise or Poisson noise), and sampling all parameters (e.g., noise level $\varphi_{1}$) from the uniform distribution. Through the degradation process, we obtain the corresponding LR image $\mathbf{x}$. Compared to other degradation models (e.g., high-order (Wang et al., 2021)), ours maintains flexibility and simplicity while covering broad scenarios.

After generating HR-LR image pairs through the degradation model, we further provide descriptions for each pair as text prompts. Consequently, we incorporate text prompts into the dataset. This process encompasses two key considerations: (1) The specific content that should be described; (2) The user-friendly method for generating corresponding descriptions concisely and effectively. Given the characteristics of image SR, we utilize text to represent degradation. Meanwhile, we represent the degradation via a discretization manner based on the binning method (Zhang et al., 2023b).

Text prompt for degradation. Typical text prompt image generation and manipulation methods (Ramesh et al., 2022; 2021; Avrahami et al., 2022) apply text prompts to describe the image content. These prompts often require semantic-level interpretation and processing of the image content. However, for the image SR task, it is crucial to prioritize fidelity to the original image. Meanwhile, LR images could provide the majority of the low-frequency information (Zhang et al., 2018c) and semantic information related to the content (Rombach et al., 2022). As shown in Fig. 3, elements like ‘building’ and ‘shutters’ in the caption prompt can be obtained from the LR image.

Therefore, we adopt the prompt for degradation, instead of the description of the overall image. This prompt can provide degradation priors and thus enhance the capability of methods to model degradation, which is crucial for image SR. As shown in Fig. 3, utilizing text to depict degradation, instead of the overall image content (Caption), yields restoration that is more aligned with the ground truth. To further demonstrate the effectiveness of text prompts for degradation, we provide more analyses in Sec. 4.2.4.

Text representation. To facilitate data generation and practical usability, we describe degradation in natural language with the approach illustrated in Fig. 2a. Overall, we describe each degradation component via a discretized binning method, and combine them in a flexible format.

First, we discretize the degradation model into several components (e.g., blur) and describe each using qualitative language via a binning method. The sampling distribution of parameters corresponding to each component is evenly divided into discrete intervals (bins). Each bin is summarized to represent the degradation. For instance, we divide the distribution of noise level $\varphi_{1}$ $[0,9]$ into three uniform intervals ($[0,3)$, $[3,6)$, and $[6,9]$), and describe them as ‘light’, ‘medium’, and ‘heavy’. Both Gaussian and Poisson noises are summarized as ‘noise’, leading to the final representation: [medium noise]. Compared to specifying degradation names and their parameters, e.g., [Gaussian noise with noise level 4.5], our discretized representation is more intuitive and user-friendly.

Finally, the overall degradation representation combines all component descriptions, i.e., [deblur description, …, resize description]. Figure 2b illustrates an example. The content of the prompt directly corresponds to the degradation. Furthermore, it is notable that, in our method, the prompt exhibits good generalization and supports flexible description formats. For instance, both arbitrary order or simplified (e.g., only noise description) prompts can still lead to satisfactory restoration outcomes. In Sec. 4.2.3, we conduct a detailed investigation of the prompt format.

Real-world application. For real-world images, users can utilize the latest multi-modal large language models (MLLMs) (Liu et al., 2023; OpenAI, 2023; Ye et al., 2024; Wu et al., 2024) to generate professional image quality assessments as prompts. This approach simplifies prompt generation for users. It also provides a pathway for improving image SR using MLLMs. Furthermore, users can fine-tune the MLLM-generated prompts based on the restoration results to achieve more personalized enhancements. More details are provided in the supplementary material.

Text prompt image models (Patashnik et al., 2021; Avrahami et al., 2022; Rombach et al., 2022) mainly employ multi-modal embedding models, e.g., CLIP (Radford et al., 2021), as text encoders. These encoders are capable of generating meaningful representations pertinent to tasks. Besides, compared to multi-modal embeddings, pre-trained language models (Devlin et al., 2019; Raffel et al., 2020) exhibit stronger text comprehension capabilities. Therefore, we attempt to apply different pre-trained text encoders to build a series of networks. These models demonstrate varying restoration performance levels, which we further analyze in Sec. 4.2.5.

We train the PromptSR using the text-image ($\mathbf{c}$, [$\mathbf{y}$, $\mathbf{x}$]) dataset generated as described in Sec. 3.1. Given an HR image $\mathbf{y}$, we add noise $\bm{\epsilon}$ through $t$ diffusion steps to obtain a noisy image $\mathbf{y}_{t}$, where $t$ is randomly sampled from $[1,T]$. The DN is conditioned on the LR image $\mathbf{x}$, noisy image $\mathbf{y}_{t}$, and text prompt $\mathbf{c}$ to predict the added noise. The training objective is formulated as:

$$\mathcal{L}=\mathbb{E}_{\mathbf{y},\mathbf{x},\mathbf{c},t,\bm{\epsilon}\sim\mathcal{N}(0,1)}[|\bm{\epsilon}-\bm{\epsilon}_{\theta}(\mathbf{y}_{t},\mathbf{x},\bm{\tau}_{\theta}(\mathbf{c}),t)|_{2}^{2}],$$

(1)

where $\bm{\epsilon}_{\theta}$ is the DN, while $\bm{\tau}_{\theta}$ is the text encoder. We freeze the weights of the text encoder and only train the DN. In this way, we can retain the original capabilities of the pre-trained model. Meanwhile, we can reduce training overhead by computing text embedding offline. After completing the training process, the PromptSR can be employed for both synthetic and real-world images. Benefiting from the multi-modal (text and image) design, it demonstrates excellent performance.

The degradation model in our proposed pipeline encompasses four operations: blur, resize, noise, and compression. Following previous methods (Wang et al., 2021; Zhang et al., 2021), the parameters for these operations are sampled from the uniform distribution. Blur: We adopt isotropic Gaussian blur and anisotropic Gaussian blur with equal probability. The kernel width $\eta$ is randomly selected from the set $\{7,9,\dots,21\}$. The standard deviation $\sigma$ is sampled from a uniform distribution $\mathcal{U}_{[0.2,3]}$. Resize: We employ area, bilinear, and bicubic interpolation with probabilities of $[0.3,0.4,0.3]$. To expand the scope of degradation, we perform two resize operations at different stages. The first resize spans upsample and downsample, where the scale factor is $\gamma_{1}\sim\mathcal{U}_{[0.15,1.5]}$. The second resize operation scales the resolution to $\frac{1}{4}$ of the HR image. Noise: We apply Gaussian and Poisson noise with equal probability. The level of Gaussian noise is $\varphi_{1}\sim\mathcal{U}_{[1,30]}$, while the level of Poisson noise is $\varphi_{2}\sim\mathcal{U}_{[0.05,3]}$. Compression: We employ JPEG compression with quality factor $q\sim\mathcal{U}_{[30,95]}$. Meanwhile, in all experiments, for simplifying implementation, unless expressly noted, the degradation and text prompt follow the fixed order and correspond one to one.

We use the LSDIR (Li et al., 2023) as the training dataset. The LSDIR contains 84,991 high-resolution images. We generate the corresponding text-image dataset using our proposed pipeline. We evaluate our method on both synthetic and real-world datasets. For synthetic datasets, we employ Urban100 (Huang et al., 2015), Manga109 (Matsui et al., 2017), and the validation (Val) datasets of LSDIR and DIV2K (Timofte et al., 2017). For real-world datasets, we utilize RealSR (Cai et al., 2019). We also employ 45 real images directly captured from the internet, denoted as Real45. We conduct all experiments with a scale factor of $\times$4. To quantitatively evaluate our method, we adopt two traditional metrics: PSNR and SSIM (Wang et al., 2004), which are calculated on the Y channel of the YCbCr color space. We also utilize several perceptual metrics: LPIPS (Zhang et al., 2018b), ST-LPIPS (Ghildyal & Liu, 2022), DISTS (Ding et al., 2020), and CNNIQA (Kang et al., 2014). We further adopt an aesthetic metric: NIMA (Talebi & Milanfar, 2018).

The proposed PromptSR consists of two components: the denoising network (DN) and the pre-trained text encoder. The DN employs a U-Net architecture with a 4-level encoder-decoder. Each level contains two ResNet (He et al., 2016; Ho et al., 2020) blocks and one cross-attention block. For more detailed information about the DN model structure, please refer to the supplementary material. For the text encoder, we apply the pre-trained multi-modal model, CLIP (Radford et al., 2021). Additionally, we discuss other large language models, e.g., T5 (Raffel et al., 2020), in Sec. 4.2.5.

We train our model on the generated text-image dataset with a batch size of 16 for a total of 1,000,000 iterations. The input image is randomly cropped to 64$\times$64. We adopt the Adam optimizer (Kingma & Ba, 2015) with $\beta_{1}$=$0.9$ and ${\beta}_{2}$=$0.99$ to minimize the training objective (Eq. 1). The learning rate is 2$\times$10^-4 and is reduced by half at the 500,000-iteration mark. For DM, we set the total time step $T$ as 2,000. For inference, we employ the DDIM sampling (Song et al., 2020) with 50 steps. We use PyTorch (Paszke et al., 2019) to implement our method with 4 Nvidia A100 GPUs.

We conduct an ablation to show the influence of introducing the text prompt into image SR. The results are listed in Tab. 2. To validate the effectiveness of the text prompts, rather than benefiting from the specialized network, we conduct experiments on ControlNet (Zhang et al., 2023a) and proposed PropmtSR. We take the LR image as the condition to ControlNet to realize SR. All four compared models are trained on LSDIR. For models that are without text prompts, we train and test using empty string. The comparison reveals that text prompts significantly enhance SR performance. It also demonstrates the universality of text prompts, applicable to various models.

Moreover, we visualize the impact of different prompts on the SR results in Fig. 5. We observe that the method can remove part of the noise for the image with severe noise when the prompt indicates [light noise] in the left instance. Conversely, a suitable prompt, i.e., [heavy noise], can restore a more realistic result. Meanwhile, for images at the right, a simplified prompt, i.e., [medium noise], can yield a relatively satisfactory result. Further refining the prompt, i.e., [+light blur], can further improve the restoration outcome. These results demonstrate the flexibility of our prompts.

We investigate the different number of resizing operations in the degradation. The results are presented in Tab. 2. We can find that the model with two resizings performs better. This is because one single resizing is fixed at $\frac{1}{4}$ in the $\times$4 SR task. Introducing an additional resizing allows for variable scales, expands the degradation scope, and enhances the generality of the model.

We investigate the different formats of the degradation and prompt. The results are revealed in Tab. 3. Firstly, in Tab. 3, we compare fixed and random degradation orders. The results indicate that random order slightly lowers performance. It may be because random order expands the degradation space (generalization), thus increasing training complexity and diminishing performance. To balance performance and generalization, we opt for the fixed order shown in Fig. 2.

Secondly, in Tab. 3, we compare three prompt formats. The comparison shows that complete prompts (Original) reveal the best performance. Meanwhile, prompt order has little effect. Moreover, the simplified prompt can yield relatively good results due to the model generalization. Overall, our method exhibits fine generalization, supporting a flexible variety of degradation and prompt.

We study the effects of different content of text prompts. The results are presented in Tab. 5. We compare three types of text prompt content. All experiments are conducted on our proposed PromptSR. The comparison shows that descriptions of degradation (Degradation) are more suitable for the SR task than image content descriptions (Caption). This is consistent with our analysis in Sec. 3.1.2. Additionally, combining both descriptions results in a slight performance drop compared to using degradation prompts alone. This could be due to the disparity between the two descriptions, which hinders the utilization of degradation information provided by text prompts.

We further explore the impact of different text encoders, with the results detailed in Tab. 5. We utilize several pre-trained text encoders: CLIP (Radford et al., 2021) (clip-vit-large) and T5 (Raffel et al., 2020) (T5-small and T5-xl). We discover that models employing different text encoders display varied performance. Applying more powerful language models as text encoders enhances model performance. For instance, T5-xl, compared to T5-small, reduces the LPIPS on the LSDIR and DIV2K validation sets by 0.0109 and 0.0162, respectively. Moreover, it is also notable that the performance of the model is not entirely proportional to the parameter size of the text encoder. Considering both model performance and parameter size, we select CLIP as the text encoder.

We evaluate our method on some synthetic test datasets: Urban100 (Huang et al., 2015), Manga109 (Matsui et al., 2017), LSDIR-Val (Li et al., 2023), and DIV2K-Val (Timofte et al., 2017) in Tab. 6. Our method outperforms others on most perceptual metrics. For instance, compared to the suboptimal model SwinIR-GAN (Liang et al., 2021), our method reduces the LPIPS by 0.0101 on the DIV2K-Val dataset. Meanwhile, compared with DiffBIR (Lin et al., 2024), our PromptSR achieves a reduction in LPIPS by 0.0294 and 0.0220 on LSDIR-Val and DIV2K-Val, respectively. Moreover, for PSNR and SSIM, the two metrics are only used as references, since they do not consistently align well with the image quality (Saharia et al., 2022b). These quantitative results demonstrate that introducing text prompts into image SR can effectively improve performance.

We show some visual comparisons in Fig. 6. We can observe that our proposed PromptSR is capable of restoring clearer and more realistic images, in some challenging cases. This is consistent with the quantitative results. Furthermore, we provide more visual results in the supplementary material.

We present the quantitative comparison on RealSR (Cai et al., 2019) in Tab. 7. Our PromptSR achieves the best performance on most perceptual and aesthetic metrics, including ST-LPIPS, CNNIQA, and NIMA. Meanwhile, it also scores well on LPIPS. These results further demonstrate the superiority of introducing text prompts into image SR tasks.

We present some visual results in Fig. 7. Except for the RealSR dataset, we also conduct an evaluation on the Real45 dataset, collected from the internet. Our proposed method also outperforms other methods on real-world datasets. More comparison are provided in the supplementary material.