FreCaS: Efficient Higher-Resolution Image
Generation via Frequency-aware
Cascaded Sampling

Diffusion models have gained significant attentions due to their abilities to generate high-quality natural images. Ho et al. (2020) pioneered the use of a variance-preserving diffusion process to bridge the gap from natural images to pure noises. Dhariwal & Nichol (2021) exploited various network architectures and achieved superior image quality than contemporaneous GAN models. Ho & Salimans (2022) introduced a novel classifier-free guidance strategy that attains both generated image quality and diversity. However, the substantial model complexity makes high-resolution image synthesis challenging. Ho et al. (2022) proposed a novel cascaded framework that progressively increases image resolutions. Rombach et al. (2022) performed the diffusion process in the latent space of a pre-trained autoencoder, enabling high-resolution image synthesis with reduced computational cost. (Esser et al., 2024) presented SD3, which employs the rectified flow matching (Lipman et al., 2022; Liu et al., 2022) at the latent space and demonstrates superior performance. Despite the great progress, it still requires substantial efforts to train a high-resolution diffusion model from scratch. Therefore, training-free higher-resolution image synthesis attracts increasing attentions.

A few methods have been developed to leverage pre-trained diffusion models to generate images of higher resolutions than the training size. MultiDiffusion (Bar-Tal et al., 2023) is among the first methods to bind multiple diffusion processes into one unified framework and generates seamless higher-resolution images. However, the results exhibit unreasonable image structures such as duplicated objects. AttnEntropy (Jin et al., 2024) alleviates this problem by re-normalizing the entropy of attention blocks during sampling. On the other hand, ScaleCrafter (He et al., 2023) and FouriScale (Huang et al., 2024) expand the receptive fields of pre-trained networks to match higher inference resolutions, thereby demonstrating improved image quality. HiDiffusion (Zhang et al., 2023) dynamically adjusts the feature sizes to match the training dimensions. DemoDiffusion (Du et al., 2024) and AccDiffusion (Lin et al., 2024) first generate a reference image at standard resolutions and then use this image to guide the generation of images at higher resolutions. Despite their success, the above mentioned approaches neglect the coarse-to-fine nature of image generation and generate image contents of all frequencies simultaneously, resulting in long inference latency and limiting their broader applications.

To address this issue, we propose an efficient FreCaS framework for training-free higher-resolution image synthesis. FreCaS divides the entire sampling process into stages of increasing resolutions, gradually synthesizing components of different frequency bands, thereby reducing the unnecessary computation of handling premature high-frequency details in early sampling stages. It is worth noting that DemoFusion (Du et al., 2024) and ResMaster (Shi et al., 2024) also employ a cascaded sampling scheme. However, there exist fundamental differences between FreCaS and them: DemoFusion and ResMaster perform a complete diffusion process at each resolution, whereas FreCaS transitions the diffusion from low to high resolutions in just one process. This distinction makes our method significantly more efficient than them while achieving better image quality.

Diffusion models. Diffusion models (Ho et al., 2020; Dhariwal & Nichol, 2021) transform complex image distributions into the Gaussian distribution, and vice versa. They gradually inject Gaussian noises into the image samples, and then use a reverse process to remove noises from them, achieving image generation. Most recent diffusion models operate in the latent space and utilize a discrete timestep sampling process to synthesize images. Specifically, for a $T$-step sampling process, a latent noise ${\bm{z}}_{T}$ is drawn from a standard Gaussian distribution, and then iteratively refined through a few denoising steps until converged to the clean signal latent ${\bm{z}}_{0}$. Finally, the natural image ${\bm{x}}$ is decoded from ${\bm{z}}_{0}$ using a decoder $\mathcal{D}$. The whole process can be written as follows:

$${\bm{z}}_{T}\sim\mathcal{N}(\textbf{0},\textbf{I})\rightarrow{\bm{z}}_{T-1}\rightarrow\cdots\rightarrow{\bm{z}}_{\text{1}}\rightarrow{\bm{z}}_{\text{0}}\rightarrow{\bm{x}}=\mathcal{D}({\bm{z}}_{\text{0}}).$$

(1)

For each denoising step, current works typically adopt the classifier-free guidance (CFG) (Ho & Salimans, 2022) to improve image quality. It predicts an unconditional denoising score ${\bm{\epsilon}}_{unc}$ and a conditional denoising score ${\bm{\epsilon}}_{{c}}$. The final denoising score is obtained via a simple extra-interpolation process as $\hat{{\bm{\epsilon}}}=(1-w)\cdot{\bm{\epsilon}}_{unc}+w\cdot{\bm{\epsilon}}_{c}$, where $w$ denotes the guidance strength.

Pixel space diffusion models exhibit a coarse-to-fine behavior in the image synthesis process (Rissanen et al., 2023; Teng et al., 2024). In this section, we show that such a behavior is also exhibited for latent diffusion models during the sampling process, which inspires us to develop a frequency-aware cascaded sampling framework for generating higher-resolution images.

PSD curves in latent space. The power spectral density (PSD) is a powerful tool for analyzing the energy distribution of signals along the frequency spectrum. Rissanen et al. (2023) and Teng et al. (2024) have utilized PSD to study the behaviour of intermediate states in the pixel diffusion process. Here, we compute the PSD of the latent signals over a collection of 100 natural images using the pre-trained SDXL model (Podell et al., 2023). Figure 1 shows the PSD curves of ${\bm{z}}_{\text{900}}$, ${\bm{z}}_{\text{600}}$, ${\bm{z}}_{\text{300}}$ and ${\bm{z}}_{\text{0}}$. The solid line denotes the PSD curve of intermediate noise corrupted latent, while the dashed line represents the PSD of Gaussian noise corrupted into the latent. The inner area between the two curves (marked with red slashes) indicates the energy of clean signal latent being synthesized. One can see that the clean image signals emerge from the low-frequency band (see ${\bm{z}}_{\text{900}}$ and ${\bm{z}}_{\text{600}}$) and gradually expand to the high-frequency band (see ${\bm{z}}_{\text{300}}$ and ${\bm{z}}_{\text{0}}$) during the sampling process. These observations confirm the coarse-to-fine nature of image synthesis in the latent diffusion process, where low-frequency content is generated first, followed by high-frequency details.

Framework of FreCaS. Based on the above observation, we developed an efficient FreCaS framework to progressively generate image contents of higher frequency bands, reducing unnecessary computations in processing premature high-frequency details in early diffusion stages. As shown in Figure 2(a), our FreCaS divides the entire $T$-step sampling process into $N+1$ stages of increasing resolutions. The initial stage performs the sampling process at the default training size $s_{\text{0}}$ with a frequency range of [0, $\frac{s_{\text{0}}}{2}$]. Each of the subsequent stages increases the sampling size to its predecessor, gradually expanding the frequency domain. At the final stage, the latent reaches the target resolution $s_{N}$, achieving a full frequency range from 0 to $\frac{s_{N}}{2}$.

Specifically, we begin with a pure noise latent ${\bm{z}}^{s_{\text{0}}}_{T}$ at stage $s_{\text{0}}$, and iteratively perform reverse sampling until obtaining the last latent in this stage, denoted by ${\bm{z}}^{s_{\text{0}}}_{L}$. Next, we transition ${\bm{z}}^{s_{\text{0}}}_{L}$ to the first latent, denoted by ${\bm{z}}^{s_{\text{1}}}_{F}$, in next stage, as illustrated by the blue dashed arrow in Figure 2(a). This procedure is repeated until the latent feature reaches the target size, resulting in ${\bm{z}}^{s_{N}}_{\text{0}}$. The final image ${\bm{x}}$ is obtained by applying the decoder to ${\bm{z}}^{s_{N}}_{\text{0}}$ so that ${\bm{x}}=\mathcal{D}({\bm{z}}^{s_{N}}_{\text{0}})$. With such a sampling pipeline, FreCaS ensures a gradual refinement of details across coarse-to-fine scales, ultimately producing a high-quality and high-resolution image with minimum computations.

For the transition between two adjacent stages, we perform five steps to convert the last latent of previous stage ${\bm{z}}^{s_{i-1}}_{L}$ to the first latent of next stage ${\bm{z}}^{s_{i}}_{F}$:

$${\bm{z}}^{s_{i-1}}_{L}\xrightarrow{\text{denoise}}\hat{\bm{z}}^{s_{i-1}}_{\text{0}}\xrightarrow{\text{decode}}\hat{\bm{x}}^{s_{i-1}}\xrightarrow{\text{interpolate}}\hat{\bm{x}}^{s_{i}}\xrightarrow{\text{encode}}{\bm{z}}^{s_{i}}_{0}\xrightarrow{\text{diffuse}}{\bm{z}}^{s_{i}}_{F},$$

(2)

where “denoise” and “diffuse” are standard diffusion operations, “decode” and “encode” are performed using the decoder and encoder, respectively, and “interpolation” adjusts the resolutions using the bilinear interpolation. To determine the timestep of ${\bm{z}}^{s_{i}}_{F}$, we follow previous works (Hoogeboom et al., 2023; Chen, 2023; Gu et al., 2023; Teng et al., 2024) to keep the signal-to-noise ratio (SNR) equivalence between ${\bm{z}}_{{L}}^{s_{i-1}}$ and ${\bm{z}}_{F}^{s_{i}}$. Please refer to Appendix A for more details.

Our FreCaS framework progressively transitions the latents to stages with higher resolutions and extended high-frequency bands. To ensure that the diffusion models focus more on generating contents of newly introduced frequencies, we propose a novel FA-CFG strategy, which assigns higher guidance strength to the new frequency components. In FreCaS, upon transitioning to stage $s_{i}$, the latent increases its resolution from $s_{i-1}$ to the higher resolution $s_{i}$, thereby expanding the frequency band from [0, $\frac{s_{i-1}}{2}$] to [0, $\frac{s_{i}}{2}$]. This inspires us to divide the latents into two components: a low-frequency component ranging from [0, $\frac{s_{i-1}}{2}$] and a high-frequency component covering the frequency interval ($\frac{s_{i-1}}{2}$, $\frac{s_{i}}{2}$]. The former preserves the generated contents from previous stages, whereas the latter is reserved for the contents to be generated in this stage. Our goal is to encourage the diffusion models to generate natural details and textures in the newly expanded frequency band.

To achieve the above mentioned goal, we propose to perform CFG on the two frequency-aware parts with different guidance strengths. The entire process is illustrated in Figure 2(b). First, we obtain the unconditional denoising score $\mathbf{{\bm{\epsilon}}}_{unc}$ and conditional denoising score $\mathbf{{\bm{\epsilon}}}_{c}$ using the pre-trained diffusion network. Then, we split the scores into a low-frequency part and a high-frequency part. The former is extracted by downsampling the scores and then resizing them back, while the latter is the residual by subtracting the low-frequency part from the original denoising scores. Subsequently, we apply the CFG strategy to the two parts with different weights. Specifically, for the low-frequency part, we assign the normal guidance strength $w_{l}$, while for the high-frequency part, we use a much higher weight $w_{h}$ to prioritize content generation in this frequency band. The final denoising score is obtained by summing up the two parts. This process can be expressed as:

$$\hat{{\bm{\epsilon}}}=\hat{\bm{\epsilon}}^{l}+\hat{\bm{\epsilon}}^{h}=(1-w_{l})\cdot{\bm{\epsilon}}^{l}_{unc}+w_{l}\cdot{\bm{\epsilon}}^{l}_{c}+(1-w_{h})\cdot{\bm{\epsilon}}^{h}_{unc}+w_{h}\cdot{\bm{\epsilon}}^{h}_{c},$$

(3)

where $\hat{\bm{\epsilon}}^{l}$ and $\hat{\bm{\epsilon}}^{h}$ are the low-frequency and high-frequency parts of $\hat{\bm{\epsilon}}$, respectively. Similarly, ${\bm{\epsilon}}^{l}_{unc}$, ${\bm{\epsilon}}^{h}_{unc}$, ${\bm{\epsilon}}^{l}_{c}$ and ${\bm{\epsilon}}^{h}_{c}$ follow the same notation.

When applied to higher resolutions, pre-trained diffusion models often present unreasonable image structures, such as duplicated objects. To address this issue, we propose to reuse the CA-maps from the previous stage to maintain layout consistency across stages. The CA-maps represent attention weights from cross-attention interactions between spatial features and textual embeddings, effectively capturing the semantic layout of the generated images. Specifically, we average the CA-maps of all cross-attention blocks when predicting ${\bm{z}}^{s_{i-1}}_{L}$ at stage $s_{i-1}$. After transitioning to stage $s_{i}$, we replace the current CA-maps of each cross-attention block using its linear interpolation with the averaged CA-maps $\overline{M}^{s_{i-1}}_{{L}}$ as follows:

$$M^{s_{i}}_{t}=(1-w_{c})\cdot M^{s_{i}}_{t}+w_{c}\cdot\overline{M}^{s_{i-1}}_{{L}},$$

(4)

where $M^{s_{i}}_{t}$ is the CA-maps at step $t$ of stage $s_{i}$. In this way, FreCaS can effectively maintain content consistency and prevent unexpected objects or textures during higher-resolution image generation.

Implementation details. We evaluate FreCaS on three widely-used pre-trained diffusion models: SD2.1 (Rombach et al., 2022), SDXL (Podell et al., 2023) and SD3 (Esser et al., 2024). The sizes of generated images are $\times 4$ and $\times 16$ the original training size. Specifically, we generate images of 1024$\times$1024 and 2048$\times$2048 for SD2.1, while $2048\times 2048$ and $4096\times 4096$ for SDXL. For SD3, we only generate images of $2048\times 2048$ due to the GPU memory limitation. We randomly select 10K, 5K, and 1K prompts from the LAION5B aesthetic subset for generating images of 1024$\times$1024, 2048$\times$2048, and 4096$\times$4096, respectively. We follow the default settings and perform a 50-step sampling process with DDIM sampler for SD2.1 and SDXL, and perform a 28-step sampling process with a flow matching based Euler solver for SD3. For $\times 4$ experiments, we employ two sampling stages at the training size and target size, respectively. For $\times 16$ experiments, we employ three sampling stages at the training size, $4\times$ training size and $16\times$ training size, respectively. More details can be found in Appendix B.

Evaluation metrics. We employ the Fréchet Inception Distance (FID) (Heusel et al., 2017) and Inception Score (IS) (Salimans et al., 2016) to measure the quality of generated images. Following He et al. (2023), we also employ $\text{FID}_{b}$ as the metric, which is computed on the samples of training size and target size. As suggested by Du et al. (2024), we report $\text{FID}_{p}$ and $\text{IS}_{p}$, which compute the metrics at patch level, to better evaluate the image details. The CLIP score (Radford et al., 2021) is utilized to measure the text prompt alignment of generated images. As in previous works (Zhang et al., 2023), we measure the model latency on a single NVIDIA A100 GPU with a batch size of 1. We generate five images and report the averaged latency of the last three images for all methods. Moreover, we conduct a user study and employ the non-reference image quality assessment metrics to further evaluate our FreCaS. Please refer to Appendix C for the details.

For experiments on SD2.1, we compare FreCaS with DirectInference, MultiDiffusion (Bar-Tal et al., 2023), AttnEntropy (Jin et al., 2024), ScaleCrafter (He et al., 2023), FouriScale (Huang et al., 2024) and HiDiffusion (Zhang et al., 2023). For experiments on SDXL, we compare with DirectInference, AttnEntropy, ScaleCrafter, FouriScale, HiDiffusion, AccDiffusion (Lin et al., 2024) and DemoFusion (Du et al., 2024). We further compare our FreCaS with training-based methods (Pixart-Sigma (Chen et al., 2024) and UltraPixel (Ren et al., 2024)) and super-resoution methods (ESRGAN (Wang et al., 2021) and SUPIR (Yu et al., 2024)) in Appendix D.

Quantitative results. Table 1 presents quantitative comparisons for $\times 4$ and $\times 16$ generation between FreCaS and its competitors. We can see that FreCaS not only outperforms other methods on synthesized image quality but also exhibits significantly faster inference speed. In specific, FreCaS achieves the best FID scores in all experiments of SD2.1 and SDXL, achieving clear advantages over the other methods. In terms of the IS metric, FreCaS performs the best in most cases, only slightly lagging behind DemoFusion on the $\times 16$ experiments of SDXL. (Note that DirectInference and MultiDiffusion occasionally achieve higher $\text{FID}_{p}$ and $\text{IS}_{p}$ scores because they disregard the issue of duplicated objects.) For CLIP score, FreCaS obtains the best results on 3 out of the 4 cases, except for the less challenging $\times 4$ generation with SD2.1.

While having superior image quality metrics, FreCaS demonstrates impressive efficiency. It shows more than $2\times$ speedup over DirectInference on $\times 4$ generation experiments, and shows more than $3.6\times$ speedup on the $\times 16$ generation experiments. DemoFusion, which is overall the second best method in terms of image quality, is significantly slower than FreCaS. Its latency is about $6\times$ and $7.5\times$ longer than FreCas on $\times 4$ and $\times 16$ experiments, respectively. On the other hand, HiDiffusion, which is the second faster method, sacrifices image quality for speed. For example, on the $\times 16$ experiment with SD2.1, HiDiffusion achieves a latency of 18.22s but its $\text{FID}_{b}$ score is 34.17. In contrast, FreCaS is faster (13.35s) and has a much better $\text{FID}_{b}$ score (20.11).

Qualitative results. Figure 3 illustrates visual comparisons between FreCaS and competitive approaches. From top to bottom are four groups of examples, presenting the results of $\times 4$ generation of SD2.1, $\times 16$ generation of SD2.1, $\times 4$ generation of SDXL, and $\times 16$ generation of SDXL, respectively. In each group, the top row shows the generated images, while the bottom row shows the zoomed region for better observation. From Figure 3, we can see that FreCaS effectively synthesizes the described contents while maintaining a coherent scene structure. DirectInference, MultiDiffusion and AttnEntropy often produce duplicated objects, such as the many astronauts and warriors. ScaleCrafter and HiDiffusion achieve reasonable image contents in experiments of SD2.1 but generate unnatural layouts in the experiments of SDXL, such as the excessive flowers on the ceiling in $\times 4$ experiment. Our FreCaS consistently maintains coherent image contents and layout in experiments of both SD2.1 and SDXL. AccDiffusion and DemoFusion also achieve natural image contents, but FreCaS generates clearer details such as the flowers and trains. Please refer to Appendix E for more visual results, including images with other aspect ratios.

SD3 (Esser et al., 2024) adopts a rather different network architecture from SD2.1 and SDXL, and many existing methods cannot be applied. We can only compare FreCaS with DirectInference and DemoDiffusion Du et al. (2024). Due to page limitation, please refer to Appendix F for the results.

In this section, we conduct ablation studies on $\times 4$ experiments of SDXL to investigate the effectiveness and settings of our cascaded framework, FA-CFG and CA-maps strategies.

Effectiveness of each component. We conduct a series of ablation studies to better demonstrate the effectiveness of each component of FreCaS, including the cascaded sampling framework, FA-CFG and CA-maps reutilization strategies. Please refer to Appendix G for more details.

FA-CFG strategy. The FA-CFG strategy aims to guide the model to generate content within the expanded frequency band. To achieve this, FA-CFG introduces two parameters, $w_{l}$ and $w_{h}$, to adjust the guidance strength on the low and high frequency components, respectively. When $w_{l}=w_{h}$, the FA-CFG strategy degenerates to the conventional CFG approach. We conduct a series of experiments to explore the optimal settings of the two parameters. First, we fix $w_{l}$ at 7.5 and vary $w_{h}$. The results are shown in Figure 4(a). We observe that as $w_{h}$ increases from 1.0 to 45, the $\text{FID}_{b}$ and $\text{FID}_{p}$ metrics initially decrease, indicating improved image quality. However, as $w_{h}$ becomes too high, the metrics begin to deteriorate. The sweet spot lies between 25 and 35, achieving a low $\text{FID}_{b}$ of nearly 12.65 and a low $\text{FID}_{p}$ of 17.91. We then fix $w_{h}$ at 35 and vary $w_{l}$. The results are presented in Figure 4(b). Reducing $w_{l}$ below 7.5 leads to a slight increase in $\text{FID}_{p}$ from 17.91 to 18.06, whereas increasing $w_{l}$ over 7.5 deteriorates $\text{FID}_{r}$ from 12.81 to 13.00. Compared to $w_{h}$, adjusting $w_{l}$ brings much smaller effects on those two metrics. Thus, we set $w_{l}$ to 7.5 for experiments on SD2.1 and SDXL, and set it to 7.0 for SD3.

Figure 5 provides visual examples of adjusting $w_{h}$. Increasing $w_{h}$ enhances the sharpness of details, such as clearer hair strands and more vivid flower petals. However, an excessively high value of $w_{h}$ (e.g., 45) will introduce artifacts, as highlighted by the red boxes in the figure. This underscores the importance of selecting an appropriate $w_{h}$ value to strike a balance between detail enhancement and artifact suppression. Based on these findings, we set $w_{l}$ to 7.5 and $w_{h}$ to 35 yields favorable outcomes in most of the cases.

CA-maps re-utilization. To evaluate the effect of weight $w_{c}$ in the re-utilization of CA-maps, we conduct an ablation study by varying $w_{c}$ from 0 to 1. The results are shown in Figure 4(c). Increasing $w_{c}$ continuously decreases $\text{FID}_{b}$ but increases $\text{FID}_{p}$, indicating an improvement on the image layout but a drop on image details. To balance between the two metrics, we set $w_{c}=0.6$. A visual example is shown in Figure 6. We see that this setting leads to a clearer textures on strawberry compared to $w_{c}=1.0$ and prevents the unreasonable surface of the blueberry in $w_{c}=0.0$.

Inference schedule. FreCaS uses two factors to adjust the inference schedule. The first one is the count of additional stages $N$. The second factor is the timestep $L$ of last latent in each stage. We conduct experiments on the selection of these two factors. The details can be found in Appendix G. Based on results, we set $L$ to $200$, and set $N$ to 2 for $\times 4$ experiments and 3 for $\times 16$ experiments.