When Model Knowledge meets Diffusion Model: Diffusion-assisted Data-free Image Synthesis with Alignment of Domain and Class

We generate a 512$\times$512 resolution image using Stable Diffusion 2.1 (Rombach et al., 2022) with $T=$ 30 diffusion time steps. For applying Domain Alignment Guidance (DAG), we define the gradient scaling factor $\eta$ as the product of $\lambda_{BN}$ and the guidance scale $s_{g}$. Here, $\lambda_{BN}$ is the scaling factor for $\mathcal{L}_{BN}$, and $s_{g}$ denotes the DAG guidance scale. These two parameters play a crucial role in our image generation process, and their impact on synthetic images is discussed in the following subsection. DAG is incorporated with the Classifier-Free Guidance (CFG) with a scale of 15.

To define the Class Alignment Token (CAT), we newly define a single token $S_{c}$ with no inherent meaning and add it to the vocabulary, using its embedding as the initial value. We utilize the token “newcls” as the CAT, but any arbitrary token without meaning can serve as the initial token. To optimize the embedding vector $v_{c}$ of the CAT, we employ the Adam optimizer with a learning rate of 0.005. We train for up to 30 epochs, each involving 20 gradient accumulation steps and generating images from latent noise initialized with different seeds. It is equivalent to a batch size of 20 and can be adjusted based on resource availability.

The main hyperparameter of our proposed Domain Alignment Guidance (DAG) in the diffusion sampling process is $\eta$, which is divided into the gradient flow scale $\lambda_{BN}$ and the DAG guidance scale $s_{g}$ and we can rewrite the Equation 10 as below.

To determine the optimal values for the gradient flow scale($\lambda_{BN}$) and guidance scale($s_{g}$), we conduct experiments across a wide range of values. Specifically, we test $\lambda_{BN}$ between 0.0001 and 1, while the $s_{g}$ is explored within the range of 0.1 to 100. As a result, we find the optimal settings as a $\lambda_{BN}$ of 0.01 and a $s_{g}$ of 20. The experimental results are shown in the Figure 11. Users can adjust these two parameters as needed, with the selection of hyperparameters guided by the generated images’ confidence score.

In our experiments using a single RTX 4090 GPU, the optimal CAT embedding is found in approximately 7.5 minutes. Once determined, this embedding remains fixed, enabling rapid image generation via a diffusion sampling process. Compared to existing Data-Free Image Synthesis (DFIS) methods, our DDIS enhances efficiency and cost-effectiveness for large-scale ImageNet-1k dataset generation while preserving high image quality. To illustrate this, we compare the total optimization iterations needed to synthesize 100,000 ImageNet-1k images on a single RTX 4090 GPU, shown in 4.

DeepInversion (DI) (Yin et al., 2020) requires iterative optimization for each mini-batch. With an optimal batch size of 250, around 20,000 iterations per batch are needed. Generating 100,000 images necessitates repeating this 400 times, totaling 8,000,000 iterations (20,000 iterations per a mini-batch $\times$ 400 batches). PlugInInversion (PII) (Ghiasi et al., 2022) optimizes a random input across seven progressive upsampling stages (7$\times$7 to 224$\times$224), with 400 iterations per stage. Generating 100,000 images with a batch size of 250 results in approximately 1,120,000 total iterations (400 iterations per stage $\times$ 7 stages $\times$ 400 batches).

Our DDIS optimizes only class-wise CAT embedding vectors in a low-dimensional space (1$\times$784). For ImageNet-1k (1000 classes), we perform just 30 iterations per class, totaling 30,000 iterations (30 iterations/class$\times$1000 classes). After finding the CAT embedding, generating 100,000 images involves simply sampling latent vectors without further training or optimization. While CAT optimization has a slightly longer per-iteration time than prior methods, the drastically reduced total iterations make the overall process more efficient.

Since ImageNet-1k (Russakovsky et al., 2015) consists of 1,000 classes, to evaluate whether our DDIS can avoid lexical overlapping issues and accurately align with the correct class, we generate images by inverting the ResNet-34 (Sohl-Dickstein et al., 2015) pretrained on ImageNet-1k provided by TorchVision (Paszke et al., 2019), which has a top-1 accuracy of 73.31%. We generate 10,000 images with 512$\times$512 using the Stable Diffusion 2.1 and resize them to 224$\times$224 for evaluation, matching the typical resolution used during ImageNet-1k training. For comparison, we generate 10,000 images using DeepInversion (DI) (Yin et al., 2020) and PlugInInversion (PII) (Ghiasi et al., 2022), the only Data-Free Image Synthesis (DFIS) studies addressing large-resolution image synthesis, following their official GitHub implementations and proposed their ImageNet-1k parameters. Figure 12 illustrates samples for 30 classes, including those with lexical overlapping issues. The proposed method effectively generates images that precisely match the classes learned by the model through CAT embedding optimization. For example, the class “Yellow Lady’s Slipper” (class 986), which refers to a type of flower, demonstrates how DDIS alleviates lexical overlapping problems to produce accurate class images.

We are the first to address the synthesis of non-photo-specific domain datasets in DFIS. We focus on the PACS dataset synthesis (Li et al., 2017), a benchmark commonly used in domain generalization, which consists of four domains: Photo, Art Painting, Cartoon, and Sketch.We specifically handle the Art Painting and Cartoon domains, as the Photo domain’s excellence is well-described with the ImageNet-1k experiment, and the Sketch domain represents a failure case in our study. Each domain contains seven classes: dog, elephant, giraffe, guitar, horse, house, and person. We fine-tune the ResNet-34 pre-trained on ImageNet-1k with two domains, excluding the training domain data for inference on the remaining three domains to measure top-1 accuracy. The top-1 accuracies for Art Painting and Cartoon pre-trained ResNet-34 models are 54.73% and 61.78%, respectively. We generate 512$\times$512 images using Stable Diffusion 2.1 and resize them to 224$\times$224 for evaluation, aligning with the typical PACS training resolution. For comparison, we generate 3,000 images using DeepInversion (DI), PlugInInversion (PII), and our DDIS method. We reproduce the DI and PII utilizing the official GitHub implementations and ImageNet-1k parameters matching the PACS dataset resolution.

Figure 13 shows the results of synthesizing all classes in the PACS Art Painting domain, while Figure 14 displays the results for the PACS Cartoon domain. We demonstrate that our proposed DAG-based diffusion sampling process and CAT embedding optimization align the target dataset’s domain and class information more effectively than the baseline methods.

To demonstrate the effectiveness of our methodology across a broader range of domains, we utilize Style-Aligned (Hertz et al., 2024) to create high-quality, domain-specific datasets. We select the domains of manga, caricature, and sketch and synthesized 400 images for each of the seven classes within these domains using Stable Diffusion 2.1. The synthesized datasets are illustrated in the Figure 15. Subsequently, we finetune a classifier using a ResNet-34 pre-trained on ImageNet-1k, following the same approach as with ImageNet-1k and PACS experiments. The top-1 accuracies for manga, caricature, and sketch are 85.70%, 76.80%, and 98.90%, respectively.

We demonstrate that in DAG, the gradient of $\mathcal{L}_{BN}$, which aligns synthetic image statistics with the BN layer running statistics, provides stable guidance unaffected by the time step. To avoid confusion, we compare our method with Classifier-Guidance (CG) (Dhariwal & Nichol, 2021), which only provides class guidance at the last time step $T$. First, running statistics within all BN layers are derived from the entire training set, making them more robust and stable than providing guidance based on conditional probability gradients for individual samples. Second, we demonstrate that the BN statistics of generated images remain stable across different time steps. Figure 18 illustrates the layer-wise mean and variance of images generated at every time step. We sample 30 images from the “hero” class in the Style Aligned Manga domain over time steps, passing them through a ResNet-34 pre-trained on the Manga dataset to observe the variation in layer-wise mean over time steps. The layer-wise statistics of noise samples at each time step surprisingly exhibit similar trends from time step 0 to $T$, remaining consistent across different time steps. This result indicates that the model’s internal statistics remain stable regardless of the time step, demonstrating that guidance aligning synthetic image statistics with BN layer running statistics at each time step is highly reliable.

To demonstrate the effectiveness of our Class Alignment Token, we compare our CAT with the domain-wise token. The domain-wise token is trained to observe the overall classes, while class-wise tokens are trained to observe each class independently. Figure 10 illustrates the results for the PACS-cartoon domain when using a single token embedding for all classes. We observe that the domain-wise token embedding captures the domain knowledge of the training set but generates images with ignored class-specific features, averaging across all classes. In contrast, our CAT embedding encodes class-specific information, capturing precise features corresponding to each class.

We conduct a zero-shot image synthesis experiment to evaluate whether Stable Diffusion can effectively generate images for unseen classes. First, using Stable Diffusion V3, we select 10 classes that the Stable Diffusion 2.1 model used in our study has never encountered and generate images for those classes with SD V3. Then, we train a ResNet-50 model using the dataset of these 10 classes (see Figure 19). Finally, we utilize Stable Diffusion 2.1 to synthesize the images used for training ResNet-50 with the our proposed DDIS. Surprisingly, as shown in Figure 20, although Stable Diffusion 2.1 struggles with these classes, the CAT embedding can capture the class attributes from the training set and generate images with a distribution similar to that of the original dataset.

To evaluate the effectiveness of our CAT embedding optimization, we conduct ablation studies on three design choices, synthesizing four lexically ambiguous ImageNet-1K classes: Kite (21), Tiger Cat (282), Beach Wagon (436), and Mail Bag (636) like 5 in the main paper. We compare confidence scores and visual quality under these settings. Firstly, we test three fine-tuning configurations for Stable Diffusion (SD): (b) UNet only, (c) text encoder only, and (d) Full fine-tuning, using the Cross-Entropy (CE) loss used for CAT optimization. As shown in Figures 21 and 5, fine-tuned SD produces distorted, low-confidence images, failing to generate class-aligned images. It suggests that the SD fine-tuning in Data-Free Image Synthesis (DFIS) disrupts the prior knowledge within the SD, degrading image quality. Generally, SD is fine-tuned using real data to adapt to specific domains or styles, but in DFIS, the lack of real images leads to unstable training (right side of Figure 21). Moreover, per-class SD fine-tuning leads to individual SD networks, increasing computational cost. In contrast, our method freezes SD and optimizes a single token, preserving SD’s image prior while efficiently generating high-quality, class-aligned images.

Secondly, we explore whether multiple CAT embeddings improve class expressivity by optimizing embeddings for one to five tokens. As shown in Figures 22 and 6, performance drops with more tokens. Lexical ambiguity persists, and confidence scores drop. In data-free settings with simple prompts (e.g., "A {class}"), more tokens amplify the effect of randomly initialized embeddings, hindering desired class-aligned image generation. Therefore, a single token is sufficient and more effective for encoding class information in DFIS.

Lastly, we test optimizing CAT embedding with BatchNorm (BN) loss alongside vanilla CE loss. As shown in Figures 23 and 7, BN loss negatively affects capturing class semantics by inducing synthetic images toward the averaged statistics of the entire dataset, which causes class mixing and reduces separability. Therefore, optimizing CAT embedding with CE loss alone effectively captures class-specific attributes.

In conclusion, the above design studies validate that desired concepts or complex information can be encoded by optimizing only token embeddings on frozen SD. Building on this, we adopt the idea to DFIS and demonstrate that a single CAT token effectively captures class semantics while preserving SD’s priors. Across all design choices, our approach consistently outperforms alternatives, highlighting its effectiveness for DFIS.