KD-DLGAN: Data Limited Image Generation via Knowledge Distillation

We describes the detailed methodology of the proposed KD-DLGAN in this section. As shown in Fig. 2, we introduce knowledge distillation for data-limited image generation. Specifically, leveraging CLIP [36] as the teacher model, we design two generative knowledge distillation techniques, including aggregated generative knowledge distillation that leads to less distinguishable real-fake samples for the discriminator while distilling more generalizable knowledge from the pretrained model, and correlated generative knowledge distillation that encourages the discriminator to mimic the diverse vision-language correlation in CLIP. The ensuing subsections will describe the problem definition of data-limited image generation, details of the proposed aggregated generative knowledge distillation and correlated generative knowledge distillation, and the overall training objective, respectively.

GANs greatly change the paradigm of image generation. Each GAN consists of a discriminator $D$ and a generator $G$. The general loss function for discriminator and generator is formulated as:

$$\mathcal{L}_{d}(D;x,G(z))=\mathbb{E}[\log(D(x))]+\mathbb{E}[\log(1-D(G(z))]$$

(1)

$$\mathcal{L}_{g}(D;G(z))=\mathbb{E}[\log(1-D(G(z))]$$

(2)

In data-limited image generation, the discriminator in GANs tends to memorize the exact training data and is prone to suffer from overfitting, leading to sub-optimal image generation. Recent studies show that knowledge distillation from powerful and generalizable models can relieve overfitting [9, 2, 29, 42] effectively. However, these state-of-the-art knowledge distillation methods are mainly designed for visual recognition tasks, which cannot be naively adapted to the image generation tasks. We design two novel knowledge distillation techniques that can greatly improve data-limited image generation, more details to be presented in the following subsections.

We design aggregated generative knowledge distillation (AGKD) to mitigate the discriminator overfitting by distilling more generalizable knowledge from the pretrained CLIP [36] model. Thus, we force the discriminator feature space to mimic CLIP visual feature space. Specifically, for real samples $x$, we distill knowledge from CLIP visual feature of $x$ (denoted by $I(x)$) to the discriminator feature of $x$ (last layer feature, which is denoted by $D^{f}(x)$) with L1 loss. Similarly, for generated samples $G(z)$, we also distill knowledge from CLIP visual feature $I(G(z))$ to the discriminator feature $D^{f}(G(z))$. The knowledge distillation loss $\mathcal{L}^{KD}_{AGKD}$ in AKGD can thus be formulated by:

$\displaystyle\left.\begin{aligned} \mathcal{L}^{KD}_{AGKD}=|I(x)-D^{f}(x)|+|I(G(z))-D^{f}(G(z))|\end{aligned}\right.$

AGKD also mitigates the discriminator overfitting by aggregating features of real and fake samples to challenge the discriminator learning. Specifically, we match the CLIP visual feature of real samples $I(x)$ and discriminator features of fake samples $D^{f}(G(z))$, as well as CLIP visual feature of fake samples $I(G(z))$ and discriminator features of real samples $D^{f}(x$) by the L1 loss. Such design lowers the real-fake discriminability and makes it harder to distinguish real-fake samples for the discriminator. The aggregated loss $\mathcal{L}^{AGG}_{AGKD}$ can be formulated by:

$\displaystyle\left.\begin{aligned} \mathcal{L}^{AGG}_{AGKD}=|I(x)-D^{f}(G(z))|+|I(G(z))-D^{f}(x)|\end{aligned}\right.$

For effective GAN training, the designed aggregated loss $\mathcal{L}^{AGG}_{AGKD}$ is controlled by a hyper-parameter $p$, where the loss is applied with probability $p$ or skipped with probability 1-$p$. We empirically set $p$ at 0.7 in our trained networks. Thus, the aggregated loss $\mathcal{L^{\prime}}^{AGG}_{AGKD}$ can be re-formulated by:

$\displaystyle\left.\begin{aligned} \mathcal{L^{\prime}}^{AGG}_{AGKD}=\left\{\begin{array}[]{llllll}\mathcal{L}^{AGG}_{AGKD},&&{\rm if}&q\leq p,\\ 0,&&{\rm if}&q>p,\\ \end{array}\right.\end{aligned}\right.$

where $q$ is a random number sampled between 0 and 1.

The overall AGKD loss $\mathcal{L}_{AGKD}$ can be formulated by:

$\displaystyle\left.\begin{aligned} \mathcal{L}_{AGKD}=\mathcal{L}^{KD}_{AGKD}+\mathcal{L^{\prime}}^{AGG}_{AGKD}\end{aligned}\right.$

(3)

Correlated generative knowledge distillation (CGKD) aims to improve the generation diversity by two steps: 1) it enforces the diverse correlations between generated images and texts in CLIP with a pairwise diversity loss ($\mathcal{L}^{PD}_{CGKD}$); 2) it distills the diverse correlations from CLIP to the GAN discriminator with a distillation loss ($\mathcal{L}^{KD}_{CGKD}$).

To achieve diverse image-text correlations, it first builds the correlations (indicated by inner products) between CLIP visual features of generated images $I(G(z))\in\mathbb{R}^{B\times M}$ and CLIP texts features $T\in\mathbb{R}^{K\times M}$. Here, $B$ is the batch size of generated images, $K$ is the number of texts and $M$ is the features dimension for each text feature or each image feature. For conditional datasets and unconditional datasets, we employ the corresponding image labels as input texts and predefine a set of relevant text labels as input texts, respectively. Details of text selection for our datasets are introduced in the supplementary material. Thus, the correlation $C_{T}\in\mathbb{R}^{B\times K}$ between $I(G(z))$ and $T$ (i.e., their L2-normalized inner products) can be defined as follows:

$$C_{T}=\frac{I(G(z))\cdot T^{\prime}}{||I(G(z))\cdot T^{\prime}||_{2}},$$

where $T^{\prime}$ is the transpose of $T$.

With the defined correlation, the diverse CLIP image-text correlations can be extracted in a pairwise manner. Specifically, for each image-text correlation $C_{T}[i,:]\in\mathbb{R}^{K}$, we diversify it with another image-text correlation $C_{T}[j,:]\in\mathbb{R}^{K}$ by minimizing the cosine similarity between them. Note [$i$,:] or [$j$,:] denotes the $i$-th or $j$-th row in $C_{T}$ and $j\neq i$. The pairwise diversity loss $\mathcal{L}^{PD}_{CGKD}$ can thus be defined as the sum of the cosine similarity of all pairs:

$\displaystyle\left.\begin{aligned} \mathcal{L}^{PD}_{CGKD}=\sum_{i=1}^{K}\sum_{j=1,j\neq i}^{K}Cos(C_{T}[i,:],C_{T}[j,:]),\end{aligned}\right.$

where $Cos(\overrightarrow{a},\overrightarrow{b})$ indicates the cosine similarity between the two vectors $\overrightarrow{a}$ and $\overrightarrow{b}$.

Then, the obtained diverse correlations are distilled from CLIP to the GAN discriminator, aiming to improve the generation diversity. We build the correlations $C_{S}\in\mathbb{R}^{B\times K}$ between discriminator features of generated samples $D^{f}(G(z))\in\mathbb{R}^{B\times M}$ and CLIP text features $T\in\mathbb{R}^{K\times M}$ as follows:

$\displaystyle\left.\begin{aligned} C_{S}=\frac{D^{f}(G(z))\cdot T^{\prime}}{||D^{f}(G(z))\cdot T^{\prime}||_{2}}\end{aligned}\right.$

The correlation distillation from $C_{T}$ to $C_{S}$ is defined by the L1 loss between them:

$\displaystyle\left.\begin{aligned} \mathcal{L}^{KD}_{CGKD}=|C_{T}-C_{S}|\end{aligned}\right.$

The overall CGKD loss $\mathcal{L}_{CGKD}$ can thus be defined by:

$\displaystyle\left.\begin{aligned} \mathcal{L}_{CGKD}=\mathcal{L}^{PD}_{CGKD}+\mathcal{L}^{KD}_{CGKD}\end{aligned}\right.$

(4)

The overall training objective of the proposed KD-DLGAN can thus be formulated by:

$\displaystyle\left.\begin{aligned} \mathop{min}\limits_{G}\mathop{max}\limits_{D}\mathcal{L}_{G}+\mathcal{L}_{D}\end{aligned}\right.$

(5)

where $\mathcal{L}_{G}=\mathcal{L}_{g}$ as introduced in Eq. 2 and $\mathcal{L}_{D}=\mathcal{L}_{d}+\mathcal{L}_{AGKD}+\mathcal{L}_{CGKD}$ as introduced in Eqs. 1, 3 and 4.

We conduct experiments over the following datasets: CIFAR [24], ImageNet [11], 100-shot [53] and AFHQ [39]. Datasets details are provided in the supplementary material. We perform evaluations with Frechet Inception Distance (FID) [16] and inception score (IS) [38].

Table 1 shows unconditional image generation results over 100-shot and AFHQ datasets, where we employ StyleGAN-v2 [22] as the backbone. Following the data settings in DA [53], the models are trained with 100 samples (100-shot Obama, Grumpy Cat, Panda), 160 samples (AFHQ Cat) and 389 samples (AFHQ Dog), respectively.

As Row 3 of Table 1 shows, including the proposed KD-DLGAN into DA [53] achieves superior performance across all data settings consistently as compared with DA alone (in Row 2), demonstrating the complementary relation between KD-DLGAN and DA [53]. In addition, the vanilla knowledge distillation in DA + KD (CLIP [36]) (Row 1) trains the GAN discriminator to mimic the visual feature representation of CLIP. We can observe that KD-DLGAN outperforms DA + KD (CLIP [36]) consistently as well, indicating that the performance gain in KD-DLGAN is largely attributed to our generative knowledge distillation designs instead of solely from the powerful vision-language model. Table 1 also tabulates the results of KD-DLGAN when implementing over four state-of-the-art data-limited generation approaches including LeCam-GAN [40], InsGen [44], APA [19] and ADA [21]. We can see that KD-DLGAN complement all the state-of-the-art consistently, demonstrating the superior generalization and complementary property of our proposed KD-DLGAN.

Fig. 3 shows qualitative comparison with DA [53]. It can be observed that KD-DLGAN clearly outperforms the state-of-the-art in the data-limited generation, especially in term of the generated shapes and textures.

Table 2 and Table 3 show the conditional image generation results on CIFAR-10, CIFAR-100 and ImageNet, respectively. All models employ BigGAN [5] as the backbone. CIFAR-10 and and CIFAR-100 are trained with 100% (50K images), 20% (10K images) or 10% (5K images) training data where the FIDs are evaluated over the validation sets (10K images). ImageNet is trained with 10% (~100K images), 5% (~50K images) and 2.5% (~25K images), where the evaluations are performed over the whole training set (~1.2M images).

The experiments show that our KD-DLGAN outperforms the state-of-the-art substantially. The superior performance is largely attributed to our designed generative knowledge distillation techniques in KD-DLGAN, which mitigates the discriminator overfitting and improves the generation performance effectively. We also show the results of vanilla knowledge distillation from the powerful vision-language model CLIP [36] in Row 1 of Table 2 and Table 3. We can see that KD-DLGAN outperforms the vanilla knowledge distillation method by a large margin, indicating that the performance gain is largely attributed to our generative knowledge distillation design instead of the powerful vision-language model. In addition, Table 2 and Table 3 also show the results of KD-DLGAN when implementing over the state-of-the-art data-limited image generation approaches. KD-DLGAN complements the state-of-the-art and improves the generation performance greatly.

The proposed KD-DLGAN consists of two generative knowledge distillation (KD) techniques, namely, AGKD and CGKD. The two techniques are separately evaluated to demonstrate their contributions to the overall generation performance. As Table 4 shows, including either AGKD or CGKD clearly outperforms DA [53], the state-of-the-art in data-limited image generation, demonstrating the effectiveness of the proposed AGKD and CGKD in mitigating the discriminator overfitting and improving the generation performance. In addition, combining AGKD and CGKD leads to the best generation performance which shows that the two KD techniques complement each other.

Qualitative ablation studies in Fig. 4 show that the proposed AGKD and CGKD can produce clearly more realistic generation than the baseline, demonstrating the effectiveness of these two generative knowledge distillation techniques. In addition, KD-DLGAN combining AGKD and CGKD performs the best, which further verifies that AGKD and CGKD are complementary to each other.

In this subsection, we analyze our KD-DLGAN from several perspectives. All the experiments are based on the CIFAR-10 and CIFAR-100 dataset with 10% data.

Generation Diversity: The proposed KD-DLGAN improves the generation diversity by enforcing the diverse correlation between images and texts, which eventually improves the generation performance. In this subsection, we evaluate the generation diversity with LPIPS [52]. Higher LPIPS means better diversity of generated images. As Table 5 shows, the proposed KD-DLGAN outperforms the baseline DA [53], the state-of-the-art in data-limited image generation, demonstrating the effectiveness of KD-DLGAN in improving generation diversity. We also show the results of KD-DLGAN without CGKD; it further verifies that the improved generation diversity is largely attributed to the CGKD, which distills diverse image-text correlation from CLIP [36] to the discriminator, ultimately improving the generation performance. Note we choose Alexnet model with linear configuration for LPIPS evaluation.

Generalization of KD-DLGAN: We study the generalization of our KD-DLGAN by performing experiments with different GAN architectures, generation tasks and the number of training samples. Specifically, as shown in Table 1-3, we perform extensive evaluations over BigGAN and StyleGAN-v2. Meanwhile, we benckmark KD-DLGAN on object generation tasks with CIFAR and ImageNet and face generation tasks with 100-shot and AFHQ. Besides, we perform extensive evaluations on 100-shot and AFHQ with few hundred samples, CIFAR with 100%, 20% and 10% data, ImageNet with 10%, 5% and 2.5% data.

Comparison with state-of-the-art knowledge distillation methods: KD-DLGAN is the first to explore the idea of knowledge distillation in data-limited image generation. To validate the superiority of our designs, we compare KD-DLGAN with state-of-the-art knowledge distillation methods designed for other tasks in Table 6. It shows that our KD-DLGAN outperforms the state-of-the-art knowledge distillation approaches consistently by large margins. The superior generation performance demonstrates the effectiveness of our designed generative knowledge distillation techniques for data-limited image generation.

Comparison with other Vision-Language teacher models: KD-DLGAN adopted CLIP [36] as the teacher model for knowledge distillation. We perform experiments to study how different vision-language models affect the generation performance. As shown in Table 7, different vision-language models produce quite similar FIDs. We conjecture that these pertrained models provide sufficient vision-language information for distillation and the performance gain mainly comes from our designed generative knowledge distillation techniques instead of the selected teacher model.

Comparison with other CLIP-based methods: KD-DLGAN distills knowledge from CLIP [36] to the discriminator with two novelly designed generative knowledge distillation techniques while Vision-aided GAN [25] employs off-the-shelf models as additional discriminators for data-limited generation. Table 8 compares KD-DLGAN with Vision-aided GAN. We can observe that KD-DLGAN outperforms CLIP-based Vision-aided GAN consistently, demonstrating its effectiveness in mitigating discriminator overfitting and improving generation performance.