Mode-Aware Continual Learning for Conditional Generative Adversarial Networks

We consider a conditional generative adversarial network (cGAN) that is trained on a set $S$ of source generative tasks, where each task represents a distinct class of data. The cGAN consists of two key components: the generator $\mathcal{G}$ and the discriminator $\mathcal{D}$. Each source generative task $a\in S$, which is characterized by data $X_{a}$ and its labels $y_{a}$, corresponds to a specific mode in the well-trained generator $\mathcal{G}$. Let $X_{b}$ denote the incoming target data. Here, we propose a new mode-affinity measure, called Discriminator-based Mode Affinity Score (dMAS), to showcase the complexity involved in transferring knowledge between different modes in cGAN. This measure involves computing the expectation of Hessian matrices computed from the discriminator’s loss function. To calculate the dMAS, we begin by feeding the source data $X_{a}$ into the discriminator $\mathcal{D}$ to compute the corresponding loss. By taking the second-order derivative of the discriminator’s loss with respect to the input, we obtain the source Hessian matrix. Similarly, we repeat this process using the target data $X_{b}$ as input to the discriminator, resulting in the target Hessian matrix. These matrices offer valuable insights into the significance of the model’s parameters concerning the desired data distribution. The dMAS is defined as the Fréchet distance between these Hessian matrices.

To simplify Equation (1), we approximate the Hessian matrices with their normalized diagonals as computing the full Hessian matrices in the large parameter space of neural networks can be computationally expensive. Hence, dMAS can be expressed as follows:

The procedure to compute dMAS is outlined in function dMAS() in Algorithm 1. Our metric spans a range from $0$ to $1$, where $0$ signifies a perfect match, while $1$ indicates complete dissimilarity. It is important to note that dMAS exhibits an asymmetric nature, reflecting the inherent ease of knowledge transfer from a complex model to a simpler one, as opposed to the reverse process.

In contrast to the statistical biases observed in metrics such as IS (Salimans et al., 2016) and FID (Heusel et al., 2017) (Chong & Forsyth, 2020), dMAS is purposefully crafted to cater to our specific scenario of interest. It takes into account the state of the cGAN model, encompassing both the discriminator and the generator. This sets it apart from FID, which uses the GoogleNet Inception model to measure the Wasserstein distance to the ground truth distribution. Consequently, it falls short in evaluating the quality of generators and discriminators. Instead of assessing the similarity between Gaussian-like distributions, our proposed dMAS quantifies the Fisher Information distance between between the model weights. Thus, it accurately reflects the current states of the source models. Furthermore, FID has exhibited occasional inconsistency with human judgment, leading to suboptimal knowledge transfer performance (Liu et al., 2018; Wang et al., 2018). In contrast, our measure aligns more closely with human intuition and consistently demonstrates its reliability. It is important to emphasize that dMAS is not limited to the analysis of image data samples; it can be effectively applied to a wide range of data types, including text and multi-modal datasets.

We utilize the discriminator-based mode affinity score (dMAS) to continual learning for image generation. The goal is to train a lifelong learning cGAN model to learn new modes while avoiding catastrophic forgetting of existing modes. Consider a scenario where each generative task represents a distinct class of data. At time $t$, the cGAN model has $S$ modes corresponding to $S$ learned tasks. Here, we introduce a mode-aware continual learning framework that allows the model to add a new mode while retaining knowledge from previous modes. We begin by embedding the numeric label of each data sample in cGAN, using an embedding layer in both the generator $\mathcal{G}$ and the discriminator $\mathcal{D}$ models. We then modify cGAN to enable it to take a linear combination of label embeddings for the target data. These label embeddings correspond to the most relevant modes, and the weights for these embedding features are associated with the computed dMAS from the related modes to the target. This enables cGAN to add a new target mode while maintaining all existing modes. Let $\texttt{emb}()$ denote the embedding layers in the generator $\mathcal{G}$ and the discriminator $\mathcal{D}$, and $C$ be the set of the relevant modes, $C=\{i_{1}^{*},i_{2}^{*},\ldots,i_{n}^{*}\}$. The computed mode-affinity scores from these modes to the target are denoted as $s_{i}^{*}$. Let $\sum s_{i}^{*}$ denote the total distance from all the relevant modes to the target. The label embedding for the target data samples is described as follows:

To add the new target mode without forgetting the existing learned modes, we use the target data with the above label embedding to fine-tune the cGAN model. Additionally, we utilize memory replay (Robins, 1995; Chenshen et al., 2018) to prevent catastrophic forgetting. Particularly, samples generated from relevant existing modes are used to fine-tune cGAN. The overview of the proposed approach is illustrated in Figure 1. During each iteration, training with the target data and replaying relevant existing modes are jointly implemented using an alternative optimization process. The detail of the framework is provided in Algorithm 1. By applying the closest modes’ labels to the target data samples in the embedding space, we can precisely update part of cGAN without sacrificing the generation performance of other existing modes. Overall, utilizing knowledge from past experience helps enhance the performance of cGAN in learning new modes while reducing the amount of the required training data samples. Next, we provide a theoretical analysis of our proposed method.

In the above theorem, the introduction of a new mode through mode injection inherently involves a trade-off between the mode-adding ability and the potential performance loss compared to the original model. In essence, when incorporating a new mode, the performance of existing modes cannot be improved. The detailed proof of Theorem 1 is provided in Appendix B.

In the first experiment, the ten generative tasks are defined based on the MNIST dataset, where each task corresponds to generating a specific digit (i.e., $0,1,\ldots,9$). For instance, task 0 aimed at generating images representing digit 0, while task 1 aimed at generating images depicting digit 1. The cGAN model was trained to generate images from the nine source tasks while considering one task as the target task. The cGAN model has nine modes corresponding to nine source tasks. The well-trained generator of this cGAN model served as the representation network for the source data. To evaluate the consistency of the closest modes for each target, we conduct $10$ trial runs, in which the source cGAN model is initialized randomly. The mean and standard deviation of the mode-affinity scores between each pair of source-target modes are shown in Figure 2 (a) and Figure 6, respectively. In the mean table from Figure 2 (a), the columns denote the distance from each source mode to the given target. For instance, the first column indicates that digits $6$ and $9$ are closely related to the target digit $0$. Similarly, the second column shows that digits $4$ and $7$ are the closest tasks to the incoming digit $1$. The standard deviation table from Figure 6 indicates that the calculated distance is stable, as there are no overlapping fluctuations and the orders of similarity between tasks are preserved across $10$ runs. In other words, this suggests that the tendency of the closest modes for each target remains consistent regardless of the initialization of cGAN. Thus, the computed mode-affinity score demonstrates consistent results. Additionally, we provide the atlas plot in Figure 3(a) which gives an overview of the relationship between the digits based on the computed distances. The plot reveals that digits $1,4,7$ exhibit a notable similarity, while digits $0,6,8$ are closely related. This plot provides a useful visualization to showcase the pattern and similarity among different digits.

Analogously, we define ten generative tasks for the CIFAR-10 dataset, each corresponding to a specific object, such as airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. Following the previous experiment, one task is designated as the target task, while the others are the source tasks used to train the cGAN model for image generation. The generator of the well-trained cGAN model serves as the representation network for the source tasks. We present the mean and standard deviation of computed mode-affinity scores between each pair of source-target modes over $10$ trial runs in Figure 2(b) and Figure 7, respectively. The mean table in Figure 2(b) shows the average distance of each source mode from the target (e.g., trucks are similar to automobiles, and cats are closely related to dogs). The standard deviation table in Figure 7 demonstrates the stability of the results across different initialization of cGAN. The consistent findings suggest that the computed distances from the CIFAR-10 dataset are reliable. Furthermore, in Figure 3(b), we include the atlas plot which presents an overview of the relationship between the objects based on the computed mode-affinity scores. The plot reveals that automobile, truck, ship, and airplane have a strong connection, while the other classes (i.e., bird, cat, dog, deer, horse, frog) also exhibit a significant resemblance.

Next, the CIFAR-100 dataset is utilized to define ten target tasks, each corresponding to a specific image class, such as bear, leopard, lion, tiger, wolf, bus, pickup truck, train, streetcar, and tractor. For this experiment, the cGAN model is trained on the entire CIFAR-10 dataset to generate images from ten classes or modes. Figure 2(c) and Figure 8 respectively display the mean and standard deviation of the computed mode-affinity scores between the source-target modes. The mean table in Figure 2(c) indicates the average distance from each CIFAR-10 source mode to the CIFAR-100 target mode. Notably, the target tasks of generating bear, leopard, lion, tiger, and wolf images are closely related to the group of cat, deer, and dog. Specifically, cat images are closely related to leopard, lion, and tiger images. Moreover, the target modes of generating bus, pickup truck, streetcar, and tractor images are highly related to the group of automobile, truck, airplane, and ship. The standard deviation table in Figure 8 also indicates that the computed distances are consistent across different trial runs, demonstrating the stability of the computed mode-affinity scores. Thus, the distances computed between the CIFAR-100 modes and the CIFAR-10 modes are reliable and consistent. In addition, Figure 3(c) includes an atlas plot that provides a visual representation of the relationships between objects based on the computed distances. The plot reveals a strong connection between the vehicles, such as tractors, trucks, and trains, as well as a notable closeness between the animal classes, such as lions, tigers, cats, and dogs. This plot serves as a useful tool for visualizing the similarity and relationship between different objects in the CIFAR-10 and CIFAR-100 datasets and can help to identify relevant data classes for the target class.

Additionally, we conducted a knowledge transfer experiment to assess the effectiveness of the proposed mode affinity score in transfer learning scenarios utilizing the MNIST, CIFAR-10, and CIFAR-100 datasets. In these experiments, we designated one data class as the target, while the remaining nine classes served as source tasks. Our method first computes the dMAS distance from the target to each source task. After identifying the closest task, we fine-tune the cGAN model using the target data samples with the label from the closest task. This approach helps the cGAN model to update the specific part of the model to efficiently and quickly learn the target task. The transfer learning framework is illustrated in Figure 5 and the pseudo-code is provided in Algorithm 2. The image generation performance in terms of FID scores are presented in Tables 1 and  3. Notably, our utilization of dMAS for knowledge transfer consistently outperforms the baseline methods, including Individual Learning, Sequential Fine-tuning, and Multi-task Learning. Remarkably, our approach achieves superior results while utilizing only 10% of the target data samples. When comparing our method with the FID transfer learning approach (Wang et al., 2018), we observe similar performance in most scenarios. However, in the CIFAR-100 experiment, where the target class is bus images, we intentionally trained the source model using only a limited number of truck samples, leading to an inadequately trained source model. Consequently, FID still regards the class of truck images as the closest to the target, resulting in less efficient knowledge transfer. In contrast, our method takes into account the state of the models and selects the task closest to automobile images, resulting in more effective knowledge transfer. Table 1 clearly demonstrates that our approach outperforms FID transfer learning in 10-shot, 20-shot, and 100-shot scenarios.

We apply the computed mode-affinity scores between generative tasks in the MNIST, CIFAR-10, and CIFAR-100 datasets to the mode-aware continual learning framework. In each dataset, we define two target tasks for continual learning scenarios and consider the remaining eight classes as source tasks. Particularly, (digit 0, digit 1), (truck, cat), and (lion, bus) are the targets for the MNIST, CIFAR-10, and CIFAR-100 experiments, respectively. The cGAN model is trained to sequentially update these target tasks. Here, we select the top-2 closest modes to each target and leverage their knowledge for quick adaptation in learning the target task while preventing catastrophic forgetting. First, we construct a label embedding for the target data samples based on the label embeddings of the top-2 closest modes and the computed distances, as shown in Equation (3). Next, we fine-tune the source cGAN model with the newly-labeled target samples, while also implementing memory replay to avoid catastrophic forgetting of the existing modes. After adding the first target to cGAN, we continue the continual learning process for the second target in each experiment. We compare our framework with sequential fine-tuning (Zhai et al., 2019), multi-task learning (Standley et al., 2020b), EWC-GAN (Seff et al., 2017), lifelong-GAN (Zhai et al., 2019), and CAM-GAN (Varshney et al., 2021) for the few-shot generative task with 100 target data samples. We report the FID scores of the images from the target mode, top-2 closest modes, and the average of all modes in Table 2. By selectively choosing and utilizing the relevant knowledge from learned modes, our approach significantly outperforms the conventional training methods (i.e., sequential fine-tuning, and multi-task learning) for both the first (i.e., digit $0$, truck, lion) and the second generative tasks (i.e., digit $1$, cat, and bus). The results further demonstrate that our proposed mode-aware continual learning approach significantly outperforms EWC-GAN (Seff et al., 2017) in the second target task in all datasets. Moreover, our model also achieves highly competitive results in comparison to lifelong-GAN (Zhai et al., 2019) and CAM-GAN (Varshney et al., 2021), showcasing its outstanding performance on the first and second target tasks. Although we observed a slight degradation in the performance of the top-2 closest modes due to the trade-off discussed in Theorem 1, our lifelong learning model demonstrates better overall performance when considering all the learned modes.

Moreover, we implement the proposed continual learning framework on the Oxford Flower dataset, specifically focusing on a subset containing ten distinct flower categories. Here, we define ten generative tasks, each corresponding to one of these flower categories. Our objective is to generate images of calendula flowers, thus designating it as the target task. The remaining flower categories, meanwhile, serve as the source tasks for training cGAN. To determine which source tasks share the greatest resemblance to the target task, we employ dMAS to compute the proximity between the target task and each of the source tasks. This analysis has unveiled the two closest tasks to calendula, namely goldquelle and shasta daisy. Hence, we leverage the knowledge from these related tasks to formulate the weighted target label. This label is subsequently employed with calendula samples to fine-tune the cGAN. Figure 4 represents the generated images of the top-2 closest tasks (i.e., goldquelle and shasta daisy), alongside the target task (i.e., calendula), following the fine-tuning process. The results indicate that cGAN effectively leverages the inherent similarity between goldquelle and shasta daisy to enhance its ability to generate calendula flowers. However, it’s worth noting that in some instances, the model may generate goldquelle-like images instead of calendula flowers. This occurrence can be attributed to the remarkably close resemblance between these two types of flowers.