Semantic-aware Knowledge Distillation for Few-Shot
Class-Incremental Learning

Suppose, there is a sequence of disjoint tasks $\mathcal{D}=\{\mathcal{D}^{1}$, …, $\mathcal{D}^{T}$}, where $\mathcal{C}^{t}=\left\{c^{t}_{1},...,c^{t}_{m^{t}}\right\}$ is the set of classes in the task $\mathcal{D}^{t}$. Additionally, a set of $d$-dimensional semantic class embeddings for each class label in the task $\mathcal{D}^{t}$ defined as $\mathcal{S}^{t}$ is available during training. To be more specific, in the task $\mathcal{D}^{t}=\left\{\left(\textbf{x}^{t}_{i},l^{t}_{i},\textbf{s}^{t}_{i}\right)\right\}_{i=1}^{|\mathcal{D}^{t}|}$, $\textbf{x}^{t}_{i}$ is the $i$-th sample, $l^{t}_{i}$ is its associated ground truth, and $\textbf{s}^{t}_{i}$ is its associated semantic representation. There are many training samples in the first task $\mathcal{D}^{1}$, termed the base task. However, in the following tasks $t>1$, termed the novel task, there are only a few training samples (5-shots per class) for each class. It is essential to mention that the classes between all tasks are disjoint, i.e., $\mathcal{C}^{i}\cap\mathcal{C}^{j}=\phi,\forall i,j\in\{1,\ldots,T\}$, where $i\neq j$. The objective of our work is to incrementally train a model with a unified output, while only training samples of the $t$-th task is available at the $t$-th session. At test time, we expect the trained model on task $\mathcal{D}^{t}$ to predict the output for the current task and all the previous tasks {$\mathcal{D}^{1}$, … , $\mathcal{D}^{t-1}$}.

Knowledge distillation is a common approach [26, 1, 13] for incremental learning to address the catastrophic forgetting. Even though it has shown promising results on incremental learning, it cannot be employed directly to few-shot class incremental learning (FSCIL) due to the imbalanced data and trade-off issues [32]. In this paper (see Figure 2), we will show how to successfully take advantage of knowledge distillation in the FSCIL setting. To illustrate our proposed method for knowledge distillation, we use a simplified version of our proposed architecture. In this scheme, the input image x goes into the backbone $B$, which is trained on only the first task $\mathcal{D}^{1}$, as we have many training samples in this task, and is kept frozen on other tasks. The output of the backbone $B$ is a feature representation $\textbf{g}\in\mathbb{R}^{u}$. Next, the mapping network $M$ is used to project the feature representation g into the semantic domain, where the projected feature ${\textbf{y}}\in\mathbb{R}^{d}$ is aligned with its associated semantic representation $\textbf{s}\in\mathbb{R}^{d}$. Lets assume we want to train the model with the task $\mathcal{D}^{t}$, which has $m$ or $|\mathcal{C}^{t}|$ classes, and the number of classes of the previous tasks are $n$. Also, we choose one representation for each class of the former tasks saved in a small memory $\mathcal{M}$, where each prototype $\textbf{g}^{\mathcal{M}}\in\mathbb{R}^{u}$ is the average of all training samples from each class. The aim is to map the input image $\textbf{x}_{i}$ into the semantic domain $\textbf{y}_{i}$ by the function $\textbf{y}_{i}=\kappa_{\theta}(\textbf{x}_{i})$, which consists all training parameters of the backbone $B$ and the mapping module $M$. We consider the cosine distance as the similarity between the projected feature $\textbf{y}_{i}$ and the semantic representation $\textbf{s}_{k}$, $d(\textbf{s}_{k},\textbf{y}_{i})=cos(\textbf{s}_{k},\textbf{y}_{i})$. If the output of the classifier before adding the novel task $\mathcal{D}^{t}$ is $\textbf{d}{}^{\prime}=\left[d^{\prime}(\textbf{s}_{1},\textbf{y}_{i}),...,d^{\prime}(\textbf{s}_{n},\textbf{y}_{i})\right]$, and if the output of the classifier after adding the novel task is $\textbf{d}=\left[d(\textbf{s}_{1},\textbf{y}_{i}),...,d(\textbf{s}_{n},\textbf{y}_{i}),d(\textbf{s}_{n+1},\textbf{y}_{i}),...,d(\textbf{s}_{n+m},\textbf{y}_{i})\right]$, the distillation loss is defined as,

where ${\textbf{y}}_{i}=M(\textbf{g}_{i})$, and $\textbf{g}_{i}\in B(\textbf{x}^{t}_{i})\cup\mathcal{M}$. Here, $\tau$ is the temperature scalar. The $\tau$ is set to 2 for all experiments. Also, $N_{c}$ is the number of samples in the task $\mathcal{D}^{t}$ and memory $\mathcal{M}$, i.e., $N_{c}=|\mathcal{D}^{t}|+|\mathcal{M}|$.

Additionally, we employ a cross-entropy loss as a classification loss,

This approach does not add new parameters during the incremental learning stage. Instead, it only fine-tunes $M$ with newly available data incrementally. It helps the distillation process and learning without forgetting. Also, this approach reduces the data imbalance problem of few-shot learning. The network may not have had the opportunity to see the novel objects, however, newly discovered novel objects may very well share semantic properties with the objects it has already seen among the base classes. For example, if hyena is a novel class, the network did not have any opportunity to see any hyena. However, many typical hyena attributes like ‘face’, ‘body’, etc., are seen by the network from other base class animals (e.g., Dalmatian, Saluki, Meerkat). It helps the network to understand hyena by reducing the data dependency. An intuitive illustration is shown in Figure 3.

The main challenge with the tasks containing a few training samples is the overfitting issue, as it is difficult to learn the distribution of a task sufficiently with only a few training data. To this end, in our proposed method (see Figure 4), we generate multiple embeddings, where each is designed specifically for a group of classes. We use the word vector semantic to separate classes into several groups. We design these groups based on the classes that we have in the first task $\mathcal{D}^{1}$ because the first task contains more classes with many training samples. After that, we train each embedding $E_{i}$ using a cross-entropy loss based on the superclass labels obtained in the previous stage. Each embedding observes only part of the entire set of classes, which results in embeddings that are experts on a particular superclass. For the novel tasks, we assign a superclass label to each class based on the cluster that we obtained in the previous stage.

The final proposed architecture is shown in Figure 5. The input image x goes into the backbone $B$, which is pretrained on the first task $D^{1}$ with a cross-entropy loss, to generate the global feature representation $\textbf{g}\in\mathbb{R}^{u}$. When training the other tasks $\mathcal{D}^{t},t>1$, the backbone network $B$ is kept frozen to prevent overfitting to classes with only a few training samples. Then, the extracted feature g is fed into $N$ embedding modules $E_{i},~{}i=1,\ldots,N$, where they are trained on the first task based on the superclass information and are updated for the novel tasks. To this end, the output of all embedding $\textbf{e}^{k}\in\mathbb{R}^{u}$ are fused based on the weights generated by an attention module $A$. Then, the global feature g is concatenated with the feature generated from the embedding modules e to form the feature ${\textbf{f}}\in\mathbb{R}^{2u}$. In the next stage, the feature f is projected from the visual domain to the semantic domain by the mapping module $M$ to form feature ${\textbf{y}}\in\mathbb{R}^{d}$, where it is aligned with its associated semantic representation $\textbf{s}\in\mathbb{R}^{d}$. In order to train our proposed architecture, we use the following loss function,

where $\lambda_{1}$, $\lambda_{2}$, and $\lambda_{3}$ are used to control the effect of each term in the final loss function. The pseudo code of the training procedure is shown in Algorithm 1.

To further evaluate our proposed method, we apply our design to the DFSL setting. As discussed in the related work section, DFSL is similar to FSCIL, though the only difference is that we have only two sequences of tasks in DFSL whereas there are various sequences of tasks in FSCIL. In this section, we use the setting introduced by [9]. In this experiment, we use miniImageNet, which is split into two tasks, a base task and a novel task. The base task consists of 64 classes, while the novel task has five classes which are randomly selected in an episode way from 20 classes. It is necessary to mention that the classes in the base and novel tasks are disjoint. In each episode, the training (support) set is built by selecting 1 or 5 examples of the novel classes, representing a 1-shot or 5-shot scenario. The test (query) set consists of examples of the base and novel tasks. In this experiment, we use ResNet12 as adopted in [19] as the backbone. The forgetting in the DFSL setting is calculated as the gap between joint and individual performances for base and novel classes. Individual performances for the base/novel task is calculated when only the base/novel classifier is used. $\Delta_{b}$ and $\Delta_{n}$ are used to indicate the gap between the base and novel tasks, respectively. The average of these gaps is represented as $\Delta=(\Delta_{b}+\Delta_{n})/2$, which show the amount of forgetting,

We compare our method with state-of-the-art methods in Table 2. As can be seen, our method outperforms all other methods on joint accuracy and forgetting $\Delta$ on both the 1-shot and 5-shot settings.