AFEC: Active Forgetting of Negative Transfer in Continual Learning

Continual learning needs to remember the old tasks and learn each new task effectively. Let’s consider a simple case that a neural network with parameter $\theta$ continually learns two independent tasks, task $A$ and task $B$, from their training datasets $D_{A}^{train}$ and $D_{B}^{train}$ [12]. The training dataset of each task is only available when learning the task.

Bayesian Learning: After learning $D^{train}_{A}$, the posterior distribution

incorporates the knowledge of task $A$. Then, we can get the predictive distribution for the test data of task $A$:

As the posterior $p(\theta|D_{A}^{train})$ is generally intractable (except very special cases), we must resort to approximation methods, such as the Laplace approximation [12] or other approaches of approximate inference [20]. Let’s take Laplace approximation as an example. If $p(\theta|D_{A}^{train})$ is smooth and majorly peaked around the mode $\theta_{A}^{*}=\arg\max_{\theta}\log p(\theta|D^{train}_{A})$, we can approximate it with a Gaussian distribution whose mean is $\theta_{A}^{*}$ and covariance is the inverse Hessian of the negative log posterior (detailed in Appendix B.1).

Bayesian Continual Learning: Next, we want to incorporate the new task into the posterior, which uses the posterior $p(\theta|D_{A}^{train})$ as the prior of the next task [12]:

Then we can test the performance of continual learning by evaluating

Similarly, $p(\theta|D_{A}^{train},D_{B}^{train})$ can be approximated by a Gaussian using Laplace approximation whose mean is the mode of the posterior:

This MAP estimation is also known as the Elastic Weight Consolidation (EWC) [12]:

where $L_{\rm{B}}(\theta)$ is the loss for task $B$ and $i$ is the label of each parameter. $F_{A}$ is the Fisher Information matrix (FIM) of $\theta_{A}^{*}$ on $D_{A}^{train}$ (the computation is detailed in Appendix B.1), which indicates the “importance” of parameter $i$ for task $A$. The hyperparameter $\lambda$ explicitly controls the penalty that selectively merges each $\theta_{i}$ to $\theta_{A,i}^{*}$ to alleviate catastrophic forgetting.

However, if precisely remembering task $A$ interferes with the learning of task $B$, e.g., task $A$ and task $B$ are too different, it might be useful to actively forget the original data, similar to the biological strategy of active forgetting. Based on this inspiration, we introduce a forgetting factor $\beta$ and replace $p(\theta|D_{A}^{train})$ that absorbs all the information of $D_{A}^{train}$ with a weighted product distribution [8, 19]:

where we use $m$ to denote that we are ‘mixing’ $p(\theta|D_{A}^{train})$ and $p(\theta)$ to produce the new distribution $p_{m}$. $Z$ is the normalizer that depends on $\beta$, which keeps $p_{m}(\theta|D^{train}_{A},\beta)$ following a Gaussian distribution if $p(\theta|D^{train}_{A})$ and $p(\theta)$ are both Gaussian (detailed in Appendix B.2). When $\beta\rightarrow 0$, $p_{m}$ will be dominated by $p(\theta|D^{train}_{A})$ and remember all the information about task $A$. When $\beta\rightarrow 1$, $p_{m}$ will actively forget all the information about task $A$. Modified from Eqn. (2), our target becomes:

We first need to determine $\beta$, which decides how much information from task $A$ is forgotten to maximize the probability of learning task $B$ well. A good $\beta$ should be as follows:

Since the integral is difficult to solve, we can make a grid search to determine $\beta$, which should be between 0 and 1. Next, $\,p(\theta|D_{A}^{train},D_{B}^{train},\beta)$ can also be approximated by a Gaussian using Laplace approximation (the proof is detailed in Appendix B.3), and the MAP estimation is

Then we obtain the loss function of Active Forgetting with synaptic Expansion-Convergence (AFEC):

$\theta_{e}^{*}$ are the optimal parameters for the new task and $F_{e}$ is the FIM of $\theta_{e}^{*}$ (the computation is detailed in Appendix B.1). As shown in Fig. 2, we first learn a set of expanded parameters $\theta_{e}$ with $L_{\rm{B}}(\theta_{e})$ to obtain $\theta_{e}^{*}$ and $F_{e}$. Then we can optimize Eqn. (10), where two weight-merging regularizers selectively merge $\theta_{i}$ with $\theta_{A,i}^{*}$ for the old tasks and $\theta_{e,i}^{*}$ for the new task. The forgetting factor $\beta$ is integrated into a hyperparameter $\lambda_{e}\propto\beta/(1-\beta)$ to control the penalty that promotes active forgetting. Therefore, derived from active forgetting of the original posterior in Eqn. (6), we obtain an algorithm that dynamically expands parameters to learn a new task and then selectively converges the expanded parameters to the main network. Intriguingly, this algorithm is formally consistent with the underlying mechanism of biological active forgetting (the neuroscience evidence is detailed in Appendix A), which also expands additional functional connections for a new experience (synaptic expansion) and then prunes them to the amount before learning (synaptic convergence).

As the proposed active forgetting is integrated into the third term, our method can be used in a plug-and-play way to improve continual learning (detailed in Appendix E, F). Here we use Laplace approximation to approximate the intractable posteriors, which can be other strategies of approximate inference [20] in further work. Note that $\theta_{e}^{*}$ and $F_{e}$ are not stored in continual learning, and the architecture of the main network is fixed. Thus, AFEC does not cause additional storage cost compared with regular weight regularization approaches such as [12, 36, 1, 2]. Further, it is straightforward to extend our method to continual learning of more than two tasks. We discuss it in Appendix B.4 with a pseudocode.

Now we conceptually analyze how AFEC mitigates potential negative transfer in continual learning (see Fig. 2). When learning task $B$ on the basis of task $A$, regular weight regularization approaches [12, 36, 1, 2] selectively penalize changes of the old parameters learned for task $A$, which will severely interfere with the learning of task $B$ if they conflict with each other. In contrast, AFEC learns a set of expanded parameters only for task $B$ to avoid potential negative transfer from task $A$. Then, the main network parameters selectively merge with both the old parameters and the expanded parameters, depending on their contributions to the overall representations of task $A$ and task $B$.

First, we propose CIFAR-10 regression tasks to explicitly show how negative transfer affects continual learning, and how AFEC effectively addresses this challenging issue. CIFAR-10 dataset [13] contains 50,000 training samples and 10,000 testing samples of 10-class colored images of size $32\times 32$. The regression task is to evenly map the ten classes around the origin of the two-dimensional coordinates and train the neural network to predict the angle of the origin to each class (see Fig. 3). We change the relative position of the ten classes to construct different regression tasks with mutual negative transfer, in which remembering the old knowledge will severely interfere with the learning of a new task.

As shown in Fig. 4 for continual learning of two different regression tasks, regular weight regularization approaches, such as MAS [1] and EWC [12], can effectively remember the old tasks, but limits the learning of new tasks. In particular, larger strength of the weight regularization results in better performance of the first task but worse performance of the second task. In contrast, AFEC improves the learning of new tasks on the basis of remembering the old tasks, so as to achieve better averaged accuracy. Note that EWC is equal to the ablation of active forgetting in AFEC, i.e., $\beta=0$, so the performance improvement of AFEC on EWC validates the effectiveness of our proposal. We further demonstrate the efficacy of AFEC on a variety of architectures and a larger amount of tasks (see Table 1).

In addition, we evaluate the ability of transfer learning after continual learning of two different regression tasks. We fix the feature extractor of the neural network and only fine-tune a linear classifier to predict a new task that is similar to the first task. Specifically, the similar task applies the same relative position as the first task, but rotates by $60^{\circ}$, $120^{\circ}$, $180^{\circ}$, $240^{\circ}$ or $300^{\circ}$. Therefore, if the neural network effectively remembers and transfers the relative position learned in the first task, it will be able to learn the similar task well. As shown in Table 1, AFEC can more effectively learn the similar task, while EWC is even worse than sequentially fine-tuning without weight regularization.

Dataset: We evaluate continual learning on a variety of benchmark datasets for visual classification, including CIFAR-100, CUB-200-2011 and ImageNet-100. CIFAR-100 [13] contains 100-class colored images of the size $32\times 32$, where each class includes 500 training samples and 100 testing samples. CUB-200-2011 [30] is a large-scale dataset including 200 classes and 11,788 colored images of birds, split as 30 images per class for training while the rest for testing. ImageNet-100 [9] is a subset of iILSVRC-2012 [23], consisting of randomly selected 100 classes of images and 1300 samples per class. We follow the regular preprocessing pipeline of CUB-200-2011 and ImageNet-100 as [10], which randomly resizes and crops the images to the size of $224\times 224$ before experiment.

Benchmark: We consider five representative benchmarks of visual classification tasks to evaluate continual learning in different aspects. The first three are on CIFAR-100, with forward knowledge transfer from more negative to more positive (detailed in Fig. 5), while the second two are on large-scale images. (1) CIFAR-100-SC [35]: CIFAR-100 can be split as 20 superclasses (SC) with 5 classes per superclass dependent on semantic similarity, where each superclass is a classification task. Since the superclasses are semantically different, forward knowledge transfer in such a task sequence is relatively more negative. (2) CIFAR-100 [21]: The 100 classes in CIFAR-100 are randomly split as 20 classification tasks with 5 classes per task. (3) CIFAR-10/100 [10]: The 10-class CIFAR-10 are randomly split as 2 classification tasks with 5 classes per task, followed by 20 tasks with 5 classes per task randomly split from CIFAR-100. This benchmark is adapted from [10] to keep the number of classes per task the same as benchmark (1, 2), where the large amounts of training data in the first two CIFAR-10 tasks bring a relatively more positive transfer. (4) CUB-200 [10]: The 200 classes in CUB-200-2011 are randomly split as 10 classification tasks with 20 classes per task. (5) ImageNet-100 [21]: The 100 classes in ImageNet-100 are randomly split as 10 classification tasks with 10 classes per task.

Architecture: We follow [10] to use a CNN architecture with 6 convolution layers and 2 fully connected layers for benchmark (1, 2, 3), and AlexNet [14] for benchmark (4, 5). Since continual learning needs to quickly learn a usable model from incrementally collected data, we mainly consider learning the network from scratch. Following [10], we also try AlexNet with ImageNet pretraining for CUB-200.

Baseline: First, we consider a restrict yet realistic setting of continual learning without access to the old training data, and perform multi-head evaluation [2]. Since AFEC is a weight regularization approach, we mainly compare with representative approaches that follow a similar idea, such as EWC [12], MAS [1], SI [36] and RWALK [2]. We also compare with AGS-CL [10], the SOTA method that takes advantage of weight regularization and parameter isolation, and P&C [26], which learns an additional active column on the basis of EWC to improve forward knowledge transfer. We reproduce the results of all the baselines from the officially released code of [10], where we do an extensive hyperparameter search and report the best performance for fair comparison (detailed in Appendix C). Then, we relax the restriction of using old training data and plug AFEC in representative memory replay approaches, where we perform single-head evaluation [2] (detailed in Appendix F).

Averaged Accuracy: In Table 2, we summarize the averaged accuracy of all the tasks learned so far during continual learning of visual classification tasks. AFEC achieves the best performance on all the continual learning benchmarks and is much better than EWC [12], i.e., the ablation of active forgetting in AFEC. In particular, AGS-CL [10] is the SOTA method on relatively small-scale images and on CUB-200 with ImageNet pretraining (CUB-200 w/ PT). While, AFEC achieves a better performance than AGS-CL on small-scale images from scratch and CUB-200 w/ PT, and substantially outperforms AGS-CL on the two benchmarks of large-scale images from scratch. Further, since regular weight regularization approaches are generally in a re-weighted weight decay form, AFEC can be easily adapted to such approaches (the adaptation is detailed in Appendix E) and effectively boost their performance on the benchmarks above.

Knowledge Transfer: Next, we evaluate knowledge transfer in the three continual learning benchmarks developed on CIFAR-100 in Fig. 5. We first present the accuracy of learning each new task in continual learning, where AFEC learns each new task much better than other baselines. Since continual learning of more tasks leads to less network resources for a new task, the overall trend of all the baselines is declining, indicating the necessity to improve forward knowledge transfer on the basis of overcoming catastrophic forgetting. Then we calculate forward transfer (FWT) [17], i.e., the averaged influence that learning the previous tasks has on a future task, and backward transfer (BWT) [17], i.e., the averaged influence that learning a new task has on the previous tasks (detailed in Appendix D). FWT is from more negative to more positive in CIFAR-100-SC, CIFAR-100 and CIFAR-10/100, while AFEC achieves the highest FWT among all the baselines. The BWT of AFEC is comparable as EWC, indicating that the proposed active forgetting does not cause additional catastrophic forgetting. Therefore, the performance improvement of AFEC in Table 2 is achieved by effectively improving the learning of new tasks in continual learning. In particular, AFEC achieves a much larger improvement on the learning of new tasks than P&C, which attempted to improve forward transfer of EWC through learning an additional active column. Due to the progressive parameter isolation, although AGS-CL achieves the best BWT, its ability of learning each new task drops more rapidly than other baselines. Thus, it underperforms AFEC in Table 2.

Visual Explanation: To explicitly show how AFEC improves continual learning, in Fig. 6 we use Grad-CAM [27] to visualize predictions of the latest task after continual learning on CIFAR-100-SC, where FWT is more negative as discussed above. The predictions of EWC overfit the background information since it attempts to best remember the old tasks with severe negative transfer, which limits the learning of new tasks. In contrast, the visual explanation of AFEC is much more reasonable than EWC, indicating the efficacy of active forgetting to address potential negative transfer and benefit the learning of new tasks.

Plugging-in Memory Replay: We further implement AFEC in representative memory replay approaches in Appendix F, where we perform single-head evaluation [2]. On CIFAR-100 and ImageNet-100 datasets, we follow [9, 6] that first learn 50 classes and then continually learn the other 50 classes by 5 phases (10 classes per phase) or 10 phases (5 classes per phase), using a small memory buffer of 20 images per class. AFEC substantially boosts the performance of representative memory replay approaches such as iCaRL [21], LUCIR [9] and PODNet [6].

Next, we evaluate AFEC in continual learning of Atari reinforcement tasks (Atari games). We follow the implementation of [10] to sequentially learn eight randomly selected Atari games. Specifically, we applies a CNN architecture consisting of 3 convolution layers with 2 fully connected layers and identical PPO [25] for all the methods (detailed in Appendix G). The evaluation metric is the normalized accumulated reward: the evaluated rewards are normalized with the maximum reward of fine-tuning on each task, and accumulated. We present the results of three different orders of task sequence, averaged by five runs with different random initialization.

For continual learning of Atari reinforcement tasks, forward knowledge transfer is severely negative, possibly because the optimal policies of each Atari games are highly different. We first measure the normalized rewards of learning each task with a randomly initialized network, which are 2.16, 1.44 and 1.68 on the three task sequences, respectively. That is to say, the initialization learned from the old tasks results in an averaged performance decline by $53.67\%$, $30.66\%$ and $40.56\%$, compared with random initialization. Then, we evaluate the maximum reward of learning each new task in Table 3, and the normalized accumulated reward of continual learning in Fig. 7. AFEC effectively improves the learning of new tasks and thus boosts the performance of EWC and MAS, particularly when learning more incremental tasks. AFEC also achieves a much better performance than the reproduced results of AGS-CL on its officially released code [10] (see Appendix G for an extensive analysis).