Theory on Forgetting and Generalization of Continual Learning

In this subsection, we show some insights about the impact of overparameterization. Specially, we will discuss what happens when $p$ changes under a fixed $n$.

1) More parameters can lead to zero forgetting and alleviate the negative impact of task dissimilarity on generalization error. As shown in Theorem 4.1, when $p\to\infty$, we can have that $\mathbb{E}[F_{T}]\to 0$ and Term G2 also approaches zero. In some special cases, we can further show that Term G2 is monotonically decreasing w.r.t. $p$. A more detailed discussion can be found in Section C.3.

2) Benign overfitting exists and is easier to observe with large noise and/or low task similarity. As we introduced in related work, benign overfitting has recently been discovered and studied in linear models as a first step towards understanding why DNNs can still generalize well even when heavily overparameterized. The concept of “benign overfitting” and “double-descent” is initially proposed for only a single task. We now show that such a phenomenon also exists in CL where there exists a sequence of tasks.

We provide an intuitive explanation and rigorous proof of Theorem 4.3 in Section D.8. As shown in Theorem 4.3, $\mathbb{E}[G_{T}]$ becomes larger when the noise level $\sigma$ is larger, and both $\mathbb{E}[F_{T}]$ and $\mathbb{E}[G_{T}]$ become larger when tasks are less similar (i.e., when $\sum_{i=1}^{T-1}\|{\bm{w}}_{T}^{*}-{\bm{w}}_{i}^{*}\|^{2}$ is larger). In contrast, in the overfitted situation, Term F2 and Term G2 in Theorem 4.1 (corresponding to task similarity), Term F3 and Term G3 (corresponding to noise) will go to zero when $p\to\infty$. This indicates that when the noise level is high and/or task similarity is low, the performance of CL in the overparameterized situation is more likely to be better than that in the underparameterized situation, i.e., benign overfitting exists and is easier to observe. This can be observed from Figure 1. For example, the blue curve with markers “$+$” corresponds to the largest noise (compared with other curves in Figure 1(d)) and the lowest task similarity (compared with Figure 1(c)), and it has the deepest descent curve in the overparameterized region ($p>50=n$). This observation indicates that benign overfitting is easier to observe with larger noise and lower task similarity.

3) A descent floor sometimes exists on forgetting and generalization error, especially when tasks are similar and noise is low. In Equation 11, the term $(r^{2}-r)\|{\bm{w}}_{1}^{*}\|^{2}$ first decreases and then increases as $p$ increases from $n$ to $\infty$ (i.e., $r$ increases from $0$ to $1$), while the remaining two terms decrease as $p$ increases. Thus, when $\|{\bm{w}}_{2}^{*}-{\bm{w}}_{1}^{*}\|^{2}$ (task similarity) and $\sigma^{2}$ (noise level) are relatively small, the trend of $F_{2}$ w.r.t. $p$ will be dominated by the first term, where a descent floor of forgetting exists. In the right-hand-side of Equation 12, the first term increases as $p$ increases, while the rest two terms decrease as $p$ increases. Taking the derivative of Equation 12 on $p$, we have

Here, since $\frac{1}{p-n-1}$ is very large when $p$ is close to $n$, while decreasing to zero when $p\to\infty$, we can tell that when $\sigma^{2}$ is relatively small w.r.t. ${{\bm{w}}_{1}^{*}}^{\top}{\bm{w}}_{2}^{*}$, $\frac{\partial\mathbb{E}[G_{2}]}{\partial p}$ will be positive and then negative as $p$ increases from $n+2$ to $\infty$. In other words, if these two tasks have a positive correlation (i.e., ${{\bm{w}}_{1}^{*}}^{\top}{\bm{w}}_{2}^{*}>0$) and noise is small, there exists a descent floor w.r.t. $p$ on $\mathbb{E}[G_{2}]$. Such a phenomenon can exist in other setups besides the special case of $T=2$. For example, in Figure 1(a)(c) where the ground truth for each task is exactly the same, we can observe a descent floor for the small noise cases $\sigma=0.3$ and $0.1$ (i.e., orange and green curves with markers “$\times$” and “Y”, respectively).

Generalization error monotonically decreases with task similarity whereas forgetting may not. Based on Theorem 4.1, it can be seen that the generalization error $G_{T}({\bm{w}}_{T})$ decreases when $\|{\bm{w}}^{*}_{k}-{\bm{w}}^{*}_{i}\|^{2}$ for any two different tasks $k$ and $i$ decreases, because of the positive coefficients in Term G2 in Equation 10. Intuitively, the generalization error of CL will be smaller if the tasks are more similar with each other. In contrast, the forgetting $F_{T}$ may not change monotonically with $\|{\bm{w}}^{*}_{k}-{\bm{w}}^{*}_{i}\|^{2}$, because the coefficients $c_{ij}$ in Term F2 in Equation 9 can be negative. To verify this result, we consider two different scenarios.

(1) Consider the case where $T=2$. In Equation 11, $\|{\bm{w}}_{2}^{*}-{\bm{w}}_{1}^{*}\|^{2}$ captures the task similarity between tasks 1 and 2 in terms of the optimal task models. It is clear that forgetting increases with $\|{\bm{w}}_{2}^{*}-{\bm{w}}_{1}^{*}\|^{2}$, i.e., less forgetting when the two tasks are more similar.

(2) Consider the case where $T=4$. We first assume that $\|{\bm{w}}_{i}^{*}\|^{2}=w$ for any task $i\in[1,4]$ considering the overparameterized models [17]. Suppose that task 1 and task 2 share the same set of true features, which is orthogonal to the feature set of both task 3 and task 4, i.e., ${\mathcal{S}}_{1}={\mathcal{S}}_{2}$ and ${\mathcal{S}}_{1}\cap({\mathcal{S}}_{3}\cup{\mathcal{S}}_{4})=\emptyset$. Note that

where $\langle{\bm{w}}_{i}^{*},{\bm{w}}_{j}^{*}\rangle=0$ if ${\mathcal{S}}_{i}\cap{\mathcal{S}}_{j}=\emptyset$. Therefore, we can control the value of $\|{\bm{w}}_{1}^{*}-{\bm{w}}_{2}^{*}\|^{2}$ by changing ${\bm{w}}_{2}^{*}$, without affecting the value of $\|{\bm{w}}_{i}^{*}-{\bm{w}}_{j}^{*}\|^{2}$ for any pair of $\{i,j\}\neq\{1,2\}$. Based on Theorem 4.1, it can be shown that $c_{1,2}<0$, such that increasing $\|{\bm{w}}_{1}^{*}-{\bm{w}}_{2}^{*}\|^{2}$, i.e., the tasks become less similar, will surprisingly decrease forgetting.

In order to investigate the impact of task ordering on the performance of CL, we assume that $\|{\bm{w}}_{t}^{*}\|^{2}=w$ for every task $t\in\mathbb{T}$. By ignoring the task order-independent terms in Equation 9 and Equation 10, we focus on the task order-dependent terms, i.e., Term F2 and Term G2.

1) Optimal task ordering of minimizing forgetting tends to arrange dissimilar tasks adjacently in the early stage of the sequence. As shown in Term F2, the optimal task order to minimize forgetting closely hinges upon the value of $c_{i,j}$. Based on Equation 8, $c_{i,j}$ is smaller when (1) $i$ and $j$ are smaller and (2) they are closer. Intuitively, this implies that tasks with larger $\|{\bm{w}}_{i}^{*}-{\bm{w}}_{j}^{*}\|^{2}$ should be learnt adjacently with higher priority in CL, in order to minimize the impact of the task dissimilarity on the value of $\tilde{F}_{T}({\bm{w}}_{T})$. However, finding the optimal task order for the general case is highly nontrivial due to the complex coupling across $\|{\bm{w}}_{i}^{*}-{\bm{w}}_{j}^{*}\|^{2}$ for different tasks. To verify the implication above and better understand the structure of the optimal task order, we study several special cases of the task setups.

As indicated by Proposition 4.4, the special task will be learnt in the first half of the sequence, such that the task diversity in the first half is always larger than in the second half. Besides, with the model capacity increasing ($\frac{n}{p}\rightarrow 0$), the order of the special task will move towards the beginning of the sequence, because 1) the model is less concerned about the special task since it is powerful enough to learn different features and 2) the model focuses on the performance of the majority and seeks to learn more tasks from Category II at the end of the sequence for better performance.

Proposition 4.5 clearly shows that adjacent tasks always belong to different categories in the optimal task order, which leads to a more diverse task learning sequence. Intuitively, the alternating order maximizes the memorization of each category by keeping practicing on different tasks. It can be further proved that the perfectly alternating order is also optimal for $T=6$ with three different categories (Section C.4). Based on these results, we expect that such an alternating order may minimize forgetting for more general scenarios where the tasks contain multiple categories with equal cross-category task model distance.

The findings on the optimal task order indeed share similar insights with the surprising impact of task correlation on forgetting mentioned earlier. Intuitively, learning more dissimilar tasks in the early stage facilitates the exploration of a larger feature space and expands the learnt feature space in CL, which can make the learning of similar tasks in the future much easier. In the meanwhile, the impact of task similarity among the early tasks continuously diminishes in CL with $T$ increasing, as suggested by the coefficients $c_{i,j}$ (which can be smaller for smaller $i$, $j$) in Theorem 4.1. Therefore, the negative impact of learning more dissimilar tasks on forgetting is weaker when they are learnt in the early stage, compared to being learnt in the late stage.

2) The optimal task ordering for minimizing forgetting and for minimizing generalization error are not always the same. Consider Special case I and Special case II. It can be shown that the optimal task orders for minimizing forgetting and generalization error are different in Special case I but same in Special case II. This would open up an interesting direction of finding the task order with balanced impact on forgetting and generalization error. A more detailed discussion can be found in Section C.4.

To see if our understandings about the impact of task similarity on forgetting can be carried over to CL with DNN, we conduct experiments on MNIST [29] using a convolutional neural network to investigate the impact of task similarity therein. More specifically, we consider each task $i$ as a binary classification problem which seeks to decide if an image belongs to a task-specific label subset $Y_{i}$ of the classes, i.e., $Y_{i}\subset\{0,...,9\}$ in MNIST, and we control the task similarity through the degree of class overlapping between the task-specific subsets, e.g., task $i$ and $j$ are more similar if the cardinality of $Y_{i}\cap Y_{j}$ is larger.

We first consider the case with two tasks, where we fix $Y_{1}$ for task 1 as $\{0,1,2,3,4\}$ and change $Y_{2}$ for task 2 to have different numbers of overlapping classes with $Y_{1}$. As shown in Figure 2, both forgetting and generalization error decrease when the number of overlapping classes increases, i.e., the two tasks are more similar, which is indeed consistent with our analysis for the overparameterized linear models for $T=2$. More interestingly, this result also agrees with some recent studies [42, 30, 17], which found that ‘intermediate task similarity’ leads to the worst forgetting in a two-task setup using various notions of task similarity (different from our definition of task similarity using the optimal model gap), through either empirical studies or analyzing the upper bound of forgetting. We can build the connection based on the closed form of forgetting $F_{2}$ in Equation 11.

Note that in Equation 11

and we can divide the task correlation into three cases depending on the value of $\langle{\bm{w}}_{1}^{*},{\bm{w}}_{2}^{*}\rangle$: (1) $\langle{\bm{w}}_{1}^{*},{\bm{w}}_{2}^{*}\rangle=0$: Two tasks are orthogonal in the sense that they share no common features, i.e., ${\mathcal{S}}_{1}\cap{\mathcal{S}}_{2}=\emptyset$; (2) $\langle{\bm{w}}_{1}^{*},{\bm{w}}_{2}^{*}\rangle>0$: Two tasks share some common features and are ‘positively’ correlated; (3) $\langle{\bm{w}}_{1}^{*},{\bm{w}}_{2}^{*}\rangle<0$: Two tasks share some common features but are ‘negatively’ correlated. Compared to the first case when two tasks are orthogonal, it can be easily shown that forgetting is worse when two tasks are negatively correlated even if they share some common features, which indeed corresponds to ‘the intermediate task similarity’ in [42, 30, 17]. The reason behind is that in this case task 2 updates the model in the opposite direction to the model update of task 1, which inevitably leads to more forgetting in CL. Note that in Figure 2, the non-overlapping case means that task 1 and 2 are negatively correlated because in this two-task case the image that is not in $Y_{1}$ must be in $Y_{2}$. On the other hand, the forgetting can even be negative when the two tasks are positively correlated.

We next consider the case with $T=4$, where we control the task similarity by changing $Y_{2}$ while fixing $Y_{1}$, $Y_{3}$ and $Y_{4}$. Here we let $(Y_{1}\cup Y_{2})\cap(Y_{3}\cup Y_{4})=\emptyset$ as in Section 4.2. As shown in Figure 2, forgetting surprisingly increases when task 1 and task 2 have more overlapping classes, which is also consistent with our analysis for the linear models. Indeed, this also justifies our observation that forgetting can decrease when the adjacent tasks are more dissimilar when studying the impact of task order.

We also evaluate the impact of task ordering on forgetting in CL with DNN, by constructing the tasks using a similar strategy as in Section 5.1. More specifically, we consider two different scenarios: (1) $T=6$, where the task sequence includes one special task and five same tasks; (2) $T=4$, where the task sequence includes two categories of tasks and each has two same tasks.

Figure 2 demonstrates forgetting in the first scenario w.r.t. the learning order of the special task, and three plots correspond to three different cases, respectively. It is clear that for all three cases, the optimal order of the special task to minimize forgetting is always in the first half of the sequence. For the second scenario, we evaluate forgetting in Figure 2 for all six possible task orders, where task indices $0$ and $1$ refer to the perfectly alternating order. We can see that the smallest forgetting is also achieved in the perfectly alternating order. These results indicate that our findings in Section 4.3 for the overparameterized linear models can also be carried over to CL with DNN, i.e., the optimal task order should diversify the tasks in the early stage and learn more different tasks adjacently. Such an implication is indeed consistent with the empirical observations in recent studies [31, 10]. Note that in both Figure 2 and Figure 2, we normalize forgetting w.r.t. the worst forgetting in each case.

Recently, there has been increasing interest in CL on leveraging task correlation to facilitate knowledge transfer [26, 33, 32], which first selects the most correlated old tasks with the current task and then designs algorithms to directly increase the knowledge transfer between correlated tasks. By investigating knowledge transfer in the linear models, we show that improved algorithms can be motivated to achieve better knowledge transfer.

Given a task $t$ in CL, the forward knowledge transfer [47] in the linear model can be defined as

where ${\bm{w}}^{r}_{t}$ is the learnt model of task $t$ by starting from a random model. Intuitively, Equation 13 characterizes the gap in the testing performance between ${\bm{w}}_{t}$ learnt in CL and ${\bm{w}}^{r}_{t}$ learnt from scratch, for which a positive value means that the accumulated knowledge in CL benefits the learning of the current task. As the second term in Equation 13 is independent with CL, it suffices to analyze $\mathbb{E}[\|{\bm{w}}_{t}-{\bm{w}}_{t}^{*}\|^{2}]$ for understanding the forward knowledge transfer. Based on Lemma B.2 (Appendix B), we can obtain

While it is intuitive that better forward knowledge transfer can be achieved when $\|{\bm{w}}_{i}^{*}-{\bm{w}}_{t}^{*}\|^{2}$ is smaller for the current task $t$ and the old task $i$, the impact of different old tasks on the current task is non-uniform, in the sense that a more recent old task $i$ (i.e., $t-i$ is smaller) has a larger effect on the forward knowledge transfer to task $t$. This result implies that fresher old tasks should contribute more when designing algorithms to leverage correlated old tasks to facilitate better forward knowledge transfer.

To verify this insight, we consider the TRGP algorithm proposed in [33]. Specifically, TRGP first selects the most correlated old tasks with the current task and reuses their knowledge through a scaled weight projection to facilitate forward knowledge transfer, where all the selected old tasks are treated equivalently. We slightly modify TRGP by assigning a larger weight to the selected old task that is more recent to the current task, named as TRGP+, and evaluate its performance on standard CL benchmarks (PMNIST [35] and Split CIFAR-100 [28]) and DNN architectures. As shown in Table 1, TRGP+ outperforms TRGP in both accuracy and forgetting. Assigning a larger weight to the more recent correlated old task not only improves the forward knowledge transfer, but also increases the backward knowledge transfer by forcing the learnt model of the current task to be closer to the model of those highly correlated old tasks.