HiDe-PET: Continual Learning via Hierarchical Decomposition of Parameter-Efficient Tuning

The CL problem can be generally defined as learning sequentially arriving tasks from their respective training sets $\mathcal{D}_{1},...,\mathcal{D}_{t}$ in order to perform well on their corresponding test sets. The training set and test set of each task are assumed to follow the same distribution. For supervised CL, each training set $\mathcal{D}_{t}=\{(\boldsymbol{x}_{t,n},y_{t,n})\}_{n=1}^{N_{t}}$ comprises multiple sample-label pairs, where $\boldsymbol{x}_{t,n}\in\mathcal{X}_{t}$ and $y_{t,n}\in\mathcal{Y}_{t}$ represent the sample and label elements, respectively, and $N_{t}$ denotes the size of $\mathcal{D}_{t}$. Let us consider a neural network model composed of a backbone $f_{\theta}$ with parameters $\theta$, and an output layer $h_{\psi}$ with parameters $\psi$. This model aims to learn a projection from $\mathcal{X}=\bigcup_{i=1}^{t}\mathcal{X}_{i}$ to $\mathcal{Y}=\bigcup_{i=1}^{t}\mathcal{Y}_{i}$, so that it can correctly predict the label $\hat{y}=h_{\psi}(f_{\theta}(\boldsymbol{x}))\in\mathcal{Y}$ of an unseen test sample $\boldsymbol{x}$ from observed tasks. Since $\mathcal{D}_{1},...,\mathcal{D}_{t}$ are usually limited in size and distinct in distribution, learning $f_{\theta}$ from scratch can easily converge to an undesirable local minimum. In contrast, initialization of $f_{\theta}$ with a substantial quantity of training samples external to $\mathcal{D}_{1},...,\mathcal{D}_{t}$, i.e., applying adequate pre-training, helps $\theta$ converge to a wide low-error region and thus can greatly improve the CL performance [1, 2].

Depending on the split of label space and the availability of task identity, there are three representative setups for CL [37], including task-incremental learning (TIL), domain-incremental learning (DIL), and class-incremental learning (CIL). Specifically, the label space $\mathcal{Y}_{i}$ with $i\in[t]$ is the same for DIL whereas disjoint for TIL and CIL. The task identity $i\in[t]$ is provided at test time for TIL whereas not available for DIL and CIL. As a result, CIL is often considered more challenging and realistic. Of note, although CIL is named from classification tasks, its definition can be generalized to other task types. To avoid additional resource overhead and potential privacy issues, the CL process is further restricted to be rehearsal-free [8], i.e., the sample elements of each $\mathcal{D}_{i}$ are available only when learning task $i$, which particularly increases the challenge of CIL [3].

The backbone $f_{\theta}$ of advanced pre-trained models (PTMs) often employs a transformer architecture [38] based on multi-head attention mechanisms. For example, a pre-trained vision transformer (ViT) [39] consists of multiple consecutive multi-head self-attention (MSA) layers that transform an input sample into a sequence-like output representation $\boldsymbol{r}\in\mathbb{R}^{d_{\boldsymbol{r}}\times d}$ of sequence length $d_{\boldsymbol{r}}$ and embedding dimension $d$. Let us consider the $l$-th layer with input $\boldsymbol{h}_{(l)}$ and output $\boldsymbol{h}_{(l)}^{\prime}$, where $\boldsymbol{h}_{(L)}^{\prime}$ is equivalent to $\boldsymbol{r}$ for total $L$ layers. Since the output $\boldsymbol{h}_{(l)}^{\prime}$ then becomes the input $\boldsymbol{h}_{(l+1)}\in\mathbb{R}^{d_{\boldsymbol{h}_{(l+1)}}\times d}$ of the next layer, we omit the layer identity $l$ for the sake of clarity. Then, the input $\boldsymbol{h}\in\mathbb{R}^{d_{\boldsymbol{h}}\times d}$ is further specified as the query $\boldsymbol{h}_{Q}$, key $\boldsymbol{h}_{K}$ and value $\boldsymbol{h}_{V}$, and the output $\boldsymbol{h}^{\prime}\in\mathbb{R}^{d_{\boldsymbol{h}}\times d}$ of the current layer is

where $\boldsymbol{W}_{O}$, $\boldsymbol{W}_{Q,i}$, $\boldsymbol{W}_{K,i},\boldsymbol{W}_{V,i}$ are projection matrices, $m$ is the number of heads, and $\boldsymbol{h}_{Q}=\boldsymbol{h}_{K}=\boldsymbol{h}_{V}=\boldsymbol{h}$ in ViT. The concatenation (Concat) and attention (Attn) functions are specified in their original papers [38, 39].

To facilitate effective transfer of pre-trained knowledge while preventing its catastrophic forgetting, the backbone parameters $\theta$ are usually frozen and additional lightweight parameters are introduced to instruct representation learning, referred to as the PET techniques [33]. Here we describe some representative ones (see Fig. 2):

Prompt Tuning (ProT) [9] and Prefix Tuning (PreT) [10] both prepend a few learnable parameters $\boldsymbol{p}\in\mathbb{R}^{d_{\boldsymbol{p}}\times d}$ of sequence length $d_{\boldsymbol{p}}$ and embedding dimension $d$ to $\boldsymbol{h}$, collectively known as the Prompt-based PET. For ProT in ViT, an identical $\boldsymbol{p}$ is prepended to $\boldsymbol{h}_{Q}$, $\boldsymbol{h}_{K}$ and $\boldsymbol{h}_{V}$:

where $[\cdot\,;\cdot]$ represents the concatenation operation along the dimension of sequence length. Since the output in $\mathbb{R}^{(d_{\boldsymbol{h}}+d_{\boldsymbol{p}})\times d}$ has increased dimensions, ProT is often used for only the last layer in CL [5, 7]. In contrast, PreT splits $\boldsymbol{p}$ into $\boldsymbol{p}_{K}\in\mathbb{R}^{d_{\boldsymbol{p}}/2\times d}$ for $\boldsymbol{h}_{K}$ and $\boldsymbol{p}_{V}\in\mathbb{R}^{d_{\boldsymbol{p}}/2\times d}$ for $\boldsymbol{h}_{V}$:

where the output has the same dimension as the input $\boldsymbol{h}\in\mathbb{R}^{d_{\boldsymbol{h}}\times d}$, allowing PreT to be implemented in multiple layers. In particular, the output of PreT can be reframed as

where $f_{{\rm{NonL}}}$ is the nonlinear (NonL) softmax function, and $\lambda(\boldsymbol{h})$ is a scalar that depends on the input [33].

Adapter [11] inserts lightweight neural modules between backbone layers, each usually composed of a down-projection matrix $\boldsymbol{W}_{{\rm{down}}}\in\mathbb{R}^{d\times r}$ that reduces the dimension of $\boldsymbol{h}$ with bottleneck $r$, a nonlinear (NonL) activation function $f_{{\rm{NonL}}}$ and an up-projection matrix $\boldsymbol{W}_{{\rm{up}}}\in\mathbb{R}^{r\times d}$. These modules are implemented with residual connections, acting on the output $\boldsymbol{h}^{\prime}$ in a sequential (Seq) manner, i.e.,

as well as on the input $\boldsymbol{h}$ in a parallel (Par) manner, i.e.,

LoRA [12] approximates the updates of pre-trained parameter matrix $\boldsymbol{W}\in\mathbb{R}^{d\times k}$ with a low-rank decomposition $\boldsymbol{W}+\bigtriangleup\boldsymbol{W}=\boldsymbol{W}+\boldsymbol{W}_{{\rm{down}}}\boldsymbol{W}_{{\rm{up}}}$, where $\boldsymbol{W}_{{\rm{down}}}\in\mathbb{R}^{d\times r}$, $\boldsymbol{W}_{{\rm{up}}}\in\mathbb{R}^{r\times k}$ and $r$ is the low-rank bottleneck. For ViT, LoRA is typically used to update the $\boldsymbol{W}_{Q}$ and $\boldsymbol{W}_{V}$ of a backbone layer. As a special case, when $\boldsymbol{W}$ performs linear projection of the input $\boldsymbol{h}$, the output is modified as

where $s\geq 1$ is a scalar hyperparameter [33].

As we can see, these PET techniques all amount to modulating the (intermediate) representations of $f_{\theta}$, though differing in their specific implementations.

With the widespread use of PTMs in CL, there have been a variety of methods that incorporate PET techniques on a continual basis. Most of these methods have focused on designing appropriate PET architectures to formulate lightweight parameters tailored for CL, which can be conceptually summarized into task-specific parameters, task-shared parameters, and their explicit or implicit combinations. In particular, task-specific parameters require their identity to be predicted during the testing phase, while task-shared parameters need to mitigate catastrophic forgetting when learning each task. We briefly describe these methods here, with a comprehensive summary in Appendix Table VII for further comparison.

L2P [5] constructs a prompt pool $\boldsymbol{p}_{1},...,\boldsymbol{p}_{M}$ of size $M$ and instructs pre-trained representations in a ProT manner. Each prompt is associated with a learnable key, optimized by the cosine distance of the top-$N$ keys to a query function $q(\boldsymbol{x})=f_{\theta}(\boldsymbol{x})[0]$. The most relevant prompt(s) can therefore be selected via uninstructed representations.

DualPrompt [6] employs the task-specific prompt $\boldsymbol{p}^{l}_{t}$ and task-shared prompt $\boldsymbol{p}^{l}$ to instruct respective layers in a PreT manner. The layer-wise $\boldsymbol{p}^{l}_{t}$ is associated with a task-specific key, optimized by its cosine distance to $q(\boldsymbol{x})$, and the best-matched key is selected via uninstructed representations.

S-Prompt [7] employs only the task-specific prompt $\boldsymbol{p}_{t}$ and instructs pre-trained representations in a ProT manner. The task identity is inferred from preserved task centroids with $k$-Nearest Neighbors ($k$NNs). S-Prompt also employs a multi-head output layer associated with the task identity.

CODA-Prompt [8] performs a weighted summation of the prompt pool, i.e., $\boldsymbol{p}=\sum_{i=1}^{M}\alpha_{i}\boldsymbol{p}_{i}$, and instructs multiple layers in a PreT manner. Each weighting factor $\alpha_{i}$ for $i\in[M]$ is associated to a learnable key, optimized by its cosine distance to $q(\boldsymbol{x})$. The inference of $\alpha_{i}$ can therefore construct an appropriate prompt for each input sample.

LAE [14] employs two kinds of task-shared parameters to incorporate knowledge from more recent tasks and more remote tasks, respectively, applicable to PreT, Adapter and LoRA for multiple layers. Their update speeds are regulated via combinatorial strategies such as temporary freezing and exponential moving average (EMA).

Besides, there are several relevant methods with different focuses. For example, SLCA [4] updates the entire backbone with a reduced learning rate, and further preserves pre-trained representations via dedicated covariance matrices. RanPAC [34] projects pre-trained representations to a high-dimension space and preserves them via a shared covariance matrix. These methods are often parameter-inefficient, i.e., the parameter cost is comparable to $\theta$ due to a complexity much larger than $O(d^{2})$, and therefore not prioritized in this work.¹¹1Our preliminary version [13] also employed dedicated covariance matrices as the main implementation to acquire better performance.

Taken together, three notable limitations have surfaced in current progress of harnessing PET techniques for CL. First, the above methods often rely on empirical designs, making it difficult to ascertain their effectiveness in achieving the objective of CL in a pre-training context. In particular, their performance exhibits significant variations across different PTMs and target tasks, as demonstrated in Sec. 6.2. Second, these methods predominantly center around Prompt-based PET, which has been shown to be less effective under self-supervised pre-training [36] and fine-grained tasks [40], leaving underexplored the particular behaviors and potential benefits of other alternatives. Third, these methods share some analogous strategies, such as stabilizing task-shared parameters and recovering pre-trained representations, without a comprehensive analysis of different implementations and assimilation of their respective strengths. Therefore, there is an urgent need to establish a unified framework that incorporates both theoretical and empirical insights for CL with PTMs and PET, which constitutes our main motivation.

Let us consider a CL problem for sequentially arriving tasks $i\in[t]$ with training set $\mathcal{D}_{i}$ of domain $\mathcal{X}_{i}=\bigcup_{j}\mathcal{X}_{i,j}$ and label $\mathcal{Y}_{i}=\bigcup_{j}\mathcal{Y}_{i,j}$, where $j\in[|\mathcal{Y}_{i}|]$ denotes the $j$-th class of task $i$. The goal is to learn a projection from $\mathcal{X}=\bigcup_{i=1}^{t}\mathcal{X}_{i}$ to $\mathcal{Y}=\bigcup_{i=1}^{t}\mathcal{Y}_{i}$ in order to predict the label of an unseen test sample $\boldsymbol{x}$ from all observed tasks, referred to as task-adaptive prediction (TAP). As summarized in Sec. 3.1, there are many representative setups of CL, such as CIL, DIL and TIL. Here we take CIL as a typical scenario for theoretical analysis, and leave the results of DIL and TIL to Appendix A.

CL from Scratch: When training from scratch, the TAP performance $P(\boldsymbol{x}\in\mathcal{X}^{y}|\mathcal{D})$ is to predict across all classes without distinguishing tasks, where $\mathcal{D}=\{\mathcal{D}_{1},...,\mathcal{D}_{t}\}$, $y\in[|\bigcup_{i=1}^{t}\mathcal{Y}_{i}|]$ denotes the ground truth label of $\boldsymbol{x}$, and $\mathcal{X}^{y}$ denotes the domain of class $y$. The restricted definition of CIL further posits two assumptions [41]: the domains of tasks are disjoint (i.e., $\mathcal{X}_{i}\cap\mathcal{X}_{i^{\prime}}=\emptyset$, $\forall i\neq i^{\prime}$), and the domains of classes of the same task are disjoint (i.e., $\mathcal{X}_{i,j}\cap\mathcal{X}_{i,j^{\prime}}=\emptyset$, $\forall j\neq j^{\prime}$). Through predicting which task to perform and then performing that task (i.e., there is an execution order), the CIL probability can be expressed as a hierarchical process of task-identity inference (TII) and within-task prediction (WTP):

Eq. (9) is exactly the main conclusion of a previous theoretical study of CL from scratch [41]. Given the assumptions of disjoint domains and the omitted impact of randomly initialized $\theta$, the TAP performance $P(\boldsymbol{x}\in\mathcal{X}^{y}|\mathcal{D})$ is essentially equivalent to the decomposed CIL performance ${P}(\boldsymbol{x}\in\mathcal{X}_{\bar{i},\bar{j}}|\mathcal{D})$, where $\bar{i}\in[t]$ and $\bar{j}\in[|\mathcal{Y}_{\bar{i}}|]$ denote the ground truth of an $\boldsymbol{x}$ w.r.t. the task identity and within-task index. Here we provide a more intuitive explanation with the definition of class labels and the implementation of output heads. The decomposed CIL performance can be naturally computed in a multi-head manner with two steps: inferring the task identity $\bar{i}\in[t]$, i.e., the TII performance; and predicting the within-task index $\bar{j}\in[|\mathcal{Y}_{\bar{i}}|]$, i.e., the WTP performance. In comparison, the TAP performance $P(\boldsymbol{x}\in\mathcal{X}^{y}|\mathcal{D})$ is computed by performing single-head prediction for global class $y\in[|\bigcup_{i=1}^{t}\mathcal{Y}_{i}|]$. For CL from scratch, the decomposed CIL performance of multi-head inference & prediction is equivalent to the TAP performance of single-head prediction. In this case, the decomposed CIL performance can also be computed in a single-head manner, as many traditional continual learning methods do.

CL with Pre-training: When considering the impact of pre-trained knowledge carried by parameters $\theta$, the TAP is redefined as $P(\boldsymbol{x}\in\mathcal{X}^{y}|\mathcal{D},\theta)$, while the CIL probability of TII and WTP in Eq. (9) is re-written as

It can be seen that both the TAP performance $P(\boldsymbol{x}\in\mathcal{X}^{y}|\mathcal{D},\theta)$ and the CIL performance ${P}(\boldsymbol{x}\in\mathcal{X}_{\bar{i},\bar{j}}|\mathcal{D},\theta)$ are now affected by $\theta$, but in different ways. The pre-trained parameters $\theta$ in TAP affect simultaneously all observed classes (i.e., a large output space $[|\bigcup_{i=1}^{t}\mathcal{Y}_{i}|]$), while in CIL affect separately TII and WTP (i.e., two small output spaces $[t]$ and $[|\mathcal{Y}_{\bar{i}}|]$). Accordingly, the CIL performance is upper bounded by either the TII performance $P(\boldsymbol{x}\in\mathcal{X}_{\bar{i}}|\mathcal{D},\theta)$ or the WTP performance $P(\boldsymbol{x}\in\mathcal{X}_{\bar{i},\bar{j}}|\boldsymbol{x}\in\mathcal{X}_{\bar{i}},\mathcal{D},\theta)$, whereas the TAP performance is not, as the CL tasks and pre-trained knowledge are not conditionally independent from a statistical perspective. For example, an incorrectly predicted task identity results in full errors in predicting within-task index, remaining a performance gap from rectifying the predictions of inter-task classes to make them properly balanced. This also underscores the notable difference of the multi-head inference & prediction from the single-head prediction in the context of pre-training. Such difference tends to be more pronounced if the task structure is not clear (e.g., randomly split classes of the same dataset), and has been empirically validated in our preliminary experiments [13], where the multi-head inference & prediction significantly underperforms the single-head prediction in CL with PTMs.

Therefore, we propose to further optimize TAP along with the improved TII and WTP, formulating the ultimate goal of CL as a multi-objective optimization problem, i.e.,

where $P(\boldsymbol{x}\in\mathcal{X}_{\bar{i},\bar{j}}|\mathcal{D},\theta)$ follows a similar decomposition as in Eq. (10), with TII and WTP having an execution order.

To resolve the above WTP, TII and TAP, we derive the sufficient and necessary conditions with the widely-used cross-entropy loss. Specifically, we define

where ${H}_{\rm{WTP}}$, ${H}_{\rm{TII}}$ and ${H}_{\rm{TAP}}$ are the cross-entropy values of WTP, TII and TAP, respectively. $c\in[|\bigcup_{i=1}^{t}\mathcal{Y}_{i}|]$ denotes the index of all observed classes. $\mathcal{H}(p,q)\triangleq-\mathbb{E}_{p}[\log q]=-\sum_{k}p_{k}\log q_{k}$. $\boldsymbol{1}_{\cdot}$ is a one-hot encoding function.

We now present the first theorem under the CIL setting (see Appendix A for the complete proof and the corresponding extensions to DIL and TIL settings):

With the use of cross-entropy, the CL performance tends to be better as the bounds are tightened. In Theorem 1 we have shown that good performance of WTP, TII and TAP is sufficient to ensure good performance of CL. For completeness, we now study the necessary conditions of a well-performed CL model in Theorem 2.

Theorem 2 suggests that if a CL model is well trained (i.e., with low loss error), then the WTP error, TII error and TAP error for sequentially arriving tasks are always implied to be small. Similar to the connection between TAP and CIL under different assumptions, Theorem 1 and Theorem 2 would degenerate into the main conclusion of the previous theoretical study [41] if the pre-trained knowledge carried by $\theta$ is not considered. This suggests that the presented theorems are particularly directed to the impact of pre-training on CL.

In essence, the TII probability specifies the CL problem with task-wise input samples. Although the definition of “task” in literature can generalize to an incoming training batch of distinct distribution [3], it may not be pertinent in describing realistic data streams with apparent similarity and dissimilarity. Indeed, the CL problem is often associated with the out-of-distribution (OOD) detection [42], i.e., the ability of a model to detect an unseen input sample, which has been shown to behave similarly as task prediction when training from scratch [41]. Inspired by this, we further explore the necessary conditions to optimize TII/OOD for CL in a pre-training context, in order to facilitate adaptive knowledge accumulation from more pronounced distribution changes.

We again use cross-entropy to measure the performance of TII and OOD detection, so as to establish their connection in each task. We first define the OOD detector of the $i$-th task as $P_{i}(\boldsymbol{x}\in\mathcal{X}_{i}|\mathcal{D},\theta)$. Different from the TII probability, the OOD detection probability here is a Bernoulli distribution:

where $H_{{\rm{OOD}},i}$ is the cross-entropy value of an OOD detector of task $i$, and $P_{i}(\boldsymbol{x}\in\mathcal{X}_{i}|\mathcal{D},\theta)$ can be predicted with an appropriate distance function. The TII probability can then be defined with the OOD detectors, i.e., $P(\boldsymbol{x}\in\mathcal{X}_{i}|\mathcal{D},\theta)=\frac{P_{i}(\boldsymbol{x}\in\mathcal{X}_{i}|\mathcal{D},\theta)}{\sum_{j}P_{j}(\boldsymbol{x}\in\mathcal{X}_{j}|\mathcal{D},\theta)}$.

Now we have the following theorem to explore the connection between TII and OOD detection in a pre-training context (see Appendix B for the complete proof):

Theorem 3 shows that the TII performance improves if the OOD detection performance improves, and vice versa. In particular, $H_{\rm{TII}}(\boldsymbol{x})$ converges to 0 as $\epsilon_{i}$ converges to 0. Note that the connection between TII and OOD detection in Theorem 3 for CL with pre-training is similar in form to that for CL from scratch [41]. We further derive the sufficient and necessary conditions of improving CL with WTP, OOD detection and TAP, as detailed in Appendix B.

The principal concept behind HiDe-PET stems from two unique strengths of PTMs in CL: (1) the pre-trained representations can be effectively adapted to the distribution of target tasks through implementing PET techniques, and (2) the distribution of target tasks can be efficiently recovered through preserving statistical information of their pre-trained representations. The optimization of WTP, TII and TAP is described as below.

First, we improve WTP through incorporating task-specific knowledge from $\mathcal{D}_{i}$ to capture the distribution of any task $i\in[t]$. Specifically, we employ multiple sets of task-specific parameters $\boldsymbol{e}_{1},...,\boldsymbol{e}_{t}$ to instruct pre-trained representations, implemented via mainstream PET techniques. Their concrete forms can be defined as the prompt parameters $\{\boldsymbol{p}\}$ for ProT and PreT, the projection matrices $\{\boldsymbol{W}_{{\rm{down}}},\boldsymbol{W}_{{\rm{up}}}\}$ for Adapter and LoRA, etc. When learning task $t$, we keep the previous parameters $\boldsymbol{e}_{1},...,\boldsymbol{e}_{t-1}$ frozen to avoid catastrophic forgetting. In order to inherit knowledge obtained from CL, we employ a PET ensemble strategy that initializes $\boldsymbol{e}_{t}$ with $\boldsymbol{e}_{t-1}$ and optimizes $\boldsymbol{e}_{t}$ with a weighted combination of all previous parameters $\boldsymbol{e}_{t}\leftarrow\alpha\sum_{i\in[t-1]}\boldsymbol{e}_{i}+(1-\alpha)\boldsymbol{e}_{t}$, where $\alpha\in[0,1]$ is a hyperparameter that controls the strength of obtained knowledge that facilitates $\boldsymbol{e}_{t}$ in learning task $t$. For ${H}_{\rm{WTP}}$, we then optimize $\boldsymbol{e}_{t}$ and $\psi$ via the cross-entropy (CE) loss:

Second, we improve TII and TAP through recovering the distributions of pre-trained representations to achieve the optimum over all tasks. Specifically, after learning each task $i\in[t]$, we collect the uninstructed and instructed representations, i.e., the backbone projection of $\mathcal{D}_{i}$ with $f_{\theta}(\cdot)$ and $f_{\theta,\boldsymbol{e}_{i}}(\cdot)$, respectively. We then approximate the distributions of these representations with their statistical information for subsequent recovery. Taking classification tasks as an example, for each class $c\in\mathcal{Y}_{i}$ and $i\in[t]$, we denote the approximated distributions of uninstructed and instructed representations as $\hat{\mathcal{G}}_{i,c}$ and $\mathcal{G}_{i,c}$, respectively.

For ${H}_{\rm{TII}}$, we build an auxiliary output layer $\hat{h}_{\omega}(\cdot):\mathbb{R}^{d}\rightarrow\mathbb{R}^{t}$ with parameters $\omega$, learning the projection from uninstructed representations to the task identity:

where $\hat{\mathcal{S}}_{i,c}$ is a collection of uninstructed representations sampled in a class-balanced manner from $\hat{\mathcal{G}}_{i,c}$ for $c\in\mathcal{Y}_{i}$ and $i\in[t]$. Therefore, we can determine the task identity via uninstructed representations and then obtain the corresponding instructed representations.

For ${H}_{\rm{TAP}}$, we use a similar strategy to optimize the final output layer $h_{\psi}(\cdot):\mathbb{R}^{d}\rightarrow\mathbb{R}^{|\mathcal{Y}|}$ with parameters $\psi$, learning the projection from instructed representations to all observed classes:

where $\mathcal{S}_{i,c}$ is a collection of instructed representations sampled in a class-balanced manner from $\mathcal{G}_{i,c}$ for $c\in\mathcal{Y}_{i}$ and $i\in[t]$.

Improvement of Uninstructed Representations: Since the TII process depends heavily on the uninstructed representations of $\mathcal{D}_{i}$ collected from $f_{\theta}(\cdot)$, its effectiveness tends to be severely affected by the pre-trained checkpoints and target tasks. This issue becomes particularly pronounced when implementing more advanced PET techniques, which better incorporate task-specific knowledge for WTP and are thus more sensitive to TII. To address this issue, we further deploy a set of task-shared parameters $\boldsymbol{g}$ to improve TII in a task-agnostic manner. $\boldsymbol{g}$ is implemented via mainstream PET techniques analogous to $\boldsymbol{e}_{i}$, while optimized with the cross-entropy $\mathcal{L}_{{\rm{CE}}}(\hat{\psi},\boldsymbol{g})$ to accumulate task-shared knowledge from sequentially arriving $\mathcal{D}_{i}$ for $i\in[t]$, where $\hat{\psi}$ is an interim copy of $\psi$ to avoid overwriting. We then use $f_{\theta,\boldsymbol{g}}(\cdot)$ instead of $f_{\theta}(\cdot)$ to collect uninstructed representations²²2For naming consistency, we still use “uninstructed representations” to denote the projection of $f_{\theta,\boldsymbol{g}}(\cdot)$., approximate each $\hat{\mathcal{G}}_{i,c}$ and optimize Eq. (17).

Notably, $\boldsymbol{g}$ needs to overcome its own catastrophic forgetting that leads to not only the loss of information in representation learning but also the representation shift in subsequent recovery. There are many CL methods attempting to address this challenge [14, 34, 4], but their strategies remain sub-optimal in balancing sequentially arriving tasks (see Table IV). Here we propose a simple yet effective strategy by taking advantages of first-session adaptation (FSA) [34, 43] and slow learner (SL) [4, 14]. Specifically, we learn $\boldsymbol{g}$ in the first task with a larger learning rate that is adequately strong for representation learning, and then in subsequent tasks with a smaller learning rate for further fine-tuning. In this way, task-shared knowledge is effectively incorporated into $\boldsymbol{g}$ and accumulates over time.

Recovery of Task Distributions: Since the pre-trained representations are well-distributed in general, there are many feasible strategies to approximate and preserve their distributions $\hat{\mathcal{G}}_{i,c}$ and $\mathcal{G}_{i,c}$. The most straightforward option is to save randomly selected prototypes [44], yet not adequately employing the relationships between them. For classification tasks, the class-wise distribution is typically single-peaked and thus can be naturally approximated as a Gaussian with its mean and covariance [4, 13]. In order to reduce storage complexity, dedicated covariance matrices need to be further simplified for practical use, suffering from information loss to varying degrees [4, 34, 13]. Considering both storage efficiency and task-type generality, our default implementation is to obtain multiple representation centroids with $k$-Nearest Neighbors ($k$NNs) and add Gaussian noise to them. We also provide an empirical comparison of different implementations in Table V.

Overall, the entire HiDe-PET consists of two training stages (see Algorithm 1 and Fig. 3), corresponding to the pre-trained transformer backbone and the (auxiliary) output layer. At test time, HiDe-PET first predicts the task identity $\hat{i}=\hat{h}_{\omega}(f_{\theta,\boldsymbol{g}}(\boldsymbol{x}))$ and then the overall class label $\hat{y}=h_{\psi}(f_{\theta,\boldsymbol{e}_{\hat{i}}}(\boldsymbol{x}))$ (see Fig. 4). Compared to the backbone parameters $\theta$, the trainable parameters $\boldsymbol{e}_{t}$, $\boldsymbol{g}$, $\omega$ and $\psi$, as well as the representation recovery are all lightweight, thus ensuring resource efficiency.

Within HiDe-PET, the parallel organization of $\boldsymbol{e}_{1},...,\boldsymbol{e}_{t}$ and $\boldsymbol{g}$ facilitates the incorporation of task-specific and task-shared knowledge for many representative CL scenarios. In fact, the functions of $\boldsymbol{e}_{1},...,\boldsymbol{e}_{t}$ and $\boldsymbol{g}$ are usually not exclusive, depending on the similarity and dissimilarity between task distributions. Motivated by the intrinsic connection between TII and OOD detection in our theoretical analysis, we unify the task-specific and task-shared PET architectures with a hierarchy of expandable parameter sets (see Fig. 5), which may degenerate into either case of Sec. 5.1. We further explore a particular implementation of this hierarchy in order to accumulate knowledge adaptively from more pronounced distribution changes.

Let us assume the existence of multiple parameter sets that are implemented via mainstream PET techniques and are expanded or retrieved upon the input samples. For example, the sample elements of sequentially learning tasks $i\in[t-1]$ have derived $k$ parameter sets $\boldsymbol{g}_{1},...,\boldsymbol{g}_{k}$. If the incoming $\boldsymbol{x}\in\mathcal{X}_{t}$ is identified as OOD from the previously observed distributions of $\mathcal{X}_{i}$, it learns an expanded set of parameters $\boldsymbol{g}_{k+1}$ through the task-specific loss $\mathcal{L}_{{\rm{CE}}}(\hat{\psi},\boldsymbol{g}_{k+1})$, otherwise it retrieves and updates the most relevant one $\boldsymbol{g}_{j}$ with $j\in[k]$ through $\mathcal{L}_{{\rm{CE}}}(\hat{\psi},\boldsymbol{g}_{j})$, where $\hat{\psi}$ is an interim copy of the output layer parameters $\psi$ to avoid overwriting.

OOD Detection Strategy: Given that the previous $\mathcal{X}_{i}$ are not available in CL to describe their distributions, we take inspirations from recent metric-based OOD detectors [45] and formulate an effective criterion with their uninstructed representations:

where $\hat{\mathcal{G}}_{i}$ for $i\in[t-1]$ denotes the approximated distribution of uninstructed representations. It can be further specified as $\hat{\mathcal{G}}_{i}=\bigcup_{c\in\mathcal{Y}_{i}}\hat{\mathcal{G}}_{i,c}$ for classification tasks. $\lambda_{{\rm{OOD}}}$ denotes the OOD detection threshold. $\boldsymbol{1}(\cdot)$ is the indicator function. ${\rm{Dis}}(\boldsymbol{x},\hat{\mathcal{G}}_{i})$ measures the distance of task-wise distributions, which can be implemented via the average Euclidean distance between $f_{\theta,\boldsymbol{g}}(\boldsymbol{x})$ and $\hat{\boldsymbol{r}}\sim\hat{\mathcal{G}}_{i}$. Consequently, if $\boldsymbol{x}$ is identified as OOD for all tasks $i\in[t-1]$, then it will be associated with $\boldsymbol{g}_{k+1}$. Otherwise, it will retrieve the associated $\boldsymbol{g}_{j}$ for $j\in[k]$ corresponding to the majority of the most relevant task $\hat{i}=\arg\min_{i\in[t-1]}{\rm{Dis}}(\boldsymbol{x},\hat{\mathcal{G}}_{i})$. To overcome catastrophic forgetting when updating $\boldsymbol{g}_{j}$, we employ the same strategy as for learning the task-shared parameters $\boldsymbol{g}$, e.g., a combination of FSA [43] and SL [4]. As a special case, we have $k=1$ if all input samples are identified as in-distribution, for which only $\boldsymbol{g}_{1}$ exists and is equivalent to $\boldsymbol{g}$.

Connection of PET Architectures: We now extend the above discussion with the criterion of OOD detection in Eq. (19). Given a task sequence $1,...,t$, using a larger $\lambda_{{\rm{OOD}}}$ would make $k\rightarrow 1$ and $\{\boldsymbol{g}_{1},...,\boldsymbol{g}_{k}\}\rightarrow\{\boldsymbol{g}\}$, while using a smaller $\lambda_{{\rm{OOD}}}$ would make $k\rightarrow t$ and $\{\boldsymbol{g}_{1},...,\boldsymbol{g}_{k}\}\rightarrow\{\boldsymbol{e}_{1},...,\boldsymbol{e}_{t}\}$. Therefore, the representation learning of HiDe-PET in Sec. 5.1 is equivalent to a parallel combination of these two extreme conditions for TII and WTP, respectively. This is a reasonable choice as most CL benchmarks employ randomly split classes of the same dataset as the task sequence, i.e., there is no actual task structure.

Instead, Eq. (19) applies to the apparent similarity and dissimilarity between task distributions, which is more realistic in applications and enables adaptive knowledge accumulation from CL for enhanced utilization. Here we leverage the unique property of LoRA-based PET to construct a specialized implementation, serving as a plug-in module for Algorithm 1. Unlike the commonly-used Prompt-based PET that updates only attached tokens, the LoRA-based PET specifies $\boldsymbol{g}_{1},...,\boldsymbol{g}_{k}$ as the approximated updates of $\theta$, where the most relevant $\boldsymbol{g}_{j}$ is selected and temporarily added to $\theta$ in CL. Therefore, the learning of subsequent tasks can be significantly improved from the accumulated knowledge and further contribute to it (see Fig. 5). Moreover, this allows for the flexible evolution of pre-trained knowledge with target tasks in a lifelong manner, deviating from the conventional practice of fixing it at the initial checkpoint.

In brief, Sec. 4 and 5 serve as a unified framework for CL with PTMs and PET, applicable to explore the distinct impacts of implementation strategy, PET technique and PET architecture, as well as the adaptive knowledge accumulation for enhanced utilization.

To ensure the breadth and adequacy of the experiments, we consider a variety of CL benchmarks, pre-trained checkpoints, recent strong baselines, PET techniques and evaluation metrics. For the sake of comparison fairness, we follow the official implementations to reproduce all baselines.

Benchmark: We focus on four datasets that are widely used in previous work to evaluate CL [5, 6, 8, 14, 4], and split each into 10 target tasks of disjoint classes for CIL experiments. The first two are general datasets, such as CIFAR-100 [46] of 100-class small-scale images that are common objects in the real world and ImageNet-R [47] of 200-class large-scale images that are hard examples of ImageNet-21K [48] or newly collected examples of different styles. The latter two are fine-grained datasets, such as CUB-200 [49] of 200-class bird images and Cars-196 [50] of 196-type car images. We mainly consider three representative pre-trained checkpoints that differ in paradigm and dataset, including Sup-21/1K, iBOT-21K and Sup-Weak. Specifically, Sup-21/1K [8] is essentially a supervised checkpoint that performs self-supervised learning on ImageNet-21K and then supervised fine-tuning on ImageNet-1K. iBOT-21K [51] is a self-supervised checkpoint that currently achieves the first-place classification performance for self-supervised learning on ImageNet-21K. Sup-Weak [52] is a supervised checkpoint on a subset of ImageNet-1K, in which 389 similar classes to subsequent CL are intentionally removed.

Baseline: We compare our HiDe-PET with a range of recent strong baselines as described in Sec. 3.3, including L2P [5], DualPrompt [6], S-Prompt [7], CODA-Prompt [8] and LAE [14]. In brief, these baselines cover different PET architectures, but mainly target the Prompt-based PET. LAE [14] is the most recent baseline among them, and is also the most relevant one to ours as it applies to a variety of mainstream PET techniques. Similar to the previous work [13], we modify the implementation of S-Prompt [7] by inserting task-specific prompts into multiple MSA layers in a PreT manner and using a single-head output layer, in order to evaluate the impact of PET architecture and better adapt to the CIL experiments. The modified S-Prompt is referred to as S-Prompt++. Following LAE [14], we consider three kinds of mainstream PET techniques in our HiDe-PET and our preliminary version [13], including PreT [10], LoRA [12] and Adapter [11].

Evaluation: We use $A_{i,i^{\prime}}$ to denote the prediction accuracy on task $i$ after learning task $i^{\prime}$ (with single-head evaluation for CIL), and define the average accuracy of all seen tasks as $AA_{i^{\prime}}=\frac{1}{i^{\prime}}\sum_{i=1}^{i^{\prime}}A_{i,i^{\prime}}$. After learning all tasks, we report both the final average accuracy ${\rm{FAA}}=AA_{t}$ that serves as the primary metric to evaluate CL performance, and the cumulative average accuracy ${\rm{CAA}}=\frac{1}{t}\sum_{i=1}^{t}AA_{i}$ that further reflects the historical performance. Moreover, we evaluate the behaviors of different PET techniques with the average learning accuracy ${\rm{ALA}}=\frac{1}{t-1}\sum_{i=2}^{t}A_{i,i-1}$ for learning plasticity and the final forgetting measure ${\rm{FFM}}=\frac{1}{t-1}\sum_{i=1}^{t-1}\max_{i^{\prime}\in[t-1]}(A_{i,i^{\prime}}-A_{i,t})$ for memory stability, as well as evaluate the TII performance in our HiDe-PET.

Implementation: We follow similar implementations as previous work.³³3The training regime and supervised checkpoints are identical to those in CODA-Prompt [8], which are slightly different from those in our preliminary version [13] and lead to some performance differences. Specifically, we employ a pre-trained ViT-B/16 backbone with an Adam optimizer ($\beta_{1}=0.9$, $\beta_{2}=0.999$), a batch size of 128, and a cosine-decaying learning rate of 0.001. We train Split CIFAR-100 for 20 epochs and other benchmarks for 50 epochs to ensure convergence on each task. The image inputs are resized to $224\times 224$ and normalized to $[0,1]$. The PET architecture of each baseline is also similar to its original paper, which has been shown to yield strong performance. Specifically, L2P [5] is implemented with the prompt pool size $M=30$, the prompt length $d_{\boldsymbol{p}}=5$ and the Top-5 keys. DualPrompt [6] is implemented with the prompt length $d_{\boldsymbol{p}}=5$ of task-shared prompts inserted into layers 1-2 and the prompt length $d_{\boldsymbol{p}}=20$ of task-specific prompts inserted into layers 3-5. S-Prompt++ [7] is implemented similarly to DualPrompt but replaces all task-shared prompts with task-specific prompts, inserted into layers 1-5 with the prompt length $d_{\boldsymbol{p}}=20$. CODA-Prompt [8] is implemented with the prompt pool size $M=100$ and the prompt length $d_{\boldsymbol{p}}=8$, inserted into layers 1-5. LAE [14] and our HiDe-PET are implemented with the prompt length $d_{\boldsymbol{p}}=20$ for PreT, and the low-dimension bottleneck $r=10$ for Adapter and LoRA, inserted into layers 1-5. We insert the Adapter modules in both sequential and parallel manners, while employ LoRA to update both $\boldsymbol{W}_{K}$ and $\boldsymbol{W}_{V}$. Therefore, the extra parameter costs of PreT, Adapter and LoRA are identical [14]. Note that, our HiDe-PET and our preliminary version [13] adopt a similar PET architecture as S-Prompt++, but replace the task-specific keys with an auxiliary output layer $\hat{h}_{\omega}$ to predict the task identity and further preserve statistical information of pre-trained representations.⁴⁴4We consider a lightweight implementation in terms of the auxiliary output layer and representation recovery, which slightly compromise the performance but largely improve resource efficiency.

Now we present the results of our empirical investigation, including the overall performance of all methods, an ablation study of the three hierarchical components, the distinct impacts of implementation strategy, PET technique and PET architecture, as well as the adaptive knowledge accumulation over similar and dissimilar tasks.

Overall Performance: Table I summarizes the results of all methods across various pre-trained checkpoints and CL benchmarks. It can be seen that our HiDe-PET implemented with three mainstream PET techniques achieves consistently the highest performance, and the performance lead tends to be more pronounced under the more challenging scenarios. Specifically, the performance of all methods is affected to varying degrees when considering fine-grained tasks (i.e., CUB-200 and Cars-196) and weakened pre-training in terms of self-supervised paradigm (i.e., iBOT-21K) and reduced pre-training samples (i.e., Sup-Weak). Among these competitors, the sub-optimality of Prompt-based PET in CL is clearly exposed, which underperforms LoRA/Adapter-based PET within and across methods. The LoRA/Adapter version of LAE [14] is the strongest competitor but still severely affected by the double challenges of pre-trained checkpoints and CL benchmarks. In contrast, our HiDe-PET adapts to them effectively with strong generality.

It is noteworthy that self-supervised pre-training is often considered more practical than supervised pre-training, owing to the expense of annotating massive pre-training samples [13, 4]. Meanwhile, Sup-Weak avoids potential overlap between PTMs and target tasks, providing a more restrictive scenario for CL [52]. Sup-Weak is also more analogous to the widely used setting of CIL experiments in literature, i.e., the model first learns half of the classes and then learns the other classes in multiple incremental phases, where the baselines of Prompt-based PET have been shown to perform poorly on it [54]. These considerations underscore more profound advantages of our HiDe-PET in CL. We further evaluate CL under DINOv2 of ViT-B/14 [53], a state-of-the-art self-supervised checkpoint that largely improves iBOT-21K with additional training tricks and more pre-training data, and HiDe-PET consistently outperforms LAE [14] by a wide margin (see Table II).

Ablation Study: Table III presents an ablation study to validate the effectiveness of optimizing the three hierarchical components in HiDe-PET. Specifically, we progressively incorporate the designs of within-task prediction (WTP), task-identity inference (TII) and task-adaptive prediction (TAP) on the top of a naive architecture, which consists of only task-specific parameters $\boldsymbol{e}_{1},...,\boldsymbol{e}_{t}$ implemented via mainstream PET techniques. In general, the optimization of each component all contributes to the strong performance of HiDe-PET. Although their contributions are relatively comparable under supervised pre-training and general tasks, the improvement of TAP becomes more significant under self-supervised pre-training and fine-grained tasks, demonstrating the necessity of TAP within the CL objective. Besides, the improvement of TII often becomes more apparent with WTP+TAP rather than with WTP alone, suggesting that WTP, TII and TAP operate in concert rather than in isolation.

Implementation Strategy: Now we evaluate the implementation strategy of task-shared parameters and representation recovery, which are both important for the optimization of our HiDe-PET and potentially shared by many recent methods. In contrast to task-specific parameters discussed above, task-shared parameters $\boldsymbol{g}$ aim to improve pre-trained representations in a task-agnostic manner, demanding effective strategies to mitigate catastrophic forgetting. Various strategies have been employed in previous work, including (1) fix-and-tuning (F&T) [14], which updates the output layer with frozen $\boldsymbol{g}$ in earlier epochs and then updates $\boldsymbol{g}$ for representation learning in later epochs; (2) first-session adaptation (FSA) [43], which updates $\boldsymbol{g}$ for representation learning exclusively from the first task and then fixes $\boldsymbol{g}$ in subsequent tasks; (3) slow learner (SL) [4], which reduces the learning rate of $\boldsymbol{g}$ in all tasks; and (4) exponential moving average (EMA) [14], which employs an interim copy of $\boldsymbol{g}$ to learn each task and then updates $\boldsymbol{g}$ with a small momentum.

However, most of these strategies have their own limitations. Both F&T and SL impose restrictions on the extent of updates, sacrificing the effectiveness of representation learning and suffering from potential representation shifts in subsequent recovery. FSA adeptly integrates knowledge from the first task and completely avoids representation shifts, but is unable to perform subsequent representation learning. Considering their complementary properties, we propose a simple but effective strategy that employs FSA for learning the first task and SL for learning subsequent tasks, which clearly outperforms other strategies (see Table IV). Notably, EMA can be seen as a coarse implementation of FSA+SL and indeed achieves the second-highest performance. Therefore, the task-shared parameters in HiDe-PET may also be implemented with EMA by updating $\boldsymbol{g}$ from each $\boldsymbol{e}_{i}$ for $i\in[t]$, offering a slight compromise in performance but reducing a half of the training cost.

On the other hand, we evaluate effective strategies for representation recovery. For classification tasks, the pre-trained representations of each class tend to be single-peaked and therefore can be modeled as a Gaussian with dedicated mean and covariance. Although the covariance achieves considerable performance as shown in Table V, it requires a storage complexity $O(d^{2})$ for embedding dimension $d$, which is comparable to the MSA layer of the backbone. There are three alternatives that reduces the storage complexity to $O(d)$, such as simplifying the covariance to variance, preserving randomly selected prototypes, and obtaining multiple representation centroids with $k$NNs. Among them, the multi-centroid demonstrates superior performance and is applicable to different task types, which therefore becomes our default implementation. Interestingly, the variance achieves comparable performance as the covariance and the multi-centroid under general tasks (i.e., Split ImageNet-R) and supervised pre-training (i.e., Sup-21/1K) while requires negligible parameter costs. This further strengthens the advantages of our HiDe-PET in such scenarios targeted by previous work.

Besides, we note that the proposed PET ensemble of task-specific parameters ensures efficiency and scalability due to the lightweight nature of mainstream PET techniques. As described in Sec. 3.2, ProT and PreT employ $\boldsymbol{p}\in\mathbb{R}^{d_{\boldsymbol{p}}\times d}$ with $d_{\boldsymbol{p}}\ll d$, while Adapter and LoRA employ $\boldsymbol{W}_{{\rm{down}}}\in\mathbb{R}^{d\times r}$ and $\boldsymbol{W}_{{\rm{up}}}\in\mathbb{R}^{r\times d}$ with $r\ll d$. In our default implementation, the additional parameter costs of PreT, Adapter, and LoRA for learning each task are kept the same, i.e., $d_{\boldsymbol{p}}=20$ for PreT, and $r=10$ for Adapter and LoRA, inserted into layers 1-5. Therefore, the additional parameter cost is 0.073M with embedding dimension $d=768$. Even though the model needs to learn a sufficiently long task sequence, e.g., 100 tasks, the total parameter cost is only 7.3M (around 8.5% of the ViT-B/16 backbone).

PET Technique: While mainstream PET techniques universally amount to modulating specific hidden states of the PTMs [33], their potential differences in CL are noteworthy. As mentioned above, Prompt-based PET usually lags behind LoRA/Adapter-based PET for both LAE and HiDe-PET (see Table I and Fig. 6). The performance gap tends to be more pronounced under the more challenging scenarios of pre-trained checkpoints and CL benchmarks. A major cause is the limited capacity of Prompt-based PET (i.e., updating attached tokens) in representation learning compared to LoRA/Adapter-based PET (i.e., updating network parameters), especially when handling self-supervised pre-training [36] and fine-grained tasks [40], as validated in our results on both task-specific parameters (see WTP in Table III) and task-shared parameters (see FSA+SL in Table IV).

Beyond the overall performance, the choice of PET technique also exerts distinct influences on the three hierarchical components. Compared to Prompt-based PET, LoRA/Adapter-based PET excels in WTP performance through more effectively incorporating task-specific knowledge, but reveals a heightened sensitivity to TII performance, manifested in the errors of predicting an inappropriate set of task-specific parameters (i.e., mismatched representations for each task lead to more errors in the final prediction). This effect is further compensated by learning a robust TAP function. As shown in Table III, the effectiveness of TII is remarkably pronounced when coupled with WTP+TAP for LoRA/Adapter-based PET, whereas diminishes for either Prompt-based PET or WTP alone. Moreover, our HiDe-PET outperforms its preliminary version [13] especially in LoRA/Adapter-based PET and the more challenging scenarios (see Fig. 6), thanks to the improved TII performance with task-shared parameters.

PET Architecture: The generality of our HiDe-PET is also reflected in its PET architecture, which strategically exploits both task-specific and task-shared parameters for representation learning. These two kinds of parameters are used to acquire knowledge with different levels of differentiation and need to overcome their respective challenges as described in Sec. 3.3. Within HiDe-PET, they both contribute to the outstanding performance in Table I and complement each other (see WTP in Table III and FSA+SL in Table IV). In contrast, our preliminary version [13] and LAE [14] exclusively engage either task-specific or task-shared parameters, missing out on fully harnessing the benefits of PTMs and PET. We further present an extensive comparison of our HiDe-PET, our preliminary version and LAE in terms of the overall performance, memory stability and learning plasticity, so as to better demonstrate the respective contributions of different PET architectures (see Fig. 6). It can be seen that the use of task-shared parameters in HiDe-PET can largely improve ALA of new tasks and FAA of all tasks, but probably results in a slight increase in FFM of old tasks due to the ongoing updates in our default implementation (i.e., FSA+SL). This trend is comparably more pronounced under Sup-Weak that is more demanding to update the pre-trained representations, with FAA, ALA and FFM increasing on average by 6.83%, 9.64% and 2.21%, respectively. We further evaluate alternative implementations of task-shared parameters in Table IV, where FSA has no extra forgetting but performs less well than FSA+SL. The users may select appropriate implementations according to their customised requirements of evaluation metrics.

It is noteworthy that previous work such as L2P [5] and DualPrompt [6] also explicitly or implicitly exploit both task-specific and task-shared prompts, but in a sequential manner to instruct representation learning of each task (corresponding to WTP in our framework). In contrast, our HiDe-PET optimizes these two kinds of parameters in a parallel manner to improve the three hierarchical components, allowing for a more adequate differentiation of the acquired knowledge. Interesting, using only task-shared parameters coupled with representation recovery within HiDe-PET (i.e., FSA+SL in Table IV) has already achieved better performance than these methods (see Sec. 7 for a conceptual explanation), serving as a strong baseline to evaluate current progress. The inherent connections of task-specific and task-shared parameters will be further explored below with a PET hierarchy inspired by OOD detection.

Adaptive Knowledge Accumulation: As analyzed in Sec. 5.2, the use of $\boldsymbol{e}_{1},...,\boldsymbol{e}_{t}$ and $\boldsymbol{g}$ can be seen as a special case tailored for target tasks randomly split from the same dataset, i.e., there is no actual task structure. When considering more realistic CL scenarios with apparent similarity and dissimilarity between task distributions, we devise a hierarchy of expandable parameter sets $\boldsymbol{g}_{1},...,\boldsymbol{g}_{k}$ upon OOD detection to achieve adaptive knowledge accumulation (AKA), and focus on the LoRA-based PET to examine if the pre-trained knowledge can evolve flexibly with target tasks in CL. Here we construct such scenario with the two fine-grained datasets (i.e., CUB-200 [49] and Cars-196 [50]) and another two (i.e., Dogs-120 [55] and Aircraft-102 [56]), which cover both natural and artificial objects. Each dataset is randomly split into 10 tasks. We collect 5 tasks per dataset and mix then as a task sequence (20 tasks in total) for CL, while leaving the rest 5 tasks per dataset for validation.

In CL, the OOD detection threshold $\lambda_{{\rm{OOD}}}$ determines the expansion of parameter sets. As shown in Fig. 7, using a larger $\lambda_{{\rm{OOD}}}$ tends to expand fewer parameter sets, and vice versa. In particular, the choice of $\lambda_{{\rm{OOD}}}$ is relatively insensitive and can consistently construct one parameter set for each dataset (with $\lambda_{{\rm{OOD}}}=0.7$ or $0.8$) under different pre-trained checkpoints. Then we validate the effectiveness of AKA in Table VI. Inspired by a recent work [57], we evaluate the improvements of pre-trained knowledge through the average accuracy of learning the validation tasks under large-shot or few-shot setting. Through selecting the most relevant $\boldsymbol{g}_{j}$ and adding it to $\theta$, the pre-trained backbone $f_{\theta}$ is able to learn each task more effectively. With the improved $f_{\theta}$, the overall performance of CL (i.e., FAA and CAA) for the mixed task sequence is also significantly enhanced.

Interestingly, the idea of updating the pre-trained backbone with a mixture of LoRA experts [58] has been shown effective to accumulate knowledge from multi-task learning, which is consistent with our results. In contrast, the design of the OOD detection, FSA+SL, and representation recovery enables our HiDe-PET to achieve this purpose in a lifelong manner. Besides, our HiDe-PET can also adapt to task-agnostic CL [3] through expanding $\boldsymbol{e}_{1},...,\boldsymbol{e}_{t}$ upon the OOD detection. We leave it as a further work.