Domain Incremental Lifelong Learning in an Open World

In this study, we aim to sequentially learn $N$ tasks $T_{1},\cdots,T_{N}$ that are presented in $L$ different formats $F_{1},\cdots,F_{L}$, $(L\leq N)$. Each task $T_{i}$ is presented in a specific format $F_{j}$ (such as “Classification” or “Summarization”), and each training sample of $T_{i}$ is a tuple of a context $C$, a question $Q$, and an answer $A$: ($C,Q,A$). Note that the format of each task can be easily inferred from the context-question pair ($C,Q$). Our model $g_{\theta}$ is built to predict $A$ based on $C$ and $Q$. We also consider a more challenging open domain lifelong learning setting, i.e., the model needs to predict answers for unseen tasks. Therefore, we collect another $N^{\prime}$ unseen tasks $T_{N+1},\cdots,T_{N+N^{\prime}}$ that are only used for testing. We assume that all task identities of inputs are not available in the testing phase.

We follow previous approaches to serialize the context $C$, question $Q$, and answer $A$ into text sequences Khashabi et al. (2020); Zhong et al. (2022a) and use a prompt-enhanced encoder-decoder model $g_{\theta}$ to learn each task $T_{i}$ in Diana. We use soft prompts Liu et al. (2021); Lester et al. (2021); Vu et al. (2022) in our study, i.e., each prompt is a sequence of trainable embeddings that are randomly initialized and learned in the training process. For each training sample $(C,Q,A)$ from task ${T}_{i}$, we first construct a prompt $P(C,Q)$ based on ($C,Q$). Then the encoder takes in the concatenation of $P(C,Q)$, $C$, and $Q$ and the decoder predicts $A$, i.e., $A=g_{\theta}([P(C,Q);C;Q])$, in which “$[;]$” denotes the sequence concatenation operation.

Four types of prompts are contained in $P(C,Q)$, i.e., $P(C,Q)=[P_{g};P_{f}(F_{j});P_{t}(T_{i});P_{m}(C,Q)]$ (Figure 2c). Specifically, $P_{g}$ is a general prompt, $P_{f}(F_{j})$ is a format prompt (where $F_{j}$ is the format of task $T_{i}$), $P_{t}(T_{i})$ is a task prompt and $P_{m}(C,Q)$ is a combined meta prompt. These four types of prompts are organized hierarchically so that they are shared by samples in different granularities:

1. General Prompt $P_{g}$ is shared for all training tasks so that it encodes global task knowledge.

2. Format Prompt $P_{f}(F_{j})$ is shared between tasks in the same format $F_{j}$ so that it captures format-related knowledge, i.e., knowledge that is shared between tasks in the format $F_{j}$.

3. Task Prompt $P_{t}(T_{i})$ is specifically allocated for the task $T_{i}$ and it is only shared for samples from $T_{i}$. We use $P_{t}(T_{i})$ to learn task-specific knowledge. Moreover, to explicitly model samples from unseen tasks, we enlarge the set of task prompts with $L$ extra prompts $\hat{P_{t}}(F_{1})$, $\cdots$, $\hat{P_{t}}(F_{L})$, in which each prompt $\hat{P_{t}}(F_{j})$ models the unseen task for a particular format $F_{j}$.

4. Meta Prompt $P_{m}(C,Q)$ is a dynamic combination of various instance-level prompts. Specifically, we maintain $M$ instance-level meta prompts $\{P_{m}^{i}\}_{i=1}^{M}$ and dynamically combine these prompts based on the ($C,Q$) to obtain $P_{m}(C,Q)$. $P_{m}(C,Q)$ captures the knowledge shared between similar training instances.

We expect these four types of prompts can capture knowledge from different granularities since they are shared in different scopes. Moreover, to facilitate knowledge sharing, we allocate a key vector $\bm{k}_{t}(T_{i})$ and $\bm{k}_{m}^{j}$ to each task prompt $P_{t}(T_{i})$ and meta prompt $P_{m}^{j}$, respectively, and build a fixed text encoder $h$ to map a context-question pair ($C,Q$) to a query vector $\bm{q}=h(C,Q)$. A two-stage learning process is introduced in Diana to learn these keys and $P(C,Q)$. Specifically, the first stage focuses on learning a representation space for prompt keys so that we can determine proper prompts to construct $P(C,Q)$. The second stage optimizes the constructed prompt $P(C,Q)$ and the backbone language model. These two stages are detailed in the following sections.

We first optimize key vectors assigned to each task prompt and meta prompt to construct the prompt $P(C,Q)$ for each input ($C,Q$). Note that these key vectors are only used to determine the task prompt and meta prompt in $P(C,Q)$ because the general prompt $P_{g}$ is shared by all tasks in Diana, and the format prompt $P_{f}(F_{j})$ can be determined based on the format of $C$ and $Q$ directly.

Task Prompt Keys help to determine the task prompt in $P(C,Q)$. Specifically, for a given input ($C,Q$), we first calculate its query vector $\bm{q}$ and then determine the most similar task prompt key $\bm{k}_{t}(T_{i})$ to $\bm{q}$. The task prompt $P_{t}(T_{i})$ associated with $\bm{k}_{t}(T_{i})$ is used to construct $P(C,Q)$.

Ideally, the key vector $\bm{k}_{t}(T_{i})$ for a task prompt $P_{t}(T_{i})$ should be located near samples from task $T_{i}$ and distant to samples from other tasks $T_{j}$ ($j\neq i)$. Therefore, when learning each task $T_{i}$, we maintain a small memory buffer $\mathcal{M}$ for samples from previously learned tasks $T_{j}$, ($j<i$), and design the following exponential angular triplet loss Ye et al. (2021) to enforce the above property:

in which the operator $||\cdot,\cdot||$ determines the distance between two input vectors (here we use cosine distance), ($C_{n}$, $Q_{n}$) is a negative sample extracted from the memory buffer $\mathcal{M}$:

Meta Prompt Keys help to combine these instance-level meta prompts $\{P_{m}^{i}\}_{i=1}^{M}$ to produce $P_{m}(C,Q)$. Specifically, for each input ($C,Q$), we select $M^{\prime}$ meta prompt keys that are closest to its query vector $\bm{q}=h(C,Q)$. Then $P_{m}(C,Q)$ is obtained by concatenating these $M^{\prime}$ meta prompts. Intuitively, the knowledge associated with ($C,Q,A$) is distributed in these $M^{\prime}$ meta prompts.

When learning meta prompt keys, we expect the distribution of these keys to balance two properties: diversity and locality (Figure 3). Specifically, the diversity property aims to distribute these keys to the whole vector space so that every meta prompt can be involved in the training process. The locality property aims to cluster similar meta prompts keys so that the knowledge of each sample can be better shared. For each input $C$ and $Q$, we propose the following loss to enforce the above two properties:

where $\mathcal{S}(C,Q)$ is the index set of these $M^{\prime}$ meta prompt keys that are closest to $h(C,Q)$, $\eta$ and $\gamma$ are scalar hyper-parameters for the distance margin. Specifically, the first term in Eq. 3 enforces the locality property by pulling these $M^{\prime}$ meta prompt keys around the query vector. The second term enforces the diversity property by pushing these meta prompt keys away from each other to occupy the whole vector space.

Note that Eq. 3 only involves a single query $h(C,Q)$ from the current task. This may limit the learned meta prompt keys since samples from previously learned tasks are not considered. In this study, we extend Eq. 3 to better shape the distributions of meta prompt keys with the help of the memory buffer $\mathcal{M}$, in which samples from previously learned tasks are contained. Specifically, when learning the task $T_{i}$, we first calculate query vectors for samples in $\mathcal{M}$ and then group these query vectors into $B$ clusters (we set $B=5\times i$ in our experiments, where $i$ is the number of received tasks). Centroids of these $B$ clusters are denoted as $\bm{c}_{1},\cdots,\bm{c}_{B}$. For each sample ($C,Q$) from $\mathcal{M}$, the subsequent loss is optimized:

where $\bm{c}_{k}$ is the centroid to which ($C,Q$) belong. The above loss enforces the global diversity by scattering meta prompt keys to each centroid.

Scheduled Sampling of Task Prompts When training Diana, the task ID of each sample ($C,Q$) is given so that we can directly get the task prompt $P_{t}(T_{i})$. However, naively using golden truth task IDs leads to an exposure bias issue, i.e., task IDs inferred in testing may not always be correct.

In this study, we introduce a scheduled sampling process to tackle the exposure bias issue. Specifically, for a given sample ($C,Q,A$) in the $k$-th training step, we toss a coin and use the golden truth task ID with probability $\epsilon_{k}$, or use the task ID inferred based on task prompt keys with probability $1-\epsilon_{k}$ Bengio et al. (2015). Note that when starting to learn each task, prompt keys are not well optimized, and thus the selected task ID is not accurate. Therefore, we set the value of $\epsilon_{k}$ to favor the golden truth task ID at the beginning (i.e., when $k$ is small) and gradually switch to the inferred task ID as the training proceeds (i.e., when $k$ is large), i.e., a linear decrement of $\epsilon_{k}$ is scheduled:

in which $\alpha$ and $\beta$ are scalar hyper-parameters.

Note that LL models may encounter another source of exposure bias since we may receive inputs from unseen tasks in the testing phase. In this study, we use these $L$ extra prompts $\hat{P}_{t}(F_{1}),\cdots,\hat{P}_{t}(F_{L})$ to explicitly model unseen tasks. Specifically, for each training sample ($C,Q,A$), we first determine its task format $F_{j}$ based on ($C,Q$), and allocate a small probability to use $\hat{P}_{t}(F_{j})$ as its task prompt in $P(C,Q)$. In this way, we can capture general knowledge about all tasks for a given format in $\hat{P}_{t}(F_{j})$ and expect the knowledge to facilitate handling unseen tasks.

Train with LM Loss For each training sample ($C,Q,A$), we first construct the prompt $P(C,Q)$ using approaches introduced above, and then optimize $P(C,Q)$ together with the encoder-decoder model $g_{\theta}$ using the following LM loss:

The overall loss that we optimize for Diana is:

After learning each task $T_{i}$, we select a small number of samples from $T_{i}$ based on the query vector of each sample to update the memory $\mathcal{M}$. This selection process aims to maintain diverse samples in $\mathcal{M}$. More details are in Appendix B.

See summarized training process in Algorithm 1.

When testing, we determine the prompt $P(C,Q)$ for each input context $C$ and question $Q$, and use the learned model $g_{\theta}$ to predict the answer $A$.

Adaptive Decision Boundaries (ADB) are used to select proper task prompts in the testing phase. Specifically, for each task $T_{i}$, a scalar boundary $\delta_{i}$ is constructed following the approach proposed by Zhang et al. (2021). An input ($C,Q$) is regarded as a sample from unseen tasks if its query vector $h(C,Q)$ falls outside the boundary of every task:

For samples from unseen tasks, we use the prompt $\hat{P}_{t}(F_{j})$ as its task prompt in $P(C,Q)$, where $F_{j}$ is the format of ($C,Q$).

Answer Prediction is performed with a greedy decoding process:

We use two sets of tasks to evaluate Diana:

1. decaNLP tasks: We follow Sun et al. (2019a) to select 5 tasks from the decaNLP McCann et al. (2018) to train Diana. These tasks cover 3 different formats: Span Extraction, Sequence Generation, and Text Classification. We also collect $N^{\prime}=3$ additional tasks for each of these 3 format from decaNLP to serve as unseen tasks in the testing phase, i.e., our model is trained on $N=5$ seen tasks while tested on 8 tasks;

2. QA tasks: The second set focuses on question answering (QA) benchmarks. Specifically, we use 8 QA datasets over 3 QA formats, i.e., Extractive QA, Abstractive QA and Multiple-Choice QA to train Diana. We also collect $N^{\prime}=3$ additional QA datasets for each of these three formats as unseen tasks, i.e., our model is trained on $N=8$ seen tasks while tested on 11 tasks.

Note that task IDs for all testing samples are not available in our experiments. See Appendix C,J for more details of our dataset settings.

Individual tasks from above two task sets are evaluated following McCann et al. (2018) and Zhong et al. (2022a), respectively (see Appendix C). To evaluate the LL performance of Diana, we build a performance matrix $R\in\mathbb{R}^{N\times(N+N^{\prime})}$, where $R_{i,j}$ is the model performance on task $T_{j}$ after learning task $T_{i}$. The following LL metrics are computed:

1. Average Performance $A_{N}$ and $A_{N^{\prime}}$ is defined as the average performance of the final model on $N$ seen tasks and $N^{\prime}$ unseen tasks, respectively:

2. Average Forget $F_{N}$ is defined as the average performance decrease of each task after it is learned:

In our experiments, we perform five runs with different random seeds and task orders. All reported metric scores are averages of these five runs. Ideally, we expect a strong LL model to yield high $A_{N}$ and $A_{N^{\prime}}$ scores, and low $F_{N}$ scores.

We use T5-base Raffel et al. (2020) to initialize our encoder-decoder model, and set the lengths of soft prompts $P_{g}$, $P_{f}$, $P_{t}$, $P_{m}$ to 20, 40, 40, 20, respectively. We maintain totally $M=30$ meta prompts, and for each sample ($C,Q$) we choose $M^{\prime}=5$ meta prompts to construct $P_{m}(C,Q)$. We use the AdamW Loshchilov and Hutter (2017) optimizer with a learning rate of 1e-4 and batch size of 64. Each task is trained for five epochs. We set $\eta=0.15$ and $\gamma=0.3$ in Eq. 3 and $\alpha=0.9$ and $\beta=3e-4$ in Eq. 5. We maintain $50$ samples from each learned task in the memory $\mathcal{M}$. All experiments are performed on 4 V100 GPUs, and the computational cost of our model is analyzed in Appendix G. See more details in Appendix A.

We use the following competitive baselines covering all three types of LL models:

1. Regularization-based methods: EWC Kirkpatrick et al. (2017) adopts the elastic weight consolidation approach to add regularization on parameter changes; FLCB Gao et al. (2022) uses knowledge learned from previous tasks to guide future task learning; 2. Rehearsal-based methods: ER Chaudhry et al. (2019b) replays memory samples from previous tasks to consolidate learned knowledge; DER++ Buzzega et al. (2020) augments ER with a $L_{2}$ loss on the soft labels; AFPER Mi et al. (2020) combines ER with an adaptive elastic weight consolidation mechanism; 3. Architecture-based methods: AdapterCL Madotto et al. (2021a) allocates separate adapters for different tasks; L2P Wang et al. (2022b) attaches a group of prompts on a pre-trained model to share fine-grained knowledge; DualPrompt Wang et al. (2022a) uses different prompts to encode task-invariant and task-specific knowledge; ProQA Zhong et al. (2022a) uses a unified structural prompt to implement LL models. Note that ProQA is designed for task incremental learning that requires accessing task IDs in the testing phase.

We combine ProQA and ER to implement a stronger baseline ProQA+ER, in which samples from previous tasks are replayed for the ProQA model, and we also implement a variant of Diana by removing the memory buffer Diana w/o $\mathcal{M}$. We further report the performance for sequentially fine-tuning the LL model on all tasks (Finetune) and multi-task learning (Multitask). Note that the performance of Multitask is generally regarded as the upper bound of LL models when only seen tasks are considered.

All the above baselines are implemented following the same settings of our model, including using the same backbone PLM, prompt size, and memory size used for replay. Note that for the ProQA baseline, we follow its original setting to provide task IDs for testing samples when evaluating.

We conduct ablation studies on different components of Diana. Specifically, three types of variants are implemented:

1. Each of these four prompt types is ablated: w/o general prompt, w/o format prompt, w/o task prompt, w/o meta prompt.

2. Schemes to enhance task prompts are ablated: w/o Sched. Sampling removes the scheduled sampling scheme and only uses the ground truth task IDs in training; w/o G.T. Identity is similar to the above variant. Instead, it only uses predicted task IDs in training; w/o Neg. Samples only uses positive samples to train task prompt keys, i.e., the second term in Eq. 1 is removed; w/o ADB uses fixed decision boundaries instead of ADBs to detect unseen tasks.

3. Schemes to enhance meta prompts are ablated: w/o Sample Dive. does not enforce the diversity property of the meta prompt keys, i.e., the second term in Eq. 3 is removed; w/o Memory Dive. does not use samples from previous tasks to enhance the diversity property, i.e., the loss $\mathcal{L}^{\prime}_{m}$ (Eq. 4) is removed; w/o Loc. does not enforce the locality property of the meta prompt keys, i.e., the first term in Eq. 3 is removed; w/o Cluster does not cluster samples in $\mathcal{M}$, i.e., $\bm{c}_{k}$ in Eq. 4 is replaced with the query vector of each sample from $\mathcal{M}$.

Table 3 shows the performance of the above variants on QA tasks. It can be observed that Diana outperforms all the above variants. We can also see that: (1) “w/o Meta Prompt” lowers the LL performance by a large margin. This indicates that these fine-grained meta prompts are important in building lifelong models. (2) The scheduled sampling scheme helps to learn better task prompts and thus improves the LL performance. (3) ADB improves model performance on unseen tasks (i.e., $A_{N^{\prime}}$) by a large margin. (4) Enforcing the diversity property of meta prompt keys is important to obtain good key representations and facilitates the learning of each task.