Learning a Better Initialization for Soft Prompts via Meta-Learning

Pre-processing is necessary to adjust the input sample to text-to-text form. Specifically, there are two steps to pre-process the input samples.
Label Mapping We first map labels to our pre-defined concrete tokens. Taking a 5-class sentiment classification task as an example, the original labels are numbers from 0-4 which denotes the sentiment intensity. We map $0\to``terrible"$, $1\to``bad"$, $2\to``maybe"$, $3\to``good"$, $4\to``great"$ and then directly use these mapping results as label tokens. Since pre-trained language model is trained on natural languages, mapping labels to real words usually leads to decent performance Gu et al. (2022b).
Prompt Adding In the second step, we add prompts to the input sentence. we implement a hybrid prompt (Gu et al., 2022a) which combines soft prompts and hard prompts together to achieve the best performance. The prompt we applied is designed with the following template: a set of soft prompts embedding is prepended to the input sequence features, and then the manually desigend hard prompt “It was $\langle X\rangle$ . ” would be added at the end of the input sentence, where $\langle X\rangle$ denotes the mask token in the pre-trained language model.

Different from full-model tuning which tunes parameters of the entire pre-trained language model, prompt tuning only modifies the parameters of prompts prepended to the input sentence. Fine-tuning process of adapting pre-trained language model to a downstream classification task $\mathcal{T}^{down}$ could be represented by optimizing the following log-likelihood objective:

where $(x,y)$ is an input sample in $\mathcal{T}^{down}$, $x$ is an input sentence and $y$ is its label.
Prompt tuning freezes parameters of language model and only tunes soft prompts $\boldsymbol{P}$. Mathematically, hybrid prompt tuning could be represented by optimizing the following log-likelihood objective:

where $z$ is the concrete token mapped from label y, $H(x)$ denotes fitting input sentence $x$ to a hard prompt template and $[\boldsymbol{P};H(x)]$ denotes prepending soft prompts $\boldsymbol{P}$ to the beginning of input sequence.

Pre-trained prompt tuning Gu et al. (2022a) proposes to pre-train soft prompts $\boldsymbol{P}$ on rich-resource pretraining dataset $\mathcal{D}^{pre}$, in order to improve the performance on downstream tasks $\mathcal{T}^{down}$ under few-shot settings. Instead of directly training prompts on pre-training data $\mathcal{D}^{pre}$, we propose meta-learned prompt tuning to further enhance the performance of prompt tuning under few-shot settings. Pre-training data $\mathcal{D}^{pre}$ is first divided into different auxiliary meta tasks $\mathcal{T}^{meta}$ by unsupervised methods and then prompts are trained during and a model-agnostic meta-learning phase on these meta tasks.

There are mainly two methods to construct pre-training data for our algorithm. The first one is to directly choose a high resource dataset similar to the target task as pre-training data. The second one is to create pre-training data unsupervisedly. In our experiment setting, we focus on sentiment classification tasks. It’s worth mentioning that for sentiment classification task, we still couldn’t avoid relying on an existing dataset to create pseudo labels in our second method. The main difference between the first method and the second method is that the pre-training data in the method only covers one domain while the pre-training data created by the second method is an open domain. Besides, from the second method we could create unlimited data as long as we are able to collect enough large corpus from open web.

For the first method, we directly treat a 5-class sentiment classification large-scale dataset Yelp5 (Zhang et al., 2015) as pre-training data. For the second method, we adopt the method proposed by Gu et al. (2022a) to create pseudo labels for sentences from a large open-domain corpus. We train another model to annotate pseudo labels for the sentences in a large corpus.

After constructing pre-training data in section 4.1, we group the data into different clusters as auxiliary meta tasks used for meta-learning. We propose two unsupervised methods to separate the pre-training data into several meta tasks. By these unsupervised clustering methods, latent structure within pre-training data is revealed. Based on that structure, prompts could learn to incorporate some common internal features to the initialization through meta-learning. With such general information encoded in the initialization of prompts, the model can achieve great performance with limited training data from downstream tasks.

After we get a set of meta tasks $\mathcal{T}$ obtained via an unsupervised method, we utilize MAML to learn general features among these meta tasks. We first randomly initialize the parameter of soft prompts $\boldsymbol{P}$. For each meta task $\mathcal{T}_{i}$, m training samples are sampled from that task. Taking in m samples, the model output $f_{\boldsymbol{P}}$. Then we calculate the average loss $\mathcal{L}_{\mathcal{T}_{i}}(f_{\boldsymbol{P}})$ of these m samples and temporarily updates soft prompts with gradient descent.

After optimizing the prompts, we sampled another m samples and calculate the loss with the updated prompts. We add loss for $\mathcal{T}_{i}$ to total loss and repeat the same process for other meta tasks until we go over all the meta tasks. Finally, we update the prompts by minimizing the final total loss.

This is a complete process of one-step updates for prompts. We keep optimizing the prompts until the validation accuracy of meta tasks stop growing. The whole algorithm of Meta-learned prompt tuning is shown below.

As shown in the Table 1, we mainly compare the performance of full-model tuning (FT), pre-trained prompt tuning (PPT), meta-learned prompt tuning (MetaPT) on different sentiment classification tasks. We also include results of MetaPT_(Y) in the table, which is a variation of MetaPT. Instead of being trained on the pseudo data, MetaPT_(Y) is directly trained on Yelp5. According to Gu et al. (2022a), the performance of plain prompt tuning lags far behind FT and PPT, we don’t include it in our main table. We have three observations from results in Table 1.

First, MetaPT consistently achieves better results than PPT over all seven tasks. MetaPT outperforms PPT by prevent overfitting on pre-training datasete. Benefitting from MAML algorithm, MetaPT only transfer general features to the initialization of prompts and throw out redudant information. MetaPT also outperforms FT for most of the tasks. Because we utilize the extra pre-training data to train the soft prompt without destroying the inner structure of language model. However, full-model tuning achieves the best score on the Sentihood dataset because the examples in this dataset are simple and easy to learn for full-model tuning. From the example shown in Appendix B, the input of Sentihood is shorter than other datasets, and the words appeared in this sentihood is more commonly used in daily life. Also, its labels only has 2 classes good and bad. Due to above three reasons, we believe that training samples of Sentihood are close to what language model learned through pre-training and would cause less destruction to inner structure of language model through full-model tuning. Therefore, full-model tuning can quickly learn the Sentihood task with a few data points.

Second, MetaPT_(Y) also outperforms PPT and FT on most tasks. And MetaPT_(Y) achieves even better results than MetaPT on some tasks. We look into the results on SST-5 and we find that the recall score of MetaPT is much lower than MetaPT_y on the label terrible (0.22 vs. 0.61). We believe this is caused by some pseudo data samples with poor quality. When creating pseudo labels, the classifier trained by Yelp5 tends to annotate text with the label terrible with high probability, even though the text is irrelevant to terrible from the human perspective. For example, “Both parties agreed to have 1,410 troops located in juba, lueth said, and there are already 1,370 opposition soldiers in the capital.” is considered as terrible with high confidence by classifier while we don’t discover any sentiment here. This tendency results in the low quality of pseudo data with the label terrible which confuses the MetaPT during MAML. Even though Yelp5 covers only restaurant domain, MetaPT_(Y) still shows great performance when adapting to sentiment tasks in other domains. This suggests that our MetaPT is able to learn the general features, which enable the prompt be easily generalized to other domains.

Finally, MetaPT is much more stable than both PPT and FT. Most methods would suffer from high variance under few-shot setting due to their sensitivity to different training samples. The performance of FT varies tremendously when we select different few-shot samples for FT. However, MetaPT shows remarkable stability facing different training samples. The standard deviation of MetaPT is much lower than both FT and PPT across all downstream tasks. This phenomenon indicates that MetaPT is more robust under few shot settings.

We show the tendency of how performance of FT, PPT, MetaPT varies when the number of training samples increases on SST-5. As shown in Figure 2, when the number of training samples grows from 10 to 2,560, MetaPT is consistently better than PPT and FT, while PPT also has a small advantage over FT. It should be noted that, the full-model tuning method will eventually catch up with other two prompt tuning methods as the number of training samples keep increasing. All three methods will converge to similar performance when training data is sufficient.

In this section, we discuss how much pre-training data we need to obtain a good result, how the method of clustering affects the performance and how the number of clusters influences results. All the experiments are evaluated on SST-5.
Scale of pre-training data We want to minimize the pre-training data we used without undermining the performance of our model. We set the number of pre-train samples from 1,000 to 1,000,000. And then we implement Kmeans to cluster them into 10 classes for meta-learning. As the Figure 3(a) shows, the accuracy grows rapidly when the number pre-training data increases from 1,000 to 10,000. After 10,000 training samples, the performance does not change much as the number of training samples increases. This result suggests that more pre-training data samples are not necessarily along with better performance in our method. When the size of pre-train data reaches the level of 10,000, it is enough for our model to get acceptable performance.
Methods of clustering We design four different methods of clustering to get meta tasks. They are K-means clustering, LDA clustering, random clustering, and label clustering. For K-means clustering, we apply a Sentence-BERT-base model to extract sentence features from pre-training samples and then implement K-means to cluster these samples into different groups according to their sentence features. For LDA clustering, we group training samples according to the hidden themes extracted by the LDA topic model. For the random method, we randomly split the total pre-train dataset into different groups. For the label method, we cluster the data samples with the same label into the same group. When the clustering number is large than the number of labels, we just randomly split samples with the same label into more groups.We fixed the cluster number as 10 in this experiment. From the result shown in Table 2 we notice that K-means clustering is most effective and LDA is next to it. When we cluster randomly or according to their labels, the performance of MetaPT degrades to the same level as PPT.

Number of clusters We examine the cluster number from 3-30 and compare performance. We fix the clustering method as K-means. As shown in the Figure 3(b), the accuracy grows rapidly at first as cluster number increases but later it converges. Considering both effectiveness and efficiency, MetaPT is able to achieve promising results when k=10. We also visualize the result of K-means clustering when cluster number equals to 3, 6, 10 in Figure 4. From the TSNE of our K-means clusters, we could see that data is well grouped into different clusters according to their sentence embeddings. After we reduce the sentence features of Different samples to two-dimensionality, different samples in the same clusters are close to each other and are distinguishable from samples in other clusters, which demonstrates that meta tasks derived from K-means indeed contain useful common latent features.