Iteratively Prompt Pre-trained Language Models for Chain of Thought

with a set of training data $\{q_{i},C_{q_{i}}\}_{i=1}^{N}$.

Prompter Instantiation. While there are many plausible design choices for the Prompter $f_{W}$, here we instantiate it with a transformer-based language model (shown in Figure 2). The prompts are designed to be contextualizations (by the Prompter) of a set of special tokens w.r.t. the current step input context, linearly projected into the PLM’s embedding space by a trainable matrix (omitted in the figure due to space limit). In this work, we adopt an Encoder-Decoder PLM and use prefix-prompts in the implementation; hence we have prompts that are prepended to both the PLM’s encoder inputs and decoder inputs. Note that our design could be easily adapted to other types of PLMs (e.g., encoder-only/decoder-only models) and different prompt positionings (e.g., infix, postfix).

Comparison with Prompt/Prefix-Tuning. Both Prompt-Tuning Lester et al. (2021) and Prefix-Tuning Li and Liang (2021) model the prompt $T$ as a context-agnostic parameter. In Prompt-Tuning, $T$ has the same identity as in our approach which is a set of virtual input tokens (Encoder Prompts & Decoder Prompts in Figure 2). In Prefix-Tuning, $T$ is modeled to be the set of activations (keys & values in the transformer attention blocks) of the virtual prompt tokens across all PLM layers. Let $D$ be the embedding dimension of the PLM, $h$ be the number of layers in the PLM, $d$ be the embedding dimension of the Prompter ($d\leq D$), and $l$ be the length of the prompt tokens (both encoder & decoder prompts). Then the number of trainable parameters is $\Theta(d\cdot(D+l))$ for our proposed method, $\Theta(l\cdot D)$ for Prompt-Tuning and $\Theta(l\cdot h\cdot D)$ for Prefix-Tuning. It can thus be seen that our proposed method scales comparatively with Prompt-Tuning, slower than Prefix-Tuning, and overall maintains a manageable amount of trained parameters as the PLM scales up (which increases $D$ and $h$).

Continuous v.s. Discrete Prompts. While modeling $T$ as discrete tokens in the PLM’s vocabulary could increase the readability of the prompts, a discrete space is much less expressive than its continuous counterpart, and optimization over a discrete space could be highly inefficient. Also, despite being inside the vocabulary, the searched discrete prompts could still have low interpretability as seen by the given examples in Shin et al. (2020). Hence, we follow prior work Zhong et al. (2021); Li and Liang (2021); Lester et al. (2021); Qin and Eisner (2021) and model the prompts to be continuous virtual tokens instead of discrete tokens.

2WikiMultiHopQA Ho et al. (2020). 2Wiki is a recent large scale multi-hop QA dataset, which contains in total over 192k (167k train, 12.5k development, and 12.5k test) samples constructed jointly from Wikipedia and Wikidata. Since the test set is private, we randomly split the original development set into our development & test set (6k samples each). The dataset format largely follows HotpotQA Yang et al. (2018), but includes more diverse reasoning types of questions and detailed annotations of evidence paths for each question. Here, an evidence path is an ordered list of (subject entity, relation, object entity) knowledge base triplets. We use the question as the query $q$, and use a simple template to convert each triplet in the evidence path into a natural language statement, forming $C_{q}$. Due to the large training set size and limited computing budget, we randomly sample $10\%$ of the training data to form our final training set, which has the side benefit of largely reducing the test/train overlap (more details in §4.2).

R4C Inoue et al. (2020). R4C is another recent multi-hop QA dataset containing annotated evidence paths. The dataset contains 4.6k examples (2.4k train, 2.2k development) constructed on top of HotpotQA, where the authors used crowd-sourcing efforts to collect the evidence paths in the form of simple subject-verb-object natural language sentences. Again, we randomly split the development set (there’s no test set given) into our development and test set (1.1k samples each). We use the question as our query $q$ and use the annotated evidence sentences as $C_{q}$.

LoT Talmor et al. (2020b). The dataset involves reasoning over a set of taxonomic relations, constructed from ConceptNet and WordNet. Each example consists of a hypothesis (e.g., “A whale has a belly button”) which we treat as query $q$, and a set of simple facts including hypernym rules (e.g., “A whale is a mammal”, “A whale is a vertebrate”) and properties (e.g., “A mammal has a belly button”, “A vertebrate has a tail”). By reasoning over the facts and selecting the correct chain of hypernym rule & property (“A whale is a mammal”, “A mammal has a belly button”), one could verify or deny the given hypothesis. One subtle issue of directly using the gold hypernym rule and property as $C_{q}$ is, during the first step, it would be difficult to directly identify the correct object entity without looking ahead on the properties in the second step. Therefore, for the first step, we concatenate all the hypernymic objects appearing in the dataset w.r.t. to the same subject to form $c_{1}$. We drop samples from the original training set where the relevant facts are not (or only partially) provided, and obtain 9.4k/1.2k/1.2k samples for training/development/testing.

Intrinsic Evaluation. Here, we directly measure the quality of recalled knowledge. While there are standard metrics for evaluating text generation such as BLEU and ROUGE, these metrics generally fail to capture the entity-centric nature of the recalled knowledge we wish to examine (more details are included in Appendix A.3). Therefore, we propose a set of measures that are better suited for the tasks in our experiments. For 2Wiki and R4C, we evaluate the ratio where the recalled knowledge contains the answer entity (Ans.$\bm{R}$); we also compute the ratio among only those samples where the answer entity does not appear in the query (Ans.$\bm{\hat{R}}$). For 2Wiki and LoT, we also evaluate the evidence coverage of recalled contexts by computing the average ratio of gold evidence appearing in the recalled context (Evi.$\bm{R}$) and the ratio of samples where all gold evidence are recalled (Evi.$\bm{R^{*}}$) as a more strict measure. For 2Wiki, we use the entities from the annotated KB triples as evidence. For LoT, we consider the hypernym rule/property as evidence, where in the 1st step, we deem the hypernym rule as correct if the gold object is recalled, and use exact match for the recalled property in the 2nd step.

Extrinsic Evaluation. We also conduct extrinsic evaluation by measuring how much the recalled knowledge help find the response to the query. Similar to reading comprehension, we concatenate all recalled knowledge as the contexts, and use a reader which tries to infer the answer given the query and contexts. For 2Wiki and R4C, we first pre-train the reader using the ground truth contexts, and then fine-tune it on the recalled contexts³³3We found in our preliminary experiments that this approach gives the best results across different methods.; for LoT, we use a rule-based reader directly⁴⁴4LoT is constructed using templates, and therefore a rule-based reader can perfectly solve the inference task (100% accuracy when ground truth contexts are given, see Table 2).. We report Exact Match (EM) and Answer F1 scores for 2Wiki & R4C, and EM score for LoT where the answer is restricted to yes/no.

Architectures & hyperparameters. We use BART-large Lewis et al. (2020) for our PLM and RoBERTa-base Liu et al. (2019) for our prompter, which is several times smaller than the PLM⁵⁵5While our prompter is also initialized using a Pre-trained Language Model, we’ll use the term “PLM” to refer only to the larger & more knowledgeable one.. We also include some results & discussion for different prompter scales in Appendix A.7. We use another BART-large for the reader in extrinsic evaluation⁶⁶6For the reader, we intentionally choose the same architecture with the PLM for a fair comparison with PLM-QA..

Knowledge Enhancement for PLM. Since our focus is on how to make PLMs better at recalling relevant knowledge for multi-step inference, we need to make sure the PLM actually memorizes all the relevant knowledge in the first place, so that the results can be attributed solely to the effectiveness of knowledge recall. Hence, we conduct knowledge enhancement for the PLM, where we additionally pre-train the PLM to recover separately masked elements in the triplets which form the knowledge statements, a strategy similar to salient span masking Roberts et al. (2020); Guu et al. (2020). More details could be found in Appendix A.2. Note the same PLM after knowledge enhancement is used across different compared methods.

Effectiveness of Iterative Scheme & Context-Aware Prompter. Across different datasets, it can be seen that most compared methods benefit from the iterative setting (Iter) over the non-iterative setting. Moreover, our proposed iterative Context-Aware Prompter (iCAP) further outperforms Prompt/Prefix Tuning by notable gains across different datasets and metrics, approaching the performance of PLM fine-tuning (PLM-FT); in particular, on the 2Wiki dataset which has the largest scale and diversity of reasoning types, iCAP achieves more than 15% and 10% absolute gains in F1 over Prompt-Tuning & Prefix-Tuning respectively. Overall, the results clearly show the effectiveness of the iterative scheme and our proposed context-aware prompter design. However, we note that even the best results (prompting based or fine-tuning based) still far lag behind Oracle-RD which uses ground truth contexts as input, which suggests a large room for improvements with better methods for knowledge elicitation from PLMs. Some failure cases of iCAP are included in Appendix A.6.

Helpfulness of Knowledge Recall for Multi-step Inference. The result obtained by fine-tuning the PLM on (query, answer) directly without knowledge recall (PLM-QA) is outperformed by almost all other compared methods, verifying the previous findings that PLMs face difficulties in using their stored knowledge to perform multi-step inference tasks. The large gain obtained from methods based on knowledge recall shows the helpfulness of deriving a “chain of thought” (especially iteratively) from PLMs for multi-step inference.

Test-Train Overlap. For each development & test sample, we compute the ratio of knowledge statements in $C_{q}$ that also appear in the training set, meaning that during certain steps for some training samples, the model has “seen” the exact same piece of knowledge. Note that this is a rather strict measure: even if all the knowledge pieces in $C_{q}$ are seen during training, they may come from completely different samples & steps and hence organized in different ways. We summarize the overlapping ratios of development & test set samples in Table 7 in Appendix A.5. It can be seen that the down-sampling has the side benefit of greatly reducing the test-train overlap; in particular, the percentage of examples where all knowledge statements are seen during training is reduced from almost 30% to less than 2%, and more importantly, over 70% of the samples have no overlap. This suggests a rather low risk for the existence of strong spurious regularities in our setup.

Random Control Experiments. Examining the data-level statistics is helpful, but still not sufficient in terms of revealing the spurious regularities that different methods may capture. Hence, we follow Zhong et al. (2021) to conduct two random control experiments. In the Random Model experiment, we re-initialize all parameters of the PLM to clean out its internal knowledge, and proceed with the same training procedure as earlier. In this way, any positive signal obtained could only be attributed to dataset regularities captured by the method. In the Random Embedding experiment, we re-initialize only the input embeddings of the PLM, a setting analogous to the control task introduced in Hewitt and Liang (2019) (more discussions can be found in Zhong et al. (2021)). Here we only proceed with the iterative setting and conduct intrinsic evaluation, where the results are summarized in Table 3. It can be seen that 1) PLM fine-tuning captures significantly more regularities in the dataset than prompting-based methods; 2) While our proposed method captures a bit more regularities than Prompt/Prefix Tuning, they still remain at a very small level. Overall, our random control experiments show that the exploitation of spurious dataset patterns by the evaluated prompting methods is rather mild, and that by PLM fine-tuning could potentially be larger.

Prompter Attention Visualization. To see whether our proposed iCAP behaves in the way we expect, one direct approach is to examine the inner workings of the prompter. Towards this end, we visualize the attentions during the prompter forward pass at different steps. We randomly choose examples in the development/test set, and use BertViz Vig (2019) to visualize the attentions within the forward pass of the prompter after the following processing steps: 1) we aggregate the attention weights of different attention heads within the same transformer layer; 2) to better view the prompt tokens as one single unit, we average the attentions across different prompt tokens to form one “master” prompt token; 3) we drop all special tokens (BOS, EOS) for cleaner visualization. One example (the same example which we use in Figure 1) is in Figure 3, and we include more examples in Appendix A.8. As briefly illustrated earlier in §1, during the 1st step towards solving this query, the prompter should focus on the part concerning “father” of “Gwilym Lloyd George”; during the 2nd step, the prompter should integrate the answer “David Lloyd George” from the 1st step evidence and the “place of birth” part in the query to synthesize the prompt. We can see that the attention distributions at different steps accord well with our expectations. However, we note that attention visualization is only a qualitative approach; more systematic ways for examining the inner working behaviors of transformers remain an open challenge.