Can Language Models Take A Hint? Prompting for Controllable Contextualized Commonsense Inference

Recently, there has been a shift in paradigm in Natural Language Processing from pre-training and fine-tuning a model, to pre-training, prompting, and predicting Liu et al. (2021). One primary reason for this shift is the creation of ever-larger language models, which have become computationally expensive to fine-tune. Prompting can be described as converting a pre-trained language model input sequence into another sequence that resembles what the language model has seen during pre-training. Overall, most prompting research is focused on formulating the task as a cloze (fill-in-the-blanks) task. However, we consider the task of language generation, an open-ended formulation.

Recall that prefix prompting modifies the input to a language model, by adding either a hard prompt (additional words to the input sequence)Shin et al. (2020) or a soft prompt (i.e., adding trainable vectors that represent, but are not equivalent to, additional words) Li and Liang (2021); Lester et al. (2021); Liu et al. (2021).

Unlike classic prefix prompting, hinting uses both hard and soft prompts. The soft prompts are in the form of symbols that represent the different parts of the assertion (i.e., subject, relation type, and object), and the hard prompts are in the form of the actual parts of the assertion that are selected to be appended as part of the hint. Our work is similar to KnowPrompt Chen et al. (2021a), except that they use a masked language model and soft prompts for relationship extraction. AutoPrompt Shin et al. (2020) is also similar, but finds a set of "trigger" words that give the best performance on a cloze-related task, whereas we provide specific structured input for the model to guide text generation. We additionally note that although there are prompt-based relation extraction models Chen et al. (2021b), we are performing a different task which is contextual commonsense inference. Another recent contribution, P-Tuning Liu et al. (2023), shares similarities with our approach by combining trainable continuous prompt embeddings with discrete prompts. Both hinting and P-tuning share the overarching objective of enhancing prompt learnability. PTR, or prompt-tuning with rules Han et al. (2021), shares similarities with hinting as it involves encoding prior knowledge about tasks and classes. Similarly to PTR, hinting also introduces additional information to provide the model with contextual understanding of the relationships between words.

Controllable generation can be described as ways to control a language model’s text generation given some kind of guidance. One work that tries to implement controllable generation is CTRL Keskar et al. (2019). The authors supply control signals during pre-training of a general language model. A body of work in controllable generation has focused on how it can be used for summarization. Representative work that uses techniques similar to ours is GSum Dou et al. (2021).

In contrast to GSum, our method is model independent, allows for the source document to interact with the guidance signal, and contains soft prompts in the form of trainable embeddings that represent the parts of a tuple. The GSum system gives interesting insight into the fact that highlighted sentences, and the provision of triples, does in fact help with the factual correctness of abstractive summarization. We make the distinction that hinting falls more under prompting for the reason that we utilize additionally the trainable soft embeddings rather than purely additional hard tokens and that our task of contextual commonsense generation is not explored in the controllable generation works, whose main focus is on controlling unstructured text generation. Some works that are in this area are also Peng et al. (2018) who utilize what they call "control factors" as keywords or phrases that are supplied by a human-in-the-loop to guide a conversation. Another work, Diffusion-LM Li et al. (2022), developed a language model that utilizes a sequence of Gaussian noise vectors that gradually transform into words, creating an organized structure to guide what the model generates. More similar to our work, but tailored for the task of interactive story generation and without trainable soft-embeddings, is the work by Brahman et al. (2020) which uses automatically extracted keywords to generate a story. Future work we could possibly utilize the automatic keyword extraction to supply parts of a hint, rather than our approach of complete parts of an assertion, and expand this to utilize synonyms of keywords. Lastly, there is the work by See et al. (2019) which looks at controllable text generation for the purpose of conversation and utilizes an embedding give quantitative control signals as part of conditional training.

Commonsense inference is the task of generating a commonsense assertion. Discourse-aware/contextual commonsense inference is the task of, given a certain narrative or discourse, inferring commonsense assertions that are coherent within the narrativeGabriel et al. (2021). This task is particularly hard because commonsense knowledge may not be explicitly stated in text Liu and Singh (2004) and the model needs to keep track of entities and their states either explicitly or implicitly. Research into the knowledge that pre-trained language models learn has yielded good results in that they do contain various types of factual knowledge, as well as some commonsense knowledgeDa and Kasai (2019); Petroni et al. (2019); Davison et al. (2019). The amount of commonsense knowledge in these models can be improved by supplementing sparsely covered subject areas with structured knowledge sources such as ConceptNet Speer et al. (2017); Davison et al. (2019).

Knowing that these pre-trained language models may contain some commonsense information has led to the development of knowledge models such as COMETBosselut et al. (2019). This line of research has been extended from the sentence-by-sentence level in COMET, to the paragraph-level in ParaCOMET Gabriel et al. (2021). Contemporaneously, GLUCOSE Mostafazadeh et al. (2020) builds a dataset of commonsense assertions that are contextualized to a set of stories, and generalized. More recently, the idea of knowledge models Da et al. (2021) or models that can be leveraged to generate commonsense assertions has been gaining track. One recent approach has been kogito Ismayilzada and Bosselut (2022). Kogito is a toolkit for commonsense inference, which permits training and access of similar to COMET, along with providing tools for selecting a subject and a relation. Kogito utilizes the same formulation as the ParaCOMET work, in which a subject and a relation are provided. However, kogito does not tackle the more complicated general commonsense inference as in GLUCOSE. With our work, we could provide kogito with a framework to be able to train models that can perform this type of controllable, generalized inference and improve the overall training.

We now detail the task of Contextual Commonsense Inference. We are given a story $S$ composed of $n$ sentences, $S=\{S_{1},S_{2},…,S_{n}\}$ , a target sentence from that story, $S_{t}$, where $S_{t}\in S$, and a dimension/relation type $R$. Given all this, we want to generate a tuple in the form of $({subject},{R},{object})$ that represents an assertion, present or implied, in $S_{t}$ given the context $S$, and the relation type $R$.

We run tests with two variations of this task, one is the ParaCOMET variation and the other the GLUCOSE variation. In the ParaCOMET experiments, we represent $S_{t}$ with a special token. Additionally, we only generate the $object$ of the tuple. In our GLUCOSE experiments, we represent $S_{t}$ by marking it with $*$ on the left and right of the actual target sentence. Additionally, we generate at most two subject, R, object tuples: one that is the context-specific tuple, and the other is the general tuple, separated by two asterisks (**). An example of how the inputs look for both datasets can be seen earlier in the Introduction.

The mechanism we present in this work, called hinting, is a kind of mixed/hybrid prompting for generative language models. Prompting is essentially supplying additional text (i.e. prompts) to a language model to aid/guide it in a specific task. In our case, we opt to give a “hint", as to what the assertion that we want to predict contains, at the end of our input text. We chose placing the hint at the end of the input for simplicity in dataset processing, but it can be placed anywhere and we leave it as future work to explore the effects of placing hints possibly next to the target sentence or at the beginning of the input. Hinting can be seen as a hybrid of prompting the generative model with hard prompts composed of parts of what should be predicted along with soft prompts of symbols that represent those parts. These symbols are for the subject, relation (R), and object respectively. These soft prompts utilize untrained embeddings for the task. We structure hinting this way such that, after training, whenever a hint is given, the model can be guided to generate assertions about the hint’s content based on the target sentence and context.

To balance the model’s reliance on the context, its knowledge, and the hint, we determine whether to supply the hint by sampling a binomial distribution ($p=0.5$). Thus, we can control the frequency of when to supply a hint. Additionally, the content of the hint is determined by random sampling of permutations of components, up to a maximum of all but one component. Since our task is to predict the tuple, we do not want to make the model overly reliant on hints for the answer.

A simple example of hinting is the following:

Story: The hockey game was tied up. The red team had the puck. They sprinted down the ice. They cracked a shot on goal! They scored a final goal!

Target sentence: They scored a final goal!

Target assertion: (subject: the red team, relation: are capable of, object: winning the game.)

A hint can be any permutation of the target assertion, except the complete assertion, along with some symbol that indicates which part it is:

Possible Hints:
(<$|$subj$|$> the red team),
(<$|$obj$|$> winning the game),
(<$|$rel$|$> capable of),
(<$|$subj$|$> the red team, <$|$rel$|$> capable of),
(<$|$subj$|$> the red team, <$|$obj$|$> winning the game),
(<$|$rel$|$> capable of, <$|$obj$|$> winning the game)

Sampling randomly from one of those permutations, a hint for the given story, target sentence and target assertion, yields the following:

Hint: (<$|$subj$|$> the red team, <$|$rel$|$> capable of)

Putting everything together, the input for the model would be:

Story with Hint: The hockey game was tied up. The red team had the puck. They sprinted down the ice. They cracked a shot on goal! They scored a final goal! (<$|$subj$|$> the red team, <$|$rel$|$> capable of).

We note that this is a general version of how the hinting mechanism works. The dataset specific hints that we utilize are described in the Appendix.

To explore other ways of hinting, we devised another technique by swapping parts of the hint with synonyms and antonyms. We used the synsets and antsets from the WordNet Fellbaum (2010) knowledge base to find synonyms and antonyms respectively. In the case that there is no viable synonym or antonym we keep the original word. To introduce synonyms and antonyms, the process is as follows. When a hint is generated, we copy it and we decide whether to supply it with synonyms and/or antonyms by swapping the words in the hint prompt with their synonyms or antonyms by sampling from a binomial distribution (we explored various values for $p$, with 0.5 being the most effective) to control the frequency of hints with synonyms/antonyms. The reasoning for this is that if we only supply synonyms/antonyms, we lose the control that hinting provides, so we introduce synonyms and antonyms at a certain rate to have the model still be controllable, while seeing “new” information through the synonyms/antonyms. After the swapping process, we insert a soft prompt, <$|$syn$|$> or <$|$ant$|$>, at the end of the substituted prompt to signal that the model is using a synonym or an antonym respectively. We give an example of how these hints look in the Appendix.

For our first set of experiments, we utilize the ParaCOMET Gabriel et al. (2021) dataset and the framework with the same GPT-2 model as ParaCOMET, along with a T5 Raffel et al. (2020) model to observe the effects of hinting in a sequence-to-sequence formulation of the dataset. We use the off-the-shelf (Huggingface Wolf et al. (2019)) pre-trained “base" version of these models for efficiency. For our second set of experiments with the GLUCOSE dataset, we also used the T5 model, as was done in GLUCOSE.

The aggregated results for this set of experiments can be found in Table 2. We can see here that on average, hinting does tend to improve the score even if slightly. It seems that providing a hint is beneficial and not detrimental for contextual commonsense inference. Given the way that this task is framed, a possibility that could explain the relative similarity of the performances, is that hinting in this formulation only adds the object of the triple as additional possible data that the model may see during training; the subject and the relation can be repeated with hinting. We note that the performance of the T5 model was less, and we believe that it may be lack of hyperparameter tuning, as it was seen that the model was sensitive to the learning rate and had to use a higher than usual learning rate. In these tests, we also note that hinting with synonyms tends to consistently improve the performance even further than just plain hinting, indicating that the model benefits from making associations of related concepts. We see that hinting with antonyms also tends to be beneficial, but the benefit is not as consistent as with synonyms. Interestingly, we see that combining both synonym and antonym training does not bring the best of both worlds, but more closely an average between the performance of hinting with either.

The aggregated results for this set of experiments can be found in Table 3. Once more we notice that hinting (with and without synonyms and/or antonyms) does tend to improve the performance of the contextual commonsense inference task. This suggests that hinting is indeed beneficial for the task of contextualized commonsense inference, especially when faced with the harder task of generating both a general and context dependent assertion. We believe that this improvement is because hinting gives the model the clues it may need to decide on what to focus or attend to, to generate useful inferences.

In Table 4 we see the results of our synthetic controllability test. We see that the model without hinting tends to predict about relevant things (indicated by the BLEU scores above 50), however when hints are injected, the model always predicts about what the hint was suggesting (indicated by the nearly perfect scores). We also see that supplying synonyms and antonyms does not decrease the controllability of the models.

The results for a tiny Mechanical Turk study for human evaluation of model inferences can be seen in Table 5. Overall we can see here that hinted systems are judged as slightly higher in plausibility. We also see upon looking some of the inferences that the hinted model tends to be more general and provide shorter responses than the non-hinted model (e.g., hinted inference: “satisfied" vs. non-hinted inference: “happy and satisfied").

From the results of our experiments, we can see that hinting tends to increase the performance of contextualized commonsense inference at least with regards to automated metrics and does not significantly degrade or improve human judgements. This brings the question of: Why hint at all? The primary reason is for controllability in the generation. By supplying these hints, we are teaching the model pay attention and generate inferences about a certain subject, relation, or object. This in turn, after training, can be leveraged by a user or downstream application to guide the model to generate assertions from parts that are manually supplied. Although this is not very clear within the ParaCOMET formulation, it becomes clearer in the GLUCOSE formulation of the problem. We give an illustrative example of the usefulness of hinting in Table 1. We can see that by giving a model the hint, the model could be capable of inferring about information that may not be present in the story. We note that this behavior is useful in downstream tasks such as story understanding and contextual knowledge graph generation in which we may need a model to have a specific subject or object . Lastly, hinting was designed to be simple to implement, and is model independent.

This work was a proof of concept for this technique. We acknowledge there is a large body of research on the area of prompting. The way the hinting mechanism was designed however, leaves much space to explore alternate mechanisms such as AutoPromptShin et al. (2020), including additional soft prompts such as those in Li and Liang (2021), or even replacing the contents of the hint with synonyms or related words. Because of the naivety of the approach, we do not think it is an optimal approach, and there is a large body of research that points to manual templating of prompts being less effective than learned prompts Liu et al. (2021). However, from our tests, our approach does not degrade performance, and only improves it.