SAPPHIRE: Approaches for Enhanced Concept-to-Text Generation

The CommonGen dataset is split into train, dev, and test splits, covering a total of 35,141 concept sets and 79,051 sentences. The concept sets range from 3 keywords to 5 keywords long. As the original test set is hidden, we split the provided dev set into a new dev and test split for the majority of our experiments while keeping the training split untouched. Note that we also evaluate our SAPPHIRE models on the original test set with help from the CommonGen authors (see §6.1). We will henceforth refer to these new splits as train_CG, dev_CG, and test_CG, and the original dev and test splits as dev_O and test_O. The statistics of our new splits compared to the originals can be found in Table 1. We attempt to keep the relative sizes of the new dev and test splits and the distribution of concept set sizes within each split similar to the originals.

We perform experiments using pretrained language generators, specifically BART and T5 (both base and large versions). BART Lewis et al. (2020) is a Transformer-based seq2seq model trained as a denoising autoencoder to reconstruct original text from noised text. T5 Raffel et al. (2020) is another seq2seq Transformer with strong multitask pretraining. We use their HuggingFace codebases.

We train two seeded instances of each model on train_CG, evaluating their performance on dev_O, and comparing our numbers to those reported in Lin et al. (2020) to benchmark our implementations. These essentially serve as the four baseline models for our ensuing experiments. We follow the hyperparameters from Lin et al. (2020), choose the epoch reaching highest ROUGE-2 on the dev split, and use beam search for decoding.²²2See Appendix A for further details. From Table 2, we see that our re-implemented models match or exceed the original reported results on most metrics across different models.

For our experiments, we use a gamut of automatic evaluation metrics. These include those used by Lin et al. (2020), such as BLEU Papineni et al. (2002), CIDEr Vedantam et al. (2015), SPICE Anderson et al. (2016), and Coverage (Cov). Barring Cov, these metrics measure the similarity between generated text and human references. Cov measures the average % of input concepts covered by the generated text. We also introduce BERTScore Zhang et al. (2020), which measures token-by-token BERT Devlin et al. (2019) embeddings similarity. It also measures the similarity between the generated text and human references, but on a more semantic (rather than surface token) level. When reporting BERTScore, we multiply by 100. For all metrics, higher corresponds to better performance.

We begin by conducting an analysis of the four baselines implemented and discussed in §2.2, which we refer to henceforth as BART-base-BL, BART-large-BL, T5-base-BL, and T5-large-BL. One aspect we were interested in is whether the number of input concepts affects the quality of generated text. We conduct a comprehensive correlation study of the performance of the four baselines on dev_O w.r.t. the number of input concepts.

As seen from Table 3, the majority of the metrics are positively correlated with concept set size across the models. ROUGE-L, CIDEr, and SPICE have small correlations that are mainly statistically insignificant, demonstrating that they are likely uncorrelated with concept set size. Coverage is strongly negatively correlated, showing that there is a higher probability of concepts missing from the generated text as concept set size increases.

There are two major takeaways from this. Firstly, increased concept set size results in greater overall performance. Secondly, models have difficulty with coverage given increased concept set size. This motivates our first set of improvements, which involves augmenting the concept sets with additional words in hopes of 1) increasing performance of the models and 2) improving their coverage, as we hope that training with more input concepts will help models learn to better cover them in the generated text. This is discussed more in §4.1.

We conduct a qualitative analysis of the baseline model outputs. We observe that several outputs are more like phrases than full coherent sentences, e.g. “body of water on a raft”. Some generated texts are also missing important words, e.g. “A listening music and dancing in a dark room” is clearly missing a noun before listening. A large portion of generated texts are quite generic and bland, e.g. “Someone sits and listens to someone talk”, while more detailed and specific statements are present in the human references. This can be seen as an instance of the noted “dull response” problem faced by generation models Du and Black (2019); Li et al. (2016), where they prefer safe, short, and frequent responses independent of the input.

Another issue is the way sentences are pieced together. Certain phrases in the outputs are either joined in the wrong order or with incorrect connectors, leading to sentences that appear to lack commonsense. For example, “body of water on a raft” is illogical, and the phrases “body of water” and “a raft” are pieced together incorrectly. Example corrections include “body of water carrying a raft” and “a raft on a body of water”. The first changes the adverb on joining them to the verb carrying, and the second pieces them together in the opposite order. A similar issue occurs with the {horse, carriage, draw} example in Table 4.

Some major takeaways are that many generations are: 1) phrases rather than full sentences and 2) poorly pieced together and lack fluency and logic compared to human references. This motivates our second set of improvements, which involves recombining extracted phrases from baseline generations into hopefully more fluent and logical sentences. This is discussed more in §4.2.

The first set of improvements is concept set augmentation, which involves augmenting the input concept sets. We try augmentation using up to 1 to 5 additional words, and train-time augmentation both with and without test-time augmentation. We observed that test-time augmentation resulted in inconsistent results that were not as effective, and stick with train-time only augmentation. During training, rather than feeding in the original concept sets as inputs, we instead feed in these augmented concept sets which consist of more words. The expected outputs are the same human references. During test-time, we simply feed in the original concept sets (without augmentation) as inputs.

The second set of improvements is phrase recombination, which involves breaking down sentences into phrases and reconstructing them into new sentences which are hopefully more logical and coherent. For training, we use YAKE Campos et al. (2018) to break down the train_CG human references into phrases of up to 2, 3, and 5 n-grams long, and ensure extracted phrases have as little overlap as possible. During validation and testing, since we assume no access to ground-truth human references, we instead use YAKE to extract keyphrases from our baseline model generations.

We ignore extracted 1-grams as this approach focuses on phrases. We find words from the original concept set which are not covered by our extracted keyphrases and include them to ensure that coverage is maintained. Essentially, we form a new concept set which can also consist of phrases. Some examples can be found in Table 6.

For training Kw-aug, Att-aug, and P2T models, we follow baseline hyperparameters, barring learning rate (LR) which is tuned per-method. We train two seeds per model. See Appendix A for more.

For each model, we choose the epoch corresponding to highest ROUGE-2 on the dev split, and use beam search for decoding. The dev and test splits are different. For Kw-aug and Att-aug models, the splits are simply dev_CG and test_CG (or test_O), as we do not perform test-time augmentation. For P2T, the splits are dev_CG and test_CG (or test_O) but with the input sets replaced with new ones that include keyphrases extracted from the corresponding baseline model’s outputs.

The number of words to augment for Kw-aug and Att-aug (from 1 to 5) and maximum n-gram length of extracted keyphrases for P2T (2, 3, or 5) are hyperparameters. While we train separate versions of each model corresponding to different values of these, the final chosen model per method and model combination (such as BART-base-KW) is the one corresponding to the hyperparameter value that performs best on the dev split when averaged over both seeds. For MI, which involves no training, we select the variation (MI on the original concept set or new input sets with keyphrases up to 2, 3, or 5 n-grams) per model that performs best on the dev split, and only perform infilling using extracted keyphrases from the first seed baseline generations. These are the selected models we report the test_CG and test_O results of in §6.

We ask annotators to evaluate 48 test_CG examples from the human references, baseline outputs, and various method (excluding MI) outputs for BART-large and T5-base. We choose these two as they cover both model types and sizes, and exclude MI as it performs noticeably worse on the automatic evaluation (see §6.1). See Appendix §C for more.

The annotators evaluate fluency and commonsense of the texts on 1-5 scales. Fluency, also known as naturalness, is a measure of how human-like a text is. Commonsense is the plausibility of the events described. We do not evaluate coverage as automatic metrics suffice; coverage is more objective compared to fluency and commonsense.

We see from Tables 7 to 10 that SAPPHIRE methods outperform the baselines on most/all metrics across the models on test_CG. The only exception is MI, which performs worse other than coverage.

For BART-base, Kw-aug, Att-aug, and P2T all outperform the baseline across the metrics. For BART-large, Att-aug and P2T outperform the baseline heaviest, with noticeable increases to all metrics. For T5-base, all methods outperform the baseline, with Kw-aug performing best. Att-aug performs best for T5-large, and SAPPHIRE appears relatively less effective for T5-large. T5-large is the strongest baseline, and hence further improving its performance is possibly more difficult.

MI performs worse across most metrics except coverage, likely as MI almost always keeps inputs intact in their exact form. This is however possibly one reason for its low performance; it is less flexible. Further, as discussed in §4.2.2, MI is highly dependent on the input order. See §6.3 for more.

Table 11 contains statistical significance p-values from Pitman’s permutation tests Pitman (1937) for what we adjudged to be the best performing method(s) per model compared to corresponding baselines on test_CG. Most metrics across the methods are significant compared to the baselines.

From Table 12, we see that SAPPHIRE models outperform the corresponding baselines reported in Lin et al. (2020) on test_O. T5-large-KW and P2T outperform EKI-BART Fan et al. (2020) and KG-BART Liu et al. (2021) on both SPICE and BLEU-4, which are two SOTA published CommonGen models that use external knowledge from corpora and KGs. As SPICE is used to rank the CommonGen leaderboard⁵⁵5https://inklab.usc.edu/CommonGen/leaderboard.html, T5-large-KW and P2T would place highly. SAPPHIRE models do lag behind the SOTA published RE-T5 Wang et al. (2021), showing potential for further improvement. Further, the BART-large SAPPHIRE models perform worse than EKI-BART and KG-BART, but not by a substantial margin. We emphasize again that SAPPHIRE simply uses the data itself and the baseline generations, rather than external knowledge. Hence, SAPPHIRE’s performance gains over the baselines and certain SAPPHIRE models matching or outperforming SOTA models that leverage external information is quite impressive.

Table 13 shows human evaluation results on test_CG for human references and methods (excluding MI) using BART-large and T5-base. SAPPHIRE generally outperforms the baselines. BART-large-P2T performs noticeably higher on both fluency and commonsense. For T5-base, all three methods outperform the baseline across both metrics. Compared to humans, our best methods have comparable fluency, but still lag noticeably on commonsense, demonstrating that human-level generative commonsense reasoning is indeed challenging.

We see from Table 14 that many baseline outputs contain issues found in §3.2, e.g. incomplete or illogical sentences. Human references are fluent, logical, and sometimes more creative (e.g. example 5), which all methods still lack in comparison.

For example 1, the baseline generation “hands sitting on a chair” misses the concept “toy”, whereas our methods do not. Kw-aug and Att-aug output complete and logical sentences. For example 2, the baseline generation of “a camel rides a camel” is illogical. Our methods output more logical and specific sentences. For example 3, our methods generate more complete and coherent sentences than the baseline, which lacks a subject (does not mention who is “walking”). For example 4, the baseline generation “bus sits on the tracks” is illogical as buses park on roads. Our methods do not suffer from this and output more reasonable text. For example 5, the baseline generation “A lady sits in a sunglass.” is completely illogical. Kw-aug, Att-aug, and P2T all output logical text. For example 6, the baseline output “Someone stands in front of someone holding a hand” is generic and bland. Kw-aug, Att-aug, and P2T all output more specific and detailed text rather than simply referring to “someone”. Overall, SAPPHIRE generates text that is more complete, fluent, logical, and with greater coverage, addressing many baseline issues (§3.2).

However, SAPPHIRE methods are imperfect. P2T relies heavily on the original generation. For example 1, the baseline output “hands sitting on a chair” is extracted as a keyphrase, and used in the P2T output “hands sitting on a chair with toys”. While coverage improves, the text is still illogical. For example 2, P2T still misses the “walk” concept. While the Att-aug output of “A man is riding camel as he walks through the desert.” is more logical than the baseline’s, it is still not entirely logical as the man cannot ride the camel and walk at the same time. MI outputs logical and fluent text for examples 2 and 3. For the other examples, the generated texts are illogical, not fluent, or incomplete.

This is likely due to input permutation having a strong effect on output quality. For example, “wave” before “falls off a surf board” leads to an illogical output “A wave falls off a surf board.”, where the reverse order results in a more logical output “A man falls off a surf board and hits a wave.” As discussed in §4.2.2, our method of selecting best permutations is likely imperfect. Further, BART-MI usually does not inflect inputs and retains them in exact form, unlike the baselines and other methods which learn to inflect words (e.g. singular to plural). We believe BART-MI has potential if these weaknesses can be addressed.