High Recall Data-to-text Generation with Progressive Edit

Data-to-text (D2T) generation is the task of generating texts from structured inputs (Reiter and Dale, 1997). Previous attempts to solve this task can be classified according to whether separate stages are adopted or not. For example, one method is to generate structured inputs in correct order first, then realize the whole sentence (Puduppully et al., 2019; Wang et al., 2021; Su et al., 2021). Another is to generate the whole sentence in an end-to-end (E2E) manner using Copy mechanism (Gu et al., 2016; See et al., 2017) or just pre-trained models (Kale and Rastogi, 2020).

The two methods each have their advantages and disadvantages. Methods that have separate stages generate more confident texts than E2E models since separate stages first produce entities from structured inputs. However, the output sentence can be awkward or the overall performance may be worse than that of E2E models. This occurs because the generated output of the first stage differs from the gold label that the second stage expects. If the first stage produces a slightly incorrect result, the second stage takes over and increases the error of subsequent sentences. This is often referred to as error propagation. E2E models are free from this vulnerability but it could omit the important entities from structured inputs. To include these important entities, a separate module may be added to E2E models (e.g. Copy mechanism module), but this could generate awkward sentences and degrade system integrity. For this reason, we consider the usage of a pre-trained model without additional modules. A pre-trained E2E Transformer model (e.g. T5 (Raffel et al., 2019)) shows a competitive performance for D2T tasks (Kale and Rastogi, 2020).

Omitting important entities from structured inputs is related to recall values. In D2T, the recall value is a metric that considers not only the target sentence, but also the structured inputs. A high recall indicates that more structured inputs are included. This metric is described in detail in Section 2.

When the same target sentence is repeated, but divided by a special token (i.e. "target_1 <SEP> target_2"), we were able to make two observations. First, we find that Transformer (T5) based model generates asymmetric sentences (Figure 1); i.e., the first part of the output, which is related to target_1, is longer than the second par generated from target_2. Second, the first part of the output covers structured inputs better than the second part. We call this phenomenon Asymmetric Generation.

Asymmetric Generation can be exploited to improve the recall mentioned earlier. Based on our experiments on ToTTo corpus (Parikh et al., 2020) and WIKITABLET (Chen et al., 2020), the first part of asymmetric output shows a higher recall than the second part. It is even higher than the output of the model trained by no-repeated-targets.

We concatenate the first part with a no-repeated-target ("the first part <SEP> no-repeated-target"), then train a new model with lengthened targets (Figure 2). This process can be conducted repeatedly. We call this process Progressive Edit (ProEdit)¹¹1The name of our model comes from ProGen (Tan et al., 2020) because it progressively edits the initial output. Our experimental results on ToTTo corpus demonstrate the benefit of ProEdit in achieving the new state-of-the-art on PARENT (Dhingra et al., 2019) metric.

PARENT Metric. The PARENT metric is introduced to evaluate generated texts from structured inputs automatically.

Its precision for n-gram, denoted by $E_{p}^{n}$, is given by:

where $R_{n}^{i},G_{n}^{i}$ denote the collection of n-grams of order n in $R^{i}$ and $G^{i}$, which are i-th targets and generated texts, respectively. $Pr(g\in R_{n}^{i})$ is given by $\frac{\#G_{n}^{i},R_{n}^{i}(g)}{\#G_{n}^{i}(g)}$, where $\#G_{n}^{i}(g)$ is the count of n-gram $g$ in $G_{n}^{i}$ and $\#G_{n}^{i},R_{n}^{i}(g)$ denotes the minimum of its counts in $G_{n}^{i}$ and $R_{n}^{i}$. Entailment probability, denoted as $w(g)$, is the most important and further introduced to check whether the presence of an n-gram $g$ in a generated text is "correct" given structured inputs. Two models have been introduced to calculate of $w(g)$: the Word overlap model, and the Co-occurrence model. In the most cases, the Word overlap model is used, so we also used it too.

The recall of PARENT is computed against both the target $E_{r}(R^{i})$ and the table $E_{r}(T^{i})$. These are combined for geometric average:

$E_{r}(R^{i})$ compute the recall of generated sentences against target sentences. $E_{r}(T^{i})$ is computed as follows:

K denotes the number of records in table and $\bar{r_{k}}$ is the number of token in the value string of a record. $LCS(x,y)$ denotes the length of the longest common subsequence between x and y. The hyperparameter $\lambda$ can be obtained heuristically using Eq.3 . The key idea is that if the recall of the target against the table is high, it already covers most of the information of structured inputs, so we can assign it a big weight ($1-\lambda)$. We use this method.

We add the first part of asymmetric generation to an existing target and trained a new model by using this dataset. This process is reiterated so that after the output of the previous model is added to an existing target, a new model is trained by this dataset. As a result the output sentence goes through progressive editing. (Figure 2). ProEdit was repeated until the PARENT metric increased.

Recall and Length. In general, a longer sentence increases the recall value. The first part of Asymmetric Generation and ProEdit have longer sentences than the reference. For fair comparison, similar output lengths are generated using beam search and minimum length settings (Table 3). For beam search, the longest sentence is selected. However, selecting the longest sentence in the beam search rather reduced the recall value. In minimum length settings, the probability of the EOS token is zero until the output reaches a certain length. Setting the minimum length improves both length and recall values, but dramatically reduces precision. which led to a steady decrease in the overall F1 score. Our proposed method can improve the overall F1 score since the precision decreases slightly.

In this paper, we proposed Progressive Edit (ProEdit) process for D2T generation. It utilizes Asymmetric Generation to improve recall. We obtained the new state-of-the-art result on ToTTo.