Open-Domain Dialogue Generation Based on Pre-trained Language Models

Transformer-ED is an encoder-decoder architecture as in Vaswani et al. (2017), which utilizes Generative Pre-Training (GPT) Radford et al. (2018) to initialize the encoder and the decoder. GPT is based on left-to-right attention, where every position can only attend to previous positions in the masked multi-head attention layer of each Transformer Block. Transformer-ED first encodes dialogue history, and then the final outputs of encoder are fed into the decoder to generate a response. The encoder uses bi-directional attention, while the decoder uses left-to-right attention.

In this framework, the decoder is stacked upon the encoder outputs. This makes the fine-tuning process less effective in updating the encoder’s parameters. The potential problem with separate encoder and decoder has been noticed in a previous work on abstractive summarization Liu et al. (2018), which noticed that an explicit encoder in the Transformer-ED architecture might be redundant, i.e. the encoding step could be directly incorporated in the decoder, allowing for a more direct update of the parameters. Our experiments will confirm this problem.

Transformer-Dec is a decoder-only architecture utilizing GPT. It encodes dialog history using only left-to-right attention, in the same way as in GPT pre-training. However, the fact that only left-to-right attention is used may limit the scope of context when generating responses. Many previous studies have shown that the context after the token is useful for the encoding of the token. It might be better to use bi-directional attention whenever possible.

Transformer-MLM and Transformer-AR have an identical decoder-only architecture ²²2Since the architecture is not explicit encoder-decoder, we categorize it into decoder-only though BERT is usually viewed as a pre-trained encoder. that employs different type embeddings and self-attention masks for the source/encoder and target/decoder sides: they use bi-directional attention on the source side and left-to-right attention on the target side.

The pre-trained BERT Devlin et al. (2018) is a bi-directional architecture using MLM as the pre-training objective. The same attention mechanism is used in the encoder part of Transformer-MLM/AR, but the decoder part uses left-to-right attention. On the loss function, Transformer-MLM fine-tunes model parameters using MLM as BERT, which masks a certain percentage of tokens at the target side and tries to predict them; while Transformer-AR uses a different auto-regressive objective, which tries to predict the next tokens successively.

The frameworks we described have been recently applied to dialogue generation. For personalized response generation, Wolf et al. (2019) uses Transformer-Dec and Zheng et al. (2019) utilizes Transformer-ED. Lin et al. (2019) uses Transformer-Dec for empathetic response generation. The training data for fine-tuning could be limited in practice. Several studies propose to further pre-train models using large-scale dialogue datasets before fine-tuning: Zhang et al. (2019b) trains Transformer-Dec on 147M Reddit data, Dong et al. (2019) trains Transformer-MLM on natural language understanding and natural language generation datasets, Shuster et al. (2019) trains Transformer-AR on large-scale Reddit data and then jointly trains on 12 dialog sub-tasks, and Bao et al. (2019) trains a variant of Transformer-AR on large-scale Reddit and Twitter data. Another trend emerging in the literature Adiwardana et al. (2020); Roller et al. (2020); Bao et al. (2020b) is to train Transformer variants with billions of parameters on extremely large-scale dialogue dataset, e.g. elaborately pre-processed Reddit.

The purpose of our study is different: we do not intend to augment training data to push the performance. Instead, we want to examine the intrinsic characteristics of different frameworks for dialogue generation by comparing them on the same datasets. In addition, it may often be the case that the training data for fine-tuning is limited in some domains. So, we also need to investigate the behaviors of the frameworks in such situations. Therefore, in the next section, we will compare the 4 frameworks on 3 public datasets, and examine them with two data scales – millions and 100K.

We use three public datasets that are widely used for open-domain dialog generation. Some important characteristics of the datasets are summarized in Table 5. For each dataset, we also evaluate model performance using only 100K training data with the same validation set and test set, in order to simulate the situation of limited data. Many dialogue datasets that have human annotations such as PersonaChat Zhang et al. (2018) and EmpatheticDial Rashkin et al. (2019) are in similar small scale.

Twitter Dialog Corpus ³³3https://github.com/Marsan-Ma-zz/chat_corpus is collected from Twitter consisting of 2.6M (message, response) pairs. We filtered out a sample if its history length is longer than 72 words or shorter than 6 words. Samples whose response is longer than 36 words or shorter than 6 words are also removed. As results, we keep 2M samples.

Reddit Conversational Corpus ⁴⁴4https://github.com/nouhadziri/THREDDziri et al. (2019) is a high-quality 3-turns conversational dataset collected from 95 selected subreddits. The dataset contains 9.2M samples. We applied the same length constraints and then randomly sampled 3M samples.

Ubuntu Dialogue Corpus V2.0 ⁵⁵5https://github.com/rkadlec/ubuntu-ranking-dataset
-creator Lowe et al. (2017) is two-person conversations extracted from the Ubuntu chat logs of technical support for various Ubuntu-related problems. We split a conversation that has more than 3 turns into several 3-turn dialogues. Besides, we filtered out samples to have similar dialogue history and response length as the other two datasets.

We equip all transformer variants with an identical decoding algorithm⁶⁶6From UniLM, https://github.com/microsoft/unilm/ to avoid extra factor affecting the generation quality, which uses beam search with beam size of 4, prevents duplicated uni-grams, and sets minimum response length that encourages diverse generation Roller et al. (2020). The minimum response length is set to make the average length of generated responses match with the average target length of the dataset. Generation results are evaluated after applying an identical word tokenization method. With two P100 GPU devices, we fine-tune all models for 4 epochs on Twitter and Reddit dataset, and for 5 epochs on Ubuntu dataset. Maximum input length is set to 128. Our methods (PF-free and FG-free, which will be described in Section 4.2) do not add parameters or increase runtime in comparison to Transformer-MLM. More implementation details are given in Appendix C.

Transformer-ED performs the worst. We observe that it is the only framework that tends to generate safe responses such as ”i am not sure what you are talking about” (except on million-scale Reddit dataset). This phenomenon is also revealed by its substantially lower Distinct scores. When training with 100K data, the generated responses are almost bland, which can be observed in generation samples (Appendix E). As mentioned earlier (section 2.1), the Transformer-ED architecture might be redundant Liu et al. (2018) – the separate encoder may not be necessary and be difficult to optimize.

Transformer-AR obtains the highest BLEU and CIDEr scores on all three million-scale datasets. Since the only difference from Transformer-Dec is that Transformer-AR applies bi-directional attention on source side, we can explain the difference in performance by the attention mechanism. This suggests that bi-directional attention on the source side helps the model to better encode the dialogue history and thus produce better responses. Human evaluation results also show that Transformer-AR performs well when fine-tuning with million-scale data. However, with small-scale datasets, Transformer-AR loses its advantages, and becomes less effective than Transformer-Dec and Transformer-MLM. We believe that this is due to the discrepancies in Transformer-AR, which will be analyzed later.

Transformer-Dec is able to generate the most diverse responses. We attribute this to the left-to-right attention on the source side, which can introduce more flexibility for generation since it does not have constraints from the right side. Furthermore, when fine-tuning data are limited, Transformer-Dec has the best performance on both Twitter and Reddit datasets according to human evaluation.

Transformer-MLM outperforms other frameworks in terms of BLEU metrics when fine-tuning data are limited, which might have benefited from the bi-directional attention on the source side. Although with million-scale dataset its automatic evaluation scores are not the best, human evaluation shows that its performance is comparable to Transformer-Dec and Transformer-AR. Furthermore, we observe that Transformer-Dec and sometimes Transformer-AR will simply completely copy the input sentence in the output (see examples in Appendix E, Table 11 and 12), but this seldom happens with Transformer-MLM. We conjecture that the MLM objective contributed in alleviating the issue.

Overall, our experiments suggest model architecture is strongly related to the success of fine-tuning a pre-trained model for dialogue generation. The best configuration uses bi-directional attention on the source side and combines encoder and decoder in the same multi-layer transformer blocks.

In Table 1, we emphasized the pretrain-finetune discrepancy in red. All the frameworks, except Transformer-Dec, have some discrepancies, while only Transformer-MLM has finetune-generation discrepancy: during training, the model input has random masks as shown in Figure 1, while in generation process, the input consisting does not contain masks. Discrepancies may affect the model performance since models with such discrepancies cannot best exploit the pre-trained language model or best employ the fine-tuned model. In this subsection, we try to understand how model discrepancies affect the model.

When a large training dataset is available for fine-tuning, we observe the impact of pretrain-finetune discrepancy is less severe – the model can gradually be adapted to the given task. However, if the training data are limited, the discrepancy problems may surface.

Specifically, Transformer-ED is the framework that has the largest pretrain-finetune discrepancy. We observe that its performance decreases a lot from large training data to small training data. On the other hand, Transformer-Dec has the least pretrain-finetune discrepancy, and we can observe much smaller decrease in performance (especially in human evaluation). We can also compare Transformer-MLM and Transforme-AR, which use an identical architecture. Transformer-AR has additional pretrain-finetune discrepancy due to its auto-regressive objective in fine-tuning. We can see that the performance of Transformer-AR is more reduced when fine-tuned on a small dataset: Transformer-AR performs better than Transformer-MLM with large training datasets, but loses its advantages with small training datasets according to automatic evaluation. The human evaluation can show this larger decrease more clearly.

The above analysis suggests that with a small dataset one should make efforts to reduce pretrain-finetune discrepancy to best exploit pre-trained models. In terms of finetune-generation discrepancy, since this is a problem for Transformer-MLM only, it is difficult to show its impact because no comparison with other frameworks is possible.

As we have revealed the two discrepancies as potential factors that negatively affect the model performance, in the next section, we propose approaches to reduce pretrain-finetune discrepancy and finetune-generation discrepancy, and we will test the performance of the modified models.

Let us first briefly introduce the general transformer mechanism, and then describe our methods to reduce discrepancies.

A dialog history is denoted by $x$, and a corresponding response is denoted by $y$. The input to the multi-layer transformer is the concatenation of dialog history and the response. When using MLM objective, the response is randomly masked. The input representation $\mathbf{H}^{0}\in\mathbb{R}^{n\times d_{h}}$, where $n$ is the input length and $d_{h}=768$ is the hidden dimension, is the sum of token embedding, position embedding, and type embedding at each position. The type embeddings introduce a separation between encoder/source side and decoder/target side in order to warrant different treatments in the model. Then, $\mathbf{H}^{0}$ is encoded into hidden representations of $i$-th layer $\mathbf{H}^{i}=[\mathbf{h}_{1}^{i},...,\mathbf{h}_{n}^{i}]$ by: $\mathbf{H}^{i}={\rm Trans}^{i}(\mathbf{H}^{i-1}),\quad i\in[1,L]$, where ${\rm Trans}^{i}$ denotes the $i$-th Transformer Block as shown in Figure 2(Left). The core component of a transformer block is the masked multi-head attention, whose outputs are $\mathbf{C}^{i}=[\mathbf{c}_{1}^{i},...,\mathbf{c}_{n}^{i}]$ that are computed via $\mathbf{C}^{i}={\rm Concat}(\mathbf{head}_{1},...,\mathbf{head}_{h})$, with

$$\mathbf{head}_{j}={\rm softmax}(\frac{\mathbf{Q}_{j}\mathbf{K}_{j}^{T}}{\sqrt{d_{k}}}+\mathbf{M})\mathbf{V}_{j}$$

(1)

where $\mathbf{Q}_{j},\mathbf{K}_{j},\mathbf{V}_{j}\in\mathbb{R}^{n\times d_{k}}$ are obtained by transforming $\mathbf{H}^{i-1}\in\mathbb{R}^{n\times d_{h}}$ using $\mathbf{W}_{j}^{Q},\mathbf{W}_{j}^{K},\mathbf{W}_{j}^{V}\in\mathbb{R}^{d_{h}\times d_{k}}$ respectively. $\mathbf{M}\in\mathbb{R}^{n\times n}$ is the self-attention mask matrix that determines whether a position can attend to other positions. $\mathbf{M}_{ij}\in\{0,-\infty\}$. In particular, $\mathbf{M}_{ij}=0$ allows the $i$-th position to attend to $j$-th position and $\mathbf{M}_{ij}=-\infty$ prevents from it. Figure 2 (Right) shows two $\mathbf{M}$ settings that are applied by Transformer-MLM/AR and Transformer-Dec respectively.

If we reduce the pretrain-finetune discrepancies of Transformer-AR, we get Transformer-MLM by replacing the AR objective to MLM objective. The discrepancy of Transformer-MLM comes from the left-to-right attention on the target side that has not been pre-trained in BERT. Therefore, this discrepancy cannot be eliminated during fine-tuning for a generation task. However, we might alleviate the discrepancy by using bi-directional attention also on the target side. Specifically, at inference time, to generate a new token denoted as $g_{t}$, [MASK] is fed into $t$-th position, denoted as $g_{t}$-M. Previously generated tokens $g_{<t}$ could be viewed as a special type of dialogue history, and we can apply bi-directional attention on it.

However, previously left-to-right attention is used on $g_{\leq t}$, and thus only hidden states at $(t-1)$-th and $t$-th positions need to be updated. If we directly apply bi-directional attention on $g_{\leq t}$, all hidden states at $\leq t$ positions need to be updated. In other words, this generation strategy requires to update all hidden states at each time step of generation. Our experiments show that with such high updating frequency the generated responses usually lack fluency (e.g. BLEU-2 is 1.519 on Twitter). Furthermore, to keep training consistent with inference, only one token can be masked in each training sample; otherwise, there will be conflict for the self-attention mask (Appendix A, Figure 5), i.e. different self-attention mask matrices are required for different masked tokens, while only one mask matrix can be provided per training sample. This would lead to much lower training efficiency: the loss on validation set only decreases slightly to $5.39$ from $6.27$ after four epochs, while Transformer-MLM masking 40% of the target tokens can reduce it to $4.35$.

To summarize, in generation, we cannot always update previous hidden states using bi-directional attention, and the training process should be consistent to the generation in order not to add new finetune-generation discrepancy. Therefore, we propose a simple method by setting a time-step interval for bi-directional attention on the target side to decrease updating frequency – within the interval we apply left-to-right attention and at the end of an interval we apply bi-directional attention. The corresponding training method allows us to mask multiple target tokens at the same time to guarantee training efficiency.

Figure 3 illustrates the generation process of our method with interval of 3. Before time step 3, left-to-right attention is used (e.g. t=2). At time step 3, bidirectional attention is allowed. Then left-to-right attention is used (e.g. t=5) before the end of next interval cycle (t=6). Accordingly, the training process is: given a target response, we first randomly select among all (3 in the figure) possible attention patterns (e.g. the case of t=3 or t=5 in Figure 3, where we apply bi-directional attention on $y_{0,1,2}$); then in the part of left-to-right attention, we randomly mask several tokens. We can mask multiple tokens because this part applies left-to-right attention and the masks at other positions will not influence the prediction on a given mask. We call our modified approach PF-free, which means that the pretrain-finetune discrepancy is reduced.

A model having finetune-generation discrepancy means the way that it is used in generation (inference/test) is different from the way it has been trained. Only Transformer-MLM has finetune-generation discrepancy because of its MLM objective as shown in Figure 4: during training, there is a masked token, $y_{1}$-M, before $y_{2}$-M, while in generation process, there is not a masked token before when generating the token $g_{2}$-M.

Therefore, we propose that at training time, rather than replacing the tokens with [MASK] as in vanilla MLM, we keep all original input tokens unchanged and prepend [MASK] tokens in the input sequence as illustrated. The prepended [MASK] token uses the same position embedding of the corresponding token. Then, every position after $y_{1}$-M attends to $y_{1}$ instead of the [MASK] token, and thus the finetune-generation discrepancy of MLM is eliminated. A similar method has also been explored in Bao et al. (2020a), where they introduced an extra pseudo mask in additional to [MASK] and prepend it before the original token in order to handle factorization steps of their partially auto-regressive language model.

We call the modified model FG-free, in which the finetune-generation discrepancy is reduced.

The results with PF-free, FG-free and PF&FG-free models are reported in previous tables together with other models. We can see that both methods to reduce discrepancies improve Transformer-MLM performance. Specifically, PF-free improves BLEU on Twitter and Ubuntu datasets, and it always outperforms Transformer-MLM in terms of CIDEr and Distinct scores on all three datasets. Besides, as have been discussed, Transformer-MLM that has smaller pretrain-finetune discrepancies outperforms Transformer-AR when training data are small. These results together suggest that the effort to reduce pretrain-finetune discrepancy when training data are small is beneficial.

In comparison, PG-free results in even better performances, which always brings statistically significant improvement over Transformer-MLM in all automatic metrics on all three datasets. Therefore, we can see that it is necessary to apply our method on Transformer-MLM to eliminate finetune-generation discrepancy. However, when we combine the two correction methods to reduce both discrepancies, we do not observe an cumulative effect. The relation between the two discrepancies remains unclear. This question is worth being investigated further in the future.