An Empirical Study on Context Length for Open-Domain Dialog Generation

We first focus on the effect of context length on model training. Due to computational constraints, it is often impossible to feed the entire dialog history into the model. Intuitively, giving the model as much history as possible during training should help the model learn how to generate responses since more information is available. But is this the case for Transformer-based dialog models? To figure this out, we compare models trained with the context of different lengths. As shown in Fig. 1, although GPT2 outperforms Transformer on all context length settings, we can observe similar trends for both models: Initially increasing the number of history utterances in the context can improve the performance of the model, but after the context reaches a certain length, continuing to grow the context length is no longer effective. To more concretely reflect the effect of increasing the context length on the model, we define perplexity gain $G_{i}$ as a representation of the gain brought by increasing the context length to $i$:

$$G_{i}=\min_{1\leq j<i}p_{j}-p_{i},$$

(1)

in which $p_{j}$ is the test perplexity of the model trained with context length $j$. A positive $G_{i}$ means that increasing the training length of the model to $i$ can improve performance, and a larger $G_{i}$ means a more significant improvement. As shown in Fig. 1, when the training context length exceeds 5 on DailyDialog and 9 on PersonaChat, increasing the context length will either make the model performance worse or bring minimal gain. This result suggests that, for Transformer-based dialog models, whether trained from scratch or fine-tuned from pre-trained models, the limitation of context length at training time must be carefully considered. Although longer context length in the training phase does not necessarily lead to worse model performance, it does incur unnecessary computational costs.

Previous results concern the overall effect of training context length on the model. But if we take a deep look into the dataset, we find that the context length of the samples varies a lot, ranging from 1 to 25 in both test sets. So here we raise a new question: Do dialogs of different lengths have the same preference for models? To answer this question, we group the test data according to the context length and compare the performance of models trained with different context lengths in each group separately. We denote the model that achieves the lowest perplexity on the entire set as $\mathcal{M}$, the model that achieves the lowest perplexity on group $g$ as $\mathcal{M}_{g}$. For each $g\in\left\{\text{short},\text{medium},\text{long}\right\}$, we measure the gap between $\mathcal{M}$ and $\mathcal{M}_{g}$ as

$$P_{\mathcal{M}}(g)-P_{\mathcal{M}_{g}}(g),$$

(2)

where $P_{\mathcal{M}}(g)$ is the perplexity of $\mathcal{M}$ on group $g$. As shown in Table 1, $\mathcal{M}$ is optimal on half of all groups. On the remaining groups, the gap between $\mathcal{M}$ and the optimal model is quite small. This result suggests that dialogs of different lengths do not have a clear preference for context length in the training phase. The model that performs best on the entire set is a proper choice for dialogs with varying history lengths.

Previous experiments reflect the average performance on the test set, but not all dialog samples benefit from long context. To illustrate this, we split the test set according to context length, where $\mathcal{D}_{i}$ consists of all samples with context length $i$. For each sample in $\mathcal{D}_{i}$, we use a trained model to test its perplexity with all available test context length settings. Then, we count the proportion of samples in each group that achieve optimal perplexity for each test context length. Fig. 2 shows the results on DailyDialog. No matter which test model is used, an unignorable proportion of samples in each test context length setting achieve optimal perplexity. Although most samples achieve optimal perplexity with the longest test context, this ratio shrinks as the dialog history length increases, which indicates that setting a uniform test history length for all dialogs may not be the best practice. Furthermore, we show to what extent setting different context lengths for each sample during the testing phase can improve the model’s performance. For each sample, we specify the context length that makes it the lowest perplexity at test time as its optimal context length. We compare the gap between testing with the maximum context length and the optimal context length on each group and the whole test set. As shown in Table 2, using optimal context length improves the performance of the model in each group, especially on dialogs with longer histories. This improvement is especially noticeable on the Transformer, where we can observe improvements of more than 1 point in most groups. It is surprising that removing part of the history information during the test phase can improve the test performance of the model so much. However, the optimal context length is unavailable in practice because we cannot compute the perplexity without the real responses. We have to determine the context length according to the context itself, which is left to future work.