Conversational QA Dataset Generation with Answer Revision

From a given passage $p$, the CAE module extracts an answer span $a^{s}$ that is most likely to be of interest to a questioner, considering the conversation history $h$, i.e., $P(a^{s}\mid p,h)$. We implemented the module using a pretrained BERT Devlin et al. (2018) with two fully connected (FC) layers at the top Hwang and Lee (2021). Each FC layer predicts the index of start and end tokens of the potential answer span in the passage:

where $prob^{s}_{i}$ ($prob^{e}_{j}$) represents the probability for the $i$-th ($j$-th) token in the passage being the start (end) token of the answer span.

During generation, the top $k$ answer candidates whose start and end indices are $i$ and $j$ are extracted according to the sum of probabilities $prob^{s}_{i}+prob^{e}_{j}$. The CAE module outputs the answer span with the highest sum of probabilities after deduplicating the candidates compared with the answers used in the conversation history. If the candidate set is empty after deduplication, generation is terminated. To train the module, we computed the sum of cross-entropy losses between predicted start and end indices and the ground truth indices.

Considering the input passage and conversation history, the CQG-AR module generates a conversational question and then revises the answer span that is extracted by the CAE module. The module first considers that the extracted answer span is proper for use as an answer and modifies it if it is inappropriate. To enable this process, we collected training examples of passage $p$, conversation history $h$, answer span $a^{s}$, and revised answer $a^{r}$, which contained positive (proper $a^{s}$, $a^{r}$) and negative (improper $a^{s}$, $a^{r}$) pairs. The module can preserve proper answers extracted by the CAE module by learning positive examples. Additionally, negative examples teach the module how to correct improper answer spans with better answers to the generated questions. We devised two negative sampling techniques to collect improper answer spans from proper ones.

We employed CoQA Reddy et al. (2019), which comprised 8k passages collected from seven different domains and human-annotated conversations. To investigate whether CQAG-AR could be effectively utilized to construct a CQA system in a new domain, we split the data into in-domain (Wikipedia) and out-of-domain (children’s stories, literature, news, and middle and high school English exams). The in-domain data were used to train CQAG frameworks, and the quality of synthetic data generated by the trained CQAG frameworks was evaluated using the out-of-domain data.

In addition, we used QuAC Choi et al. (2018) and DoQA Campos et al. (2020) to evaluate our framework. QuAC is based on 13k Wikipedia passages, and DoQA comprises passages collected from FQAs of three practical domains. Because the other two domains constituted only the test set, we used only the cooking domain of DoQA in our experiment. The CQAG frameworks used in experiments could generate only open-ended types of data. Therefore, closed-ended (yes and no) and unanswerable types of examples were excluded from the datasets; these were denoted by CoQA^D, QuAC^D, and DoQA^D.

To evaluate the quality of the synthetic data generated by our framework, we used two baseline CQAG frameworks:

In this section, we evaluate synthetic data generated by CQAG-AR and baseline frameworks by conducting the CQA task. In the first experiment, CQAG frameworks learned the in-domain data of CoQA^D and then generated synthetic CQA data based on the passages extracted from in-domain and out-of-domain data. Note that we constructed synthetic training and validation sets from original splits of CoQA. We implemented a simple CQA model using T5 Raffel et al. (2019) and trained several CQA models using human-annotated data (CoQA^D) and synthetic datasets. The training details, an example of synthetic conversations, and statistics of the synthetic data can be found in Appendix B and C.

Table 1 presents F1 and EM scores of CQA models, which learned the synthetic data, on the test set²²2https://github.com/google/BIG-bench of CoQA^D. The CQA models derived from CQAG-AR outperformed other models, showing significant margins of 11.8% and 13.9% for in-domain data, and average margins of 10.5% and 12.5% for out-of-domain data in F1 and EM, respectively, compared with those derived from the vanilla CQAG. These results demonstrate that the answer revision approach is considerably beneficial in terms of generating valid CQA datasets. However, we additionally found that the out-of-domain CQA models showed lower performances than the in-domain models across all CQAG frameworks. This result implies that CQAG frameworks are less robust when extracting valid CQA data from out-of-domain passages.

In Table 2, we compare the evaluation results on the test set of DoQA^D (cooking) for CQA models trained on human-annotated data (DoQA^D) and synthetic data. We obtained the synthetic data by training CQAG-AR using QuAC^D and then generating the data from the passages of DoQA training and validation sets. According to the results, the CQA model trained on synthetic data, which has the same number of examples with the human-annotated data, significantly outperformed the model trained on human-annotated data. Moreover, our framework generated a larger number of examples than the ones in the original DoQA^D, which improved the F1 score of the CQA model by 1.6%. In particular, examples in QuAC^D, which were used to train CQAG-AR, were irrelevant to the cooking domain. This result indicates that our framework effectively creates synthetic CQA data for an unseen cooking domain.

In Table 3, we classify the synthetic examples according to revision types. The distribution shows that 65.2% of answer spans extracted by the CAE module were preserved without any modification, while the other 34.8% of answers were revised. This demonstrates that the CQG-AR module could recognize invalid answer spans and selectively modify the answers. Furthermore, we found that the module could perform more complex revisions such as “multiple revision” and “complete change” though the CQG-AR module learned only examples for "reduction" and "expansion" obtained by negative sampling.

Further, we conducted human evaluation to compare the quality of synthetic data generated by the vanilla CQAG and our CQAG-AR. From the two synthetic datasets presented in Table 1, 120 examples (30 examples from each out-of-domain dataset) were sampled and three volunteers were asked to rate 80 examples according to the criteria listed in Appendix D.

Although 2.9% more synthetic questions of CQAG-AR are independent of the previous conversations than the questions of the vanilla CQAG in Table 4, the synthetic questions of the two frameworks show almost similar evaluation results. These results prove that the vanilla CQAG, which performs only question generation, and CQAG-AR, which performs question generation and answer revision in an end-to-end manner, can generate questions of similar quality. However, 21.6% more synthetic answers of CQAG-AR were judged as correct answers compared with those of the vanilla CQAG. Furthermore, the number of partially correct answers was considerably reduced through answer revision. This reveals that answer revision is effective in correcting inappropriate answer spans extracted by the CAE module into correct answers that match well with the question.

In this experiment, we tested the effectiveness of the synthetic data generated by CQAG-AR in adapting the CQA model from the Wikipedia domain to new domains (out-of-domain). We trained CQA models, which were initialized with parameters of T5-Large, in three training settings. In the In-Man setting, we trained the CQA model on the Wikipedia data of CoQA^D. In the Out-Syn setting, CQA models learned out-of-domain synthetic data. Finally, the model of In-Man setting was fine-tuned with synthetic data of each out-of-domain case in the In-Man $\rightarrow$ Out-Syn setting.

As shown in Figure 2, the models in the Out-Syn setting yielded results similar to those of the model in the In-Man setting while exhibiting better EM scores in the two domains. Notably, fine-tuning the Wikipedia CQA model using our synthetic data (In-Man $\rightarrow$ Out-Syn) improved the EM scores of the model by an average of 9.7% across all domains. This result indicates that our framework can be effectively utilized for domain adaptation in CQA.