A \DIFdelbegin\DIFdelData Quality Enhancement \DIFdelend\DIFaddbegin\DIFaddData-centric \DIFaddendFramework for Improving \DIFdelbegin\DIFdelendDomain-specific \DIFaddbegin
\DIFaddendMachine Reading Comprehension Datasets

To enhance the quality of the negative contexts \DIFdelbegin\DIFdelselected \DIFaddendfor each passage, we implemented the following procedure. \DIFdelbegin\DIFdelBertScore \DIFdel[zhang2019bertscore], a similarity score between passages , was calculated and the passages \DIFaddend\DIFaddbegin\DIFaddFor each positive context, passages \DIFaddendwere sorted in ascending order of \DIFdelbegin\DIFdelthis score \DIFaddend\DIFaddbegin\DIFaddBERTScore [zhang2019bertscore] similarity with the positive context, and the ones with the lowest score were kept to form negative contexts\DIFaddend. A global counting dictionary was maintained to prevent the replication of \DIFdelbegin\DIFdelhard negative samples \DIFaddend\DIFaddbegin\DIFaddnegative contexts \DIFaddendacross different training examples. This ensured that each negative \DIFdelbegin\DIFdelpassage \DIFaddend\DIFaddbegin\DIFaddcontext \DIFaddenddid not exceed a \DIFdelbegin\DIFdelmaximum of ten \DIFaddend\DIFaddbegin\DIFaddthreshold \DIFaddnumber of \DIFaddendoccurrences in total in the whole dataset.

In this section, we describe the various techniques used to augment the questions from MRC datasets.

For question paraphrasing, we used two models: T5³³3https://huggingface.co/Vamsi/T5_Paraphrase_Paws and Pegasus⁴⁴4https://huggingface.co/tuner007/pegasus_paraphrase. To enhance the data quality of an original dataset, for each original question, we used the two aforementioned methods to generate up to five paraphrased questions. Finally, we created five different training sets in which we grouped the most, second most, up to the least similar paraphrases for each original question together. The word similarity was calculated using a word vector model from spaCy⁵⁵5https://spacy.io/models/en##en_core_web_lg. We also generated a sixth set comprising a randomly-selected question from the list of five unique paraphrases generated.

In word substitution process, we extracted a keyword from each question with the help of the spaCy library and then \DIFdelbegin\DIFdelobtain \DIFaddend\DIFaddbegin\DIFaddobtained \DIFaddenda list of synonyms for each keyword using Natural Language Tool Kit (NLTK)’s English dictionary, WordNet⁶⁶6https://www.nltk.org/howto/wordnet.html. The top five synonyms were extracted from this list in decreasing order of word similarity calculated using the aforementioned word vector model from spaCy. We then generated five versions of the training data for each dataset such that in set 1, the keyword for each question was replaced by its most similar synonym; in set 2, the keyword for each question was replaced by its second most similar synonym and so forth, with set 5 containing the questions with the least similar synonyms as substitutes. For keywords with n < 5 synonyms, we kept the question unchanged in the first (5 - n) versions and used the synonyms as substitutes in the remaining n versions. We also created a sixth set in which we randomly selected one of the top five (or n) synonyms to substitute the keyword for each question.

We used Hugging Face Helsinki model⁷⁷7https://huggingface.co/Helsinki-NLP for back translation. In total, we generated 25 different training sets based on the number of downloads for translation from English to the respective languages, followed by the availability of translation models from the respective languages to English. We selected checkpoints based on the number of downloads, taking the top 25 most downloaded.

To understand how different the resulting questions obtained from each of the enhancement methods are, we ran pairwise comparisons between questions from each method using ROUGE-1. Results are shown in Section 8.6. \DIFaddBack-translation \DIFaddoverall yields the questions most different to the baseline and the other enhancement methods.

Since MRC relies on extracting the exact answer (i.e., text span) from a passage, we could not apply any of the automatic data quality enhancement methods that we applied to questions (as explained in the previous section). However, we created new training datasets in which we manually shorten the original answers where we found this appropriate. We refer to Section 7.3 for more details.

Tables 1 and 2 \DIFaddend\DIFaddbegin\DIFaddTable 2 and Table 3 \DIFaddendshow recall@1/EM scores \DIFaddbegin\DIFaddrespectively \DIFaddendfor fine-tuned retrieval/reader models after enhancing the method of selecting negative contexts (i.e., using BertScore) \DIFaddbegin\DIFaddfor the retrieval training datasets\DIFaddend, as well as reformulation of questions using paraphrasing, word substitution\DIFdelbegin\DIFdeland back translation . \DIFaddend\DIFaddbegin\DIFadd, back translation and answer shortening for the training datasets of \DIFaddboth \DIFaddmodels. More specifically:

• •
  \DIFaddend

  The first row \DIFaddbegin\DIFadd(\DIFaddbaseline\DIFadd) in each table \DIFaddendshows the results of BioLinkBERT/BioBERT BioASQ models fine-tuned on \DIFaddbegin\DIFaddthe \DIFaddendoriginal datasets (i.e., baselines). \DIFaddbegin

• •
  \DIFaddend

  Each subsequential row shows the best results for each dataset using the four aforementioned methods \DIFdelbegin\DIFdel. The last row shows the \DIFaddend\DIFaddbegin\DIFaddfor negative contexts (only for the retrieval models) and questions (for both models) enhancement.

• •
  \DIFadd

  The following row (\DIFaddanswer shortening\DIFadd) shows recall@1/EM scores for fine-tuning of models on the training datasets in which the original answers were manually shortened if needed.

• •
  \DIFadd

  The following row (\DIFaddcontinual\DIFadd) shows the \DIFaddendresults of continual fine-tuning: starting from the best individual checkpoint, we fine-tune on the second-best training set, and so on. For example, for reader fine-tuning on the BioASQ dataset, we first took the checkpoint of fine-tuning on the \DIFdelbegin\DIFdeltest \DIFaddend\DIFaddbegin\DIFaddtraining \DIFaddendset created using paraphrasing and then continued fine-tuning on \DIFdelbegin\DIFdeltest \DIFaddend\DIFaddbegin\DIFaddtraining \DIFaddendsets created using back translation. Finally, we took the newest checkpoint and continued fine-tuning on the \DIFdelbegin\DIFdeltest \DIFaddend\DIFaddbegin\DIFaddtraining \DIFaddendset created using word substitution.

  \DIFaddbegin
• •
  \DIFadd

  The last row (\DIFaddaugmentation\DIFadd) shows recall@1/EM scores for fine-tuning of models on the training datasets created by concatenating all data enhanced training sets if they showed fine-tuning improvements when using individually (i.e., rows 2-6 for retrieval models and rows 2-5 for reader models).

For retrieval fine-tuning (\DIFdelbegin\DIFdelsee Table 1\DIFaddend\DIFaddbegin\DIFaddTable 2\DIFaddend), the most significant improvement of \DIFaddbegin\DIFadd+\DIFaddend8.3 (+33%) from baseline was achieved for BioASQ dataset when using back translation on the Catalan language. \DIFdelbegin\DIFdelFor the other three datasets, the most significant improvement was achieved when using word substitution methods. The enhanced method \DIFaddend\DIFaddbegin\DIFaddThe enhanced methods \DIFaddendof selecting negative contexts and word substitution improved all four datasets, while paraphrasing and back translation caused a decrease in recall@1 scores for SleepQA dataset. Continual retrieval fine-tuning showed improvements when compared to baselines for all datasets, however, only for the \DIFaddbegin\DIFaddCOVID-QA \DIFaddand \DIFaddendcpgQA \DIFdelbegin\DIFdeldataset continual fine-tuning \DIFaddend\DIFaddbegin\DIFadddatasets it \DIFaddendwas better than the best results of individual fine-tuning\DIFdelbegin\DIFdelon test set enhanced with word substitution method. \DIFaddend\DIFaddbegin\DIFadd.

For fine-tuned reader models (\DIFdelbegin\DIFdelsee Table 1\DIFaddend\DIFaddbegin\DIFaddTable 3\DIFaddend), the most significant improvement of 2.1 (+40%) from baseline was achieved for BioASQ dataset when using back translation on the Dutch language, as well as paraphrasing. Continual reader fine-tuning \DIFdelbegin\DIFdeldid not increase the EM scores of any \DIFaddend\DIFaddbegin\DIFaddincreased the EM score only for \DIFaddcpgQA \DIFaddenddataset compared with the corresponding baselines. \DIFaddbegin\DIFaddLastly, augmentation was better than the best results of individual fine-tuning only for the \DIFaddSleepQA \DIFadddataset with the total increase of 2.6 (+4%). \DIFaddend

Evaluation of fine-tuned reader models. \DIFaddend\DIFaddbegin\DIFaddGreater relative improvements with \DIFaddback-translation \DIFaddcompared to other methods could be supported by this method creating more diverse questions (Section 8.6). However, \DIFaddback-translation \DIFaddgains are hardly inconsistent from a dataset to the other. Moreover, we noticed that translation and paraphrasing with PEGASUS gave questions noticeably more different than the other data enhancing techniques. \DIFaddend

We first generated 48 enhanced test \DIFaddend\DIFaddbegin\DIFaddIn total, the data-centric methods that we described in the previous section enabled us to generate 28 enhanced training \DIFaddendsets for retrieval fine-tuning and \DIFdelbegin\DIFdel43 \DIFaddend\DIFaddbegin\DIFadd24 \DIFaddendfor reader fine-tuning. Subsequently\DIFaddbegin\DIFadd, \DIFaddendwe fine-tuned \DIFaddbegin\DIFaddall \DIFaddendretrieval/reader models \DIFdelbegin\DIFdel. Time spent on fine-tuning depicted in Tables 3 and 4 is for one \DIFaddend\DIFaddbegin\DIFaddon a single \DIFaddendNVIDIA-A40 GPU \DIFaddbegin\DIFaddwith \DIFaddend46GB \DIFdelbegin\DIFdel. \DIFaddend\DIFaddbegin\DIFaddof GPU RAM.

We show fine-tuning time in Table 4 and Table 5. \DIFaddendFor example, we used one GPU for five hours to fine-tune retriever model on BioASQ dataset to achieve a 33% improvement in recall@1 score. At the same time, we used one GPU for 22 hours to fine-tune retriever model on SleepQA dataset only to achieve a decrease in recall@1 score for 2%. \DIFdelbegin\DIFdelThe last row in Tables 3 and 4 shows \DIFaddend\DIFaddbegin

The last two rows in tables show \DIFaddendthe time needed for continual\DIFaddbegin\DIFadd/augmentation \DIFaddendfine-tuning only. However, in order to determine the order in which to fine-tune for continual learning, \DIFaddbegin\DIFaddor which datasets to use for concatenation, \DIFaddendall individual checkpoints need to be created. Hence, to obtain the total time for continual learning\DIFaddbegin\DIFadd/augmentation\DIFaddend, one needs to add up times from all previous rows as well.

In this subsection, we describe how we built the final version of datasets from Table 1. \DIFaddendWhere necessary, we divided passages from the original text corpus into one or more parts, so their length was less than 300 words. This step was done so that all passages were of a similar length across different datasets and that the same model hyperparameters can be used for fine-tuning retrieval and reader models. Then we removed those labels for which the answer could not be found in the corresponding positive context. Finally, we divided each original dataset into three parts (in the ratio of 80:10:10) to create training, development, and test sets. \DIFdelbegin\DIFdelTable LABEL:tabA1 \DIFaddend\DIFaddbegin\DIFaddTable 1 \DIFaddendshows the original number of passages in each text corpora, the original number of labels, and the final numbers after the aforementioned adjustments were done.

BioASQ\DIFadd: The original dataset did not include positive passages, but instead contained links to the journal articles where the answers could be found. To obtain positive passages, we first retrieved them from the individual links provided in the dataset, and then divided them into passages no longer than 300 words. Only triples that contained the exact answers in the retrieved passages were included in the final dataset. We encountered a challenge in that, of the 5,821 triplets of the factoid type identified, only 16% had the exact answers that could have been found in the provided passages.

COVID-QA\DIFadd: We first divided the original corpus into passages containing no more than 300 words. We also removed redundant keywords, such as ’introduction:’, ’introductions:’, ’objective:’, ’objectives:’, ’conclusion:’, ’conclusions:’, ’method:’, ’methods:’, ’background:’, ’backgrounds:’, ’result:’, ’results:’, ’result(s):’, and ’aim:’. Additionally, we eliminated leading and trailing spaces and changed all letters to lowercase. To ensure dataset accuracy, further manual cleaning was carried out. This included filling in incomplete words, removing medical abbreviations, and correcting missing brackets such as "()" and "[]\DIFadd".

cpgQA\DIFadd: To prepare the text corpus, we partitioned passages into segments of no more than 300 words, resulting in a corpus of 235 passages. Unfortunately, this division caused some answers to be separated from their corresponding positive contexts due to issues such as inaccurate sentence tokenization and answer fragmentation between two adjacent passages. These discrepancies were addressed through manual intervention. It should be noted that no labels were excluded from the original dataset as a result of this cleaning procedure.

SleepQA \DIFaddThe original dataset already contained passages shorter than 300 words, and all answers were found in their provided passages. We eliminated leading and trailing spaces and changed all letters to lowercase.

BioASQ\DIFadd: The original answers varied from two to more than 120 words in length. Our focus was on shortening the answers that were excessively long, and thus all questions longer than 30 words were manually reviewed. The primary adjustments made to the answers involved isolating the main response to the corresponding question, resulting in lengthy sentences being truncated into shorter phrases. This approach effectively reduced answer length for both the test and training sets by a significant degree. The mean answer length for the training set decreased from \DIFadd30.9 \DIFaddto \DIFadd17.6 \DIFaddwords (Figure 1), while the mean answer length for the test set decreased from \DIFadd26.1 \DIFaddto \DIFadd18.4 \DIFaddwords (Figure 2).

COVID-QA\DIFadd: In the original dataset, the length of the answers was not more than 120 words. However, some answers contained incomplete words at the beginning and/or end of sentences. To improve the dataset’s accuracy, these words were either manually removed or completed. Moreover, scientific abbreviations were eliminated manually to improve the accuracy of exact matches. Unfortunately, this had no significant effect on the mean length of answers for both the training and test sets. This result can be attributed to the training set’s prevalence of sentences with only one or two abbreviations. In other cases, completing the incomplete words also had no effect on the mean word count of the sentences.

cpgQA\DIFadd: In both the training and test sets, answers were shortened manually by removing extraneous phrases and articles (such as "a/an/the") from the beginning of the responses. Before shortening, the mean answer length in the training set was \DIFadd12.7 \DIFaddwords, which was reduced to \DIFadd12.4 \DIFaddwords. In the test set, the mean answer length before shortening was \DIFadd12.1 \DIFaddwords, which was shortened to \DIFadd11.6 \DIFaddwords. The minimal difference in the mean number of words is due to the fact that most answers in the original dataset were clear and concise.

sleepQA\DIFadd: The initial average answer lengths for the sleepQA dataset are \DIFadd10.15 \DIFaddand \DIFadd9.13 \DIFaddfor the train and test set respectively, making it the dataset with the shortest average answer length among all datasets studied. We focused on cutting down answers more than 15 words long, which range up to 40 words long. The cutting of answer was done by extracting the main phrases of the answers which directly responds to the associated questions. The resulting cleaned answers are in the form of shorter, more concise phrases instead of wordy full sentences. The final average answer lengths after the cleaning process are \DIFadd9.11 \DIFaddand \DIFadd8.01 \DIFaddfor the train and test set respectively.

Hyperparameters of retrieval models fine-tuning are shown in \DIFdelbegin\DIFdelTable 6\DIFaddend\DIFaddbegin\DIFaddTable 6\DIFaddend, and of reader models in \DIFdelbegin\DIFdelTable 7\DIFaddend\DIFaddbegin\DIFaddTable 7\DIFaddend. When fine-tuning retrieval models on \DIFdelbegin\DIFdeltest \DIFaddend\DIFaddbegin\DIFaddtraining \DIFaddendsets in which method of selecting the negative contexts for each passage was enhanced, we changed other negatives hyperparameters to reflect the number of negative contexts in the corresponding \DIFdelbegin\DIFdeltest \DIFaddend\DIFaddbegin\DIFaddtraining \DIFaddendset (e.g., 1 to 5). Additionally, when fine-tuning reader models on different datasets, we set eval step to 50 for BioASQ, COVID-QA and cpgQA datasets and 500 for the SleepQA dataset. The reason behind this is because the SleepQA dataset has 4,000 labels in the train set, while the other datasets have less than 1,000 labels. For continual retrieval fine-tuning, we set the num train epochs to 60, and for reader to 30. Other parameters were left the same.

Using the enhanced method of selecting negative contexts, we produced five different \DIFdelbegin\DIFdelenhanced test \DIFaddend\DIFaddbegin\DIFaddtraining \DIFaddendsets for each dataset (see \DIFdelbegin\DIFdelTable 8\DIFaddend\DIFaddbegin\DIFaddTable 8\DIFaddend). Although generally, this method produced enhanced \DIFdelbegin\DIFdeltest \DIFaddend\DIFaddbegin\DIFaddtraining \DIFaddendsets for each dataset, it is not possible to conclude which number of negatives improved the fine-tuning process the best, as this is very much dataset-specific. The last row in \DIFdelbegin\DIFdelTable 8 \DIFaddend\DIFaddbegin\DIFaddTable 8 \DIFaddendshows the time (in hours) needed to generate all five \DIFdelbegin\DIFdeltest \DIFaddend\DIFaddbegin\DIFaddtraining \DIFaddendsets for each dataset using A100 GPU 40GB. While for most of the datasets, the generation process took around one hour, for SleepQA \DIFdelbegin\DIFdelone \DIFaddendit took more than one day!

For question paraphrasing, we used T5 and Pegasus as they are based on Transformer architecture and utilize transfer learning, in which resource-rich sources can be efficiently adapted for resource-poor target fields, such as the domain-specific datasets [yu2018modelling]. T5 was trained on the Google PAWS dataset. Previous research showed that the Pegasus method produces paraphrases that are semantically more different, while the T5 method is found to keep more of the original meaning [martin2023using]. We found that the Pegasus consistently produces the same set of paraphrased questions, regardless of the number generated. For T5, we generated paraphrased questions up to 50 times, after which we took the first five unique paraphrases. For several questions (between 3% for cpgQA dataset and 12% for COVID-QA dataset), T5 failed to produce the required number of unique paraphrases, for which cases we added the original question to the set of five paraphrased questions. Although we used two different libraries, question paraphrasing failed to enhance \DIFdelbegin\DIFdeltest \DIFaddend\DIFaddbegin\DIFaddtraining \DIFaddendset quality for cpgQA dataset altogether. Generating \DIFdelbegin\DIFdeltest \DIFaddend\DIFaddbegin\DIFaddtraining \DIFaddendsets took around 15 hours for SleepQA dataset and 3 hours for other datasets on one NVIDIA TESLA P100 GPU 16GB (Kaggle).

Word substitution is the process of substituting similar words (such as synonyms or words with similar embeddings) \DIFdelbegin\DIFdelfor words in the original training instances \DIFaddend\DIFaddbegin\DIFaddfrom the original data \DIFaddend[pappas2022data]. This method for enhancing the original \DIFdelbegin\DIFdeltest \DIFaddend\DIFaddbegin\DIFaddtraining \DIFaddendsets increased almost all recall@1/EM scores for all datasets for both \DIFaddbegin\DIFaddretrieval/reader \DIFaddendfine-tuning\DIFdelbegin\DIFdelof retrievaland reader models\DIFaddend, except for the reader models for cpgQA and COVID-QA datasets. In cases when applying word substitution on the original dataset did not increase the EM scores for the reader fine-tuning, the scores stayed the same as for the corresponding baselines (i.e., this method did not worsen them). Moreover, the generation of \DIFdelbegin\DIFdeltest \DIFaddend\DIFaddbegin\DIFaddtraining \DIFaddendsets took only 11 minutes for SleepQA dataset and around two minutes for other datasets on one NVIDIA TESLA P100 GPU 16GB (Kaggle).

The main idea behind the back translation method is to use machine translation from a source to a pivot language and back, obtaining paraphrases. \DIFdelbegin

Automatic evaluation of fine-tuned retrieval models using recall@1 for back translation. Baseline recall@1 scores for \DIFdelBioASQ\DIFdel, \DIFdelCOVID-QA\DIFdel, \DIFdelcpgQA \DIFdeland \DIFdelSleepQA \DIFdeldatasets are: \DIFdel25.0\DIFdel, \DIFdel42.5\DIFdel, \DIFdel66.4\DIFdel, and \DIFdel46.8\DIFdel.

In total, we generated 25 different training sets for Spanish (es), French (fr), German (de), Russian (ru), Chinese (zh), Arabic (ar), Dutch (nl), Finnish (fi), Hungarian (hu), Multiple Languages (mul), Ukrainian (uk), Hindi (hi), Danish (da), Czech (cs), Romance Languages (roa), Bulgarian (bg), Catalan (ca), Afrikaans (af), Estonian (et), Turkic Languages (trk), Slavik Languages (sla), Indonesian (id), Slovak (sk), Tagalog (tl), and Kinyarwanda (rw) pivot languages. Back translation has been used as a data augmentation method for several different NLP tasks [feng2021survey, shorten2021text]. \DIFdelbegin\DIFdelResults from Tables 15 and 16 showed that fine-tuning on test sets enhanced by back translation decreased performance for many pivot languages. \DIFaddendGenerally, it produced the best results for BioASQ dataset. The generation of \DIFdelbegin\DIFdeltest \DIFaddend\DIFaddbegin\DIFaddtraining \DIFaddendsets took 10 hours for SleepQA dataset and around two hours for other datasets on one NVIDIA TESLA P100 GPU 16GB (Kaggle).

Table 17 shows \DIFaddend\DIFaddbegin\DIFaddTable 17 shows the \DIFaddendmean and standard deviation for different data quality enhancement methods for retrieval fine-tuning. \DIFdelbegin\DIFdelTable 18 shows \DIFaddend\DIFaddbegin\DIFaddTable 18 shows the \DIFaddendmean and standard deviation for different data quality enhancement methods for reader fine-tuning.

In the following tables, we show the average similarity compute with ROUGE-1 metric between questions obtained with each of the enhancement techniques, over all four datasets {\DIFaddBioASQ\DIFadd,\DIFaddCovidQA\DIFadd,\DIFaddcpgQA\DIFadd,\DIFaddSleepQA\DIFadd}, with Retrieval (first four tables) then Reader (next four).