Cascading Adaptors to Leverage English Data to Improve Performance of Question Answering for Low-Resource Languages

Last few years have seen emergence of transformer based pretrained models like BERTDevlin et al. (2019), XLNetYang et al. (2019), T5Raffel et al. (2020), XLM-RoBERTaConneau et al. (2020) etc. The pretrained models have shown significant improvement in various downstream tasks like question answering, NER, Machine translation and speech recognition when used with word level utilities.Delobelle et al. (2020); Pires et al. (2019); Pfeiffer et al. (2020a); Pires et al. (2019); Pandya and Bhatt (2021); Saha et al. (2021); Murthy et al. (2019); Park et al. (2008); Baxi et al. (2015); Raffel et al. (2020).

The emergence of multilingual models: mBERT Devlin et al. (2019) and XLM-RoBERTaConneau et al. (2020) made it possible to leverage English data to improve the performance of low-resource languages. In this paper, we continue to investigate the effectiveness of multilingual pretrained transformer models in improving the performance of question answering systems in a low-resource setup using the cascading of language and task adaptersPfeiffer et al. (2021, 2020a); Bapna and Firat (2019). Our work¹¹1Our code and trained models are available at : https://github.com/Bhavik-Ardeshna/Question-Answering-for-Low-Resource-Languages contributes by evaluating cross-lingual performance in seven languages - Hindi, Arabic, German, Spanish, English, Vietnamese and Simplified Chinese. Our models are evaluated on the combination of XQuADArtetxe et al. (2020) and MLQALewis et al. (2020) datasets which are similar to SQuAD Rajpurkar et al. (2016) .

To this end, our contributions are as follows:

• •

  We have trained multilingual variants of transformers, namely mBert and XLM-RoBERTa with a QA dataset in seven languages. Both the MLQA and XQuAD datasets contain validation and test sets for the above languages but not the training set. To finetune the model we have combined the test set of XQuAD and MLQA datasets and evaluated the model with the MLQA development dataset as the test dataset. By splitting the dataset in this way we can get train and test data with the considerable length for low-resource languages which helped us to conduct various experiments. Table 1 highlights the size of our train and test set for all the above-mentioned languages.

• •

  We exhaustively analysed the fine-tuned models by evaluating them with the tasks adapter²²2Pre-trained task adapters from https://adapterhub.ml/explore/qa/squad1/ Pfeiffer et al. (2021, 2020a). We conducted the experiments in two different setups, HoulsbyHoulsby et al. (2019) and PfeifferPfeiffer et al. (2021, 2020b). These two setups enabled us to compare our language model variants with their multilingual counterparts and understand the different factors that lead to better results on the downstream tasks.

• •

  We have also attempted a series of two different experiments by stacking language adapters and task adapter³³3Pre-Trained Language Adapters from https://adapterhub.ml/explore/text_lang/ in different ways. We first analyze the fine-tuned model by stacking language-specific adapter with the XLM-RoBERTa_base ⁴⁴4XLM-RoBERTa_base https://huggingface.co/deepset/roberta-base-squad2. After fine-tuning the language-specific adapter we augment the task-specific adapter upon the previously fine-tuned language adapter. We analyze both the experiments separately and conclude that multiple adapters with the transformer-based model perform notably better.

• •

  Due to limited training, the transfer-learning performance of the transformer is poor on the low-resource languages as well as on the languages unseen during the pretrainingKakwani et al. (2020). The multi-task adapter (MAD-X) Pfeiffer et al. (2020b) outperforms the state-of-the-art models in cross-lingual transfer across a representative set of typologically diverse languages on question answering. To avoid the training of model individually for multiple languages while maintaining the performance, we used cross-lingual transfer by switching heads of language adapter from the source language to the target language.

In this section we describe our approach of training the task adapter and the language adapters in 4 different setup.

We have performed 4 different analysis as represented in Figure 1. Details of all 4 setups are shown below:

To study the impact of the task adapter and the language adapters, we have conducted experiments as shown in Setup B and Setup C. Our observations from Table 3 and Table 4 indicates that the trained language adapter (Setup C: language adapters only) improves the performance for Hindi, German, Spanish, Chinese and English languages over the usage of task adapter(Setup B). However, instead of using language adapters only the stack of task and the language adapters lower EM and F1 score for languages other than Arabic.

We have compared two task adapter architectures and noted that the usage of different task adapter architectures have negligible performance impact on majority of the languages. As a result, no clear distinction can be drawn from this observation, which can be used to guide future research.

High-resource languages that use the Left-to-right(LTR) scripting approach dominate the training of pretrained transformer models. The Arabic language follows Right-to-Left (RTL) scripting style. The general poor performance in the Arabic language could be due to a variation in scripting technique. This also demonstrates that, regardless of the downstream task, the language structure has a significant impact on overall performance.

The Chinese language has a symbolic language structure and can be written in a variety of forms (right-to-left, or vertically top-to-bottom). The degraded findings in Chinese compared to other low-resource languages are most likely due to the language’s writing flexibility.

The PARAM Shavak HPC computer facility is used for some of our experiments. We are grateful to the Gujarat Council of Science and Technology (GUJCOST) for providing this facility to the institution so that deep learning studies are being carried out effectively.

We have investigated the efficacy of cascading adapters with transformer models to leverage high-resource language to improve the performance of low-resource languages on the question answering task. We trained four variants of adapter combinations for - Hindi, Arabic, German, Spanish, English, Vietnamese, and Simplified Chinese languages. We demonstrated that by using the transformer model with the multi-task adapters, the performance can be improved for the downstream task. Our results and analysis provide new insights into the generalization abilities of multilingual models for cross-lingual transfer on question answering tasks.