Miko Team: Deep Learning Approach for Legal Question Answering in ALQAC 2022

Information retrieval is the problem of gathering information sources related to searchable information. Document retrieval is the information retrieval on text data. This task aims to find articles related to a query on a set of articles.

The traditional method commonly used is the vector space model [5] such as TF-IDF [6], BM25 [7], and Word2Vec [8] or Doc2Vec [9]. In this method, the text is represented by a vector of terms. The relevance between a query and an article is evaluated by calculating the similarity between the two vectors (vector representing the query and vector representing the article). This method has limitations in understanding the context when a word will always produce a fixed feature vector, while the advent of deep language models has overcome this weakness.

Some use the above traditional method to extract features and then put them into a pair of text classification models using Random Forest, K-Nearest Neighbor, Support Vector Machine, or ensemble of classification models [10]. This approach proves to be more effective as the classification model continues to learn from the acquired features.

Most recently, the introduction of deep pre-trained language models such as BERT, ElMo [11], and XLNet [12] has solved the limitation in understanding the context of traditional representation models. The vector representing the word is now defined based on its context instead of being fixed. In addition, many studies have shown that further training BERT or completely pre-training the model from scratch using a domain-specific corpus can significantly improve its performance, SciBERT [13] and BioBERT [14] are examples. The results show that the finetuned BERT according to the data domain is more efficient than the original model. Studies combining a traditional model (e.g., BM25) with a deep pre-trained language model (e.g., BERT) achieved impressive results in the COLIEE [15] and ALQAC [16] legal data processing competitions last year. Inspired by this success, we apply a similar pipeline with the focus on RoBERTa - a variant of BERT that has been trained on a large amount of data that proves to be more effective than the original one in the question answering task.

Traditional approaches for open-domain question answering are based on knowledge graphs [17] [18] using a variety of models. First, the question goes into the semantic analysis model, which is trained to predict queries. These queries will guide what to do in the knowledge graph to get the answer. Although this approach works, there are many models that are trained independently, making development and improvement difficult with accumulative errors.

In 2020, [19] proposed a way to implement end-to-end question answering in conjunction with knowledge graphs. This makes using the knowledge graph simpler and more efficient.

Besides using the knowledge graph, the method of text indexing and finding the answer in the relevant passage using the deep language model also shows high efficiency [20]. Inspired by this study and the document retrieval task in this competition, we propose the answer extraction method using the RoBERTa model for the question answering task.

BM25 is a ranking algorithm often used by the search engine to estimate the relevance between a set of documents to a query. It ranks the documents based on the query terms appearing in each record. We use this method because we can use it directly without training, and the computation time is fast enough.

BM25 will help us find the most relevant articles to generate “hard-enough” negative training samples and reduce the amount of data that the text pair classification model needs to predict.

The following text preprocessors were used prior to the application of BM25:

• •

  Word segmentation using pyvi²²2https://github.com/trungtv/pyvi

• •

  Convert text to lowercase

• •

  Remove punctuation and stopwords

The dataset in this task consists of official samples (520 questions and 1377 articles) and additional samples from last year’s Zalo legal text retrieval competition (3196 questions and 61425 articles).

To ensure that the data length is within the range of the text pair classification model, we checked the length distributions of questions and articles of both datasets and got the results as shown in Figure 2. Observing this distribution, we can see that many data articles are over 2000 characters in length and concentrated in Zalo’s dataset. The data that is too long when included in the model will be truncated, which results in loss of information leading to errors that reduce the effectiveness of the trained model.

We broke a long article into multiple sub-articles according to fixed window size and stride to deal with the lengthy article problem. Besides, we will ignore data pairs whose question is too long.

• •

  In the training phase, we apply article segmentation to separate a given article into passages. Then use the BM25 algorithm to find the most relevant passage of a given query question, and this passage is present for the whole article. The scenario is that the given article is too long and it contains much unnecessary information. With this approach, we can limit the searching space and improve the quality of sentence pairs to train the RoBERTa model.

• •

  In the inference phase, we also apply the article segmentation to handle the long article problem. The result with the highest probability will be the final result of the question and article pair.

For each question, the lexical matching model will give the most relevant top k articles (including the true relevant articles). Articles that are not in the relevant set will be used as negative samples.

Finally, we shuffle the data, taking 20% of the generated in the official data as the dev set and using the remainder for training the deep learning model.

To improve the performance of RoBERTa on the law domain, we finetuning it with 4GB of legal text data. We collect this data in two different methods:

• •

  Collected directly from 2 websites vbpl.vn and lawnet.vn.

• •

  Extract sentences close to the legal topic from the news corpus.

In the first method, the collected data information including the number of articles (# articles) and the number of sentences (# sentences) is provided in Table I below. The number of sentences here is the sentences that are retained after the selection process (ie. removing non-Vietnamese and duplicate sentences).

In the second method, following the work in [23], we first build a set of legal documents called ”in-domain” data to extract legal documents from the news corpus. We then proceed to make a statistical language model (base language model) on this in-domain dataset. Use this model to evaluate the perplexity score on each sentence in another corpus and select sentences having a score within the threshold.

Perplexity is a popular metric used to evaluate language models. Based on the perplexity score, we can know how good our language model is in our data domain. In this case, we assume our base language model is good enough because it is learned from ”in-domain” data. We use it to score other sentences to see if each of them is close to the ”in-domain” that we extract from the articles provided by the ALQAC organizer. Next, we extracted data close to in-domain data from the news corpus³³3https://github.com/binhvq/news-corpus and obtained 2GB of data close to the legal domain. Table II shows some sentences in the news dataset and their corresponding perplexity scores.

The data collected from both above methods is used to finetuning the mask language model of the xlm-roberta-large⁴⁴4https://huggingface.co/xlm-roberta-large [24] to make the model more efficient in the legal domain. Before using the data for training, we performed a number of pre-processing steps including:

• •

  Standardize the typing of the accent (eg. oà to òa),

• •

  Standardize the Vietnamese charset

• •

  Convert abbreviation to full-form (eg. HĐXX to hội đồng xét xử)

As shown earlier, we use the text pair classification model to derive the final result from the list of most relevant articles retained by the lexical matching model.

We merge question text and article text into a single sentence separated by the ”SEP” token of the BERT model along with the corresponding label of the question and article pairs. In this way, the model can recognize and learn the relationship between question and article.

We consider this a binary classification problem, the label is 1 if the question and article are related and 0 otherwise.

The computational time of the deep learning model is much slower than that of the BM25 ranking function. Therefore, we use BM25 on each question to retain the 150 articles most relevant to the question. Only the 150 most relevant articles are fed into the deep learning model to generate the final results. For the long article, we continue to apply the method of dividing into small documents like the data generation process.

During the experiment, we use the F1 score to evaluate the quality of the model on the dev set. The official evaluation metric for this task on the private test is the F2 score as follows:

In this task, we submit two different methods with corresponding results presented in Table III. The first one (ie. xlm-roberta-large) is the model finetuning from the original xlm-roberta-large model and we did not submit it. For the submitted methods, we use the RoBERTa-law to finetuning the text pair classification task, keeping only the best relevant article for each question based on the model’s predicted probabilities in both submitted files. The dev set here is a random 20% of the official dataset after generated.

The difference between the two submissions is the choice of $k$ when generating data using BM25:

• •

  The first submission uses $k_{1}=150$ for the official dataset and $k_{2}=20$ for the Zalo dataset.

• •

  The second submission is the voting results of 2 models: (1) using $k=150$ for both data sets and (2) the model above.

Our approach to this task is based on the results of the first task. First, we search for text related to the question, then use our RoBERTa-law to finetuning the question answering task to find the answer.

To improve efficiency when finetuning this question answering task, we finetuning with additional question answering data mailong25/bert-vietnamese-question-answering⁵⁵5https://github.com/mailong25/bert-vietnamese-question-answering/tree/master/dataset then finetuning to this task’s dataset.

In this task, we used three different settings corresponding to the results shown in Table IV using the F1 score on our dev set and private test. The dev set here is random pick 15% of the official dataset:

• •

  Only uses the first submission of task 01.

• •

  Use both submissions of task 01.

• •

  Only uses the second submission of task 01.