Leveraging Domain Agnostic and Specific Knowledge for Acronym Disambiguation

Word sense disambiguation (WSD) is an open problem concerned with identifying which sense of a word is used in a text (Navigli 2009). It is a core and difficulty in natural language processing tasks, which affects the performance of almost all downstream tasks. The methods to solve word sense disambiguation are usually divided into two categories: knowledge-based and supervised (Wang, Wang, and Fujita 2020; Barba et al. 2020).

Knowledge-based methods usually rely on amounts of statistical information and can be easily extended to other low-resource languages (Agirre, López de Lacalle, and Soroa 2014; Scarlini, Pasini, and Navigli 2020). For example, SensEmBERT (Scarlini, Pasini, and Navigli 2020), a knowledge- and BERT-based method that combines the expressive power of language modeling with the vast amount of knowledge contained in the semantic network, produces high-quality latent semantic representations of the meanings of the word in different languages. And it can achieve competitive results attained by most of the supervised neural approaches on the WSD tasks. On the other hand, supervised methods require lots of labeled data to learn word representations (Bevilacqua and Navigli 2020; Wang, Wang, and Fujita 2020). Of course, this defect can be alleviated through semi-supervised methods (Barba et al. 2020) by jointly leveraging contextualized word embedding and the multilingual information to project some sense labels.

Furthermore, acronym disambiguation is more challenging since we need to identify the acronym first and then to understand the text to determine the correct meaning of acronyms. Recently, an effective solution is to extract acronym definitions from unstructured texts by computing the Levenshtein string edit distance between any pair of long forms (Ciosici, Sommer, and Assent 2019), which is an entirely unsupervised acronym disambiguation method. And researches also attempt to incorporate hand crafted features (Li et al. 2018), word embeddings (Charbonnier and Wartena 2018; Ciosici, Sommer, and Assent 2019), graph structures (Prokofyev et al. 2013; Veyseh et al. 2020b), and deep learning architectures (Jin, Liu, and Lu 2019; Blevins and Zettlemoyer 2020), and have achieved significant effects in this task. Specifically, a supervised method named GAD (Veyseh et al. 2020b), which utilizes the syntactic structure of sentences to extend ambiguous acronyms in sentences by combining Bidirectional Long Short-Term Memory (BiLSTM) with Graph Convolutional Networks (GCN), provides a strong baseline on acronym disambiguation tasks in the scientific domain.

Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al. 2019) is a self-supervised learning method that trains based on a large number of corpora to express better features for word embedding. And its network architecture utilizes the multi-layer transformer structure (Vaswani et al. 2017). The feature representation of BERT could be directly adopted as word embedding features for downstream tasks. Besides, BERT provides a model for transfer learning of other tasks. It can be fine-tuned or fixed according to tasks and then treated as a feature extractor. BERT was significantly undertrained, and there have been many fine-grained improvements or specific domain variants of it (Beltagy, Lo, and Cohan 2019; Liu et al. 2019; Scarlini, Pasini, and Navigli 2020; Lee et al. 2020).

Acronym disambiguation is formulated as a sequence classification problem in general (Veyseh et al. 2020b). Formally, given an input sentence $\bm{s}=w_{1},w_{2},...,w_{n}$ and the position of the acronym, i.e., $p$, the goal is to disambiguate the acronym $w_{p}$, that is, predicting the true long form $\bm{l}$ from all candidate long forms of $w_{p}$. Specifically, in this paper, we simplify it into a binary classification problem. That is, given an input sample consists of the sentence $\bm{s}$ with acronym $w_{p}$ and the candidate long form $\bm{l}$, i.e., $\bm{x}=(\bm{s};\bm{l})$, our purpose is to predict the probability of $\bm{l}$ being the right long form of $w_{p}$. We assign a label $y\in\{0,1\}$ on each sample in training dataset to indicate whether $\bm{l}$ is a true long form of $w_{p}$ in sentence $\bm{s}$ or not. In the testing phase, the long form with the highest prediction probability among the candidate long form set of a sentence would be chosen as its final result.

Figure 1 exhibits the schematic illustration of the proposed hdBERT model. As mentioned previously, we design a hierarchical integration model comprising three major components, each plays a different role in final prediction. The first two context-based components, i.e., RoBERTa (Liu et al. 2019) and SciBERT (Beltagy, Lo, and Cohan 2019) modules, distill representations of the sentence and the candidate long forms. Specifically, as a robustly optimized method trained on vast amounts of general domain corpora, we use RoBERTa to capture the general and fine-grained semantic information via byte-level Byte-Pair-Encoding (Sennrich, Haddow, and Birch 2016). Moreover, SciBERT, which leverages unsupervised pretraining on a large scientific corpus by WordPiece (Wu et al. 2016) tokenization strategy, is exploited to represent the high-level scientific domain information. Finally, a multiple layer perceptron network is devised to fusion these two kinds of representations. In the following, we present detail of each major component.

After modeling the two complex representations above, the obtained concatenation $\bm{h}$ is fed into multiple layer perceptron network and followed by a regression layer with $\mathrm{sigmoid}$ unit, as follows:

$$\bm{h}=[\bm{h}_{\mathrm{RoBERTa}};\bm{h}_{\mathrm{SciBERT}}]$$

(5)

$$p=\mathrm{sigmoid}(\bm{W}^{\mathrm{T}}\bm{\mathrm{MLP}}(\bm{h})+b)$$

(6)

where $\bm{W}$ is the weight vector, $b$ is the bias, and $\bm{\mathrm{MLP}}(\cdot)$ represents the operation of multiple layer perceptron shown in Figure 1. Here $p$ is the predicted probability.

Finally, our model is trained with cross entropy loss with regularization. The loss function is defined as

$$\mathcal{\bm{L}}(\theta)=-\sum_{\mathcal{D}}{\left(y\log(p)+(1-y)\log(1-p)\right)}+\lambda{\|\bm{\theta}\|}_{2}^{2}$$

(7)

where $y$ is the ground truth, $\bm{\theta}$ is the parameter set of the proposed model, $\lambda$ is the regularizer parameter, and $\mathcal{D}$ is the training dataset.

The SciAD ¹¹1We won second place in the acronym disambiguation competition. https://sites.google.com/view/sdu-aaai21/shared-task dataset created from 6,786 English scientific papers aims to find the correct meaning of an ambiguous acronym in a given sentence (Veyseh et al. 2020b). It contains 62,441 sentences and a dictionary of 732 ambiguous acronyms. More statistical information is shown in Table 2. Besides, a toy sample of the SciAD dataset is shown in Table 1. The input is a sentence with an ambiguous acronym and a dictionary with possible expansions (i.e., long forms) of the acronym. In this example, the ambiguous acronym “CNN” in the input sentence is shown in boldface and the expected prediction for its correct meaning is “Convolutional Neural Network”. In addition, Figures 2 and 3 demonstrate more statistics of SciAD dataset (Veyseh et al. 2020b). More specifically, Figure 2 shows the distribution of the number of acronyms based on the number of long forms per acronym, and the distribution of the number of samples based on the number of long form per acronym is shown in Figure 3.

As mentioned previously, we convert the original SciAD dataset into a binary classification dataset named SciAD_BI during modeling. For a sentence $\bm{s}$ with acronym $w_{p}$, $y=1$ if a long form $\bm{l}$ is true for $w_{p}$, while $y=0$ for other false candidate long forms of $w_{p}$. Specifically, to alleviate the imbalance problem during training, we upsample each positive sample to equal the number of candidate long forms of its acronym. More statistics of SciAD_BI is shown in Table 4. We finally evaluate performances on SciAD’s test dataset.

We compare with several state-of-the-art and representative methods including Non-deep learning methods and Deep learning methods to verify the effectiveness of our proposed method.

To evaluate the performance of different methods, three popular metrics are adopted, namely Macro Precision, Macro Recall and Macro F1. The definitions are as follows:

$$\bm{\mathrm{Precision}}_{\mathrm{MACRO}}=\frac{\sum_{i=1}^{n}\mathrm{Precision}_{i}}{n}$$

(8)

$$\bm{\mathrm{Recall}}_{\mathrm{MACRO}}=\frac{\sum_{i=1}^{n}\mathrm{Recall}_{i}}{n}$$

(9)

$$\bm{\mathrm{F1}}_{\mathrm{MACRO}}=\frac{2\times\mathrm{Precision}_{\mathrm{MACRO}}\times\mathrm{Recall}_{\mathrm{MACRO}}}{\mathrm{Precision}_{\mathrm{MACRO}}+\mathrm{Recall}_{\mathrm{MACRO}}}$$

(10)

where $n$ is the number of total classes, $\mathrm{Precision}_{i}$ and $\mathrm{Recall}_{i}$ represent the precision and recall of class $i$ respectively. The higher $\mathrm{Precision}_{\mathrm{MACRO}}$, $\mathrm{Recall}_{\mathrm{MACRO}}$ and $\mathrm{F1}_{\mathrm{MACRO}}$ indicate the higher performance of approaches.

For models ADE, NOA, UAD, DECBAE, BEM, and GAD, please refer to Veyseh et al. for more implementation information. We implement the proposed model based on Pytorch (Paszke et al. 2019) and Transformers (Wolf et al. 2020). For models BERT, RoBERTa, and SciBERT, we fine-tune them on dataset based on their popular pretrained models. The implementation details of these models are shown in Table 3. Moreover, the information distillation components of our model are the same as model RoBERTa and SciBERT respectively. And we simply adopt three MLP layers for integration simultaneously. As mentioned previously, in the testing phase, the long form with the highest prediction probability in the candidate long form set of a sentence would be chosen as its final result. In addition, we use two V100 GPUs with 12 cores to complete all these experiments.

Table 5 demonstrates the main results of all compared methods ²²2We assume that both Veyseh et al. and this task have the same distribution of dataset due to the randomly dividing by the same ratio, making all these methods comparable. on the dataset. The major findings from the experimental results can be summarized as follows:

First, GAD achieves a better result than methods such as ADE, NOA, UAD, BEM, and DECBAE, showing the importance of syntactic structure for the acronym disambiguation task. But it still far worse than pretraining-based models like BERT and RoBERTa. Second, between the two general-domain models, RoBERTa gets better performance than BERT, indicating the advantage of more fine-grained encoding. Moreover, SciBERT is more advanced than the domain agnostic methods, i.e., BERT and RoBERTa, with about 2.26% and 1.03% increased macro F1 respectively, showing the importance of the scientific domain pretraining for this task. Furthermore, we can clearly observe that our hdBERT model outperforms all the baselines by a large margin. Its macro F1, with the reported value of 93.73%, is about 1.88% and 0.84% higher than state-of-the-art RoBERTa and SciBERT respectively. And its loss curve falls faster and converges lower than the two pretrained methods on the development dataset, as shown in Figure 4. These observations demonstrate that it is effective to model both fine-grained domain agnostic and high-level domain specific knowledge simultaneously.

However, despite the significant improvements among these approaches, performances of all models are still not as effective as humans on the dataset, especially on macro recall and macro F1, thus providing many further research opportunities for this scenario.

We further focus on studying both success and failure cases of pretraining-based models to provide more insight into acronym disambiguation. Specifically, for success case of our model in which RoBERTa and SciBERT fail, e.g., “Each SP within an SM shares an instruction unit, dedicated to the management of the instruction flow of the threads.” (DEV-6156), the true long form of “SM” is “Streaming Multiprocessors”. While both RoBERTa and SciBERT output “Shared Memory”, which may often appear in deep learning publications. It might benefit from the additional integration modeling of two different information from RoBERTa and SciBERT. However, all the three models fail in this example: “In the first stage, we train the SPM, and extract the FL and FR.” (DEV-4604) with the wrong prediction “Federated Learning” for “FL”. The true long form of “FL” is “Fixated Locations”. We guess that all models pay too much attention to “Federated Learning”, a hot phrase nowadays, and ignore the subtle information among the sentence and its different candidate long forms. It also indicates the necessity of more advanced models for this task.

As mentioned previously and shown in Table 5, all the current models are still less effective than humans in this scenario. There are still many samples that all models fail in. Some further research opportunities on this dataset are discussed in this section. First, as shown in Table 6, there are some noise data, i.e., conflicted annotation, in the SciAD dataset. For example, the acronym “RF” in boldface in sentence “Just like RF, QRF is a set of binary regression trees.” gets two different long form “Regression Forest” (TR-43200) and “Regression Function” (TR-49535) respectively. It will be some negative impacts on modeling to some extent. Furthermore, to a certain extent, samples constructed from the same sentence with different long forms are independent during our training stage. It might lose more subtle information among them. Therefore, recent methods such as self-training (Peng et al. 2019; Chi et al. 2020), adversarial learning (Goodfellow, Shlens, and Szegedy 2015; Miyato, Dai, and Goodfellow 2017; Zhu et al. 2021), and contrastive learning (Hadsell, Chopra, and LeCun 2006) are worth studying to further improve the performance.