Common Sense Knowledge Learning for Open Vocabulary Neural Reasoning: A First View into Chronic Disease Literature

The scientific community in multiple areas is now proactive in generating knowledge aimed at addressing the needs to improve public health and quality of life of the world population. Evidently, this knowledge is generated with the main aim of being persistent and reusable [1]. Ironically, its evolution and volume makes it even more difficult to access, share and interpret it. We believe that generalizing access to knowledge of scientific domains relevant to societal challenges using Natural Language Processing (NLP) and Deep Learning (DL) methods is a promising approach to advance the aforementioned problem.

Nowadays, Knowledge Bases (KBs) are important pieces of technology that organize knowledge in a way that scientists from different areas can navigate the structure and meaning of the documented empiricism. KBs are commonly applied in organizing knowledge generated for different levels of specificity in different areas of Health Care and Biomedicine [2]. On the one hand, there is a well known set of NLP techniques, called Information Extraction (IE, [3, 4]), that have been applied to automatically generate KBs and Knowledge Graphs (KGs). On the other hand, Neural Network-Based Language Models (NLMs) are becoming a very useful set of methods that can take these KBs as training data and use them to perform semantic reasoning tasks such as Common Sense Reasoning (CSR) and Knowledge Base Completion (KBC) [5, 6].

The goal of the trained NLM in the KBC task is to infer the missing parts of semantic structures, i.e. the NLM learns from training samples of the form {Subject, Predicate, Object} (i.e. SPO triples, e.g. {music, is a, form of communication}). This includes the training triples edited by the model. For instance, given an incomplete SPO triple, i.e. the subject and the predicate phrases, {music, is a, __}, the goal of the trained model is to infer the missing object phrase: form of communication. open vocabulary KBs (openKBs, for short) are generalizations of KBs, whose entries (SPO triples) are built from natural language phrases, rather than from fixed inventories of entities and relations.

The mentioned models and reasoning tasks compose a well-known methodology for knowledge processing. Nonetheless, its effectiveness is currently limited by (i) the method used to build the KB, and (ii) the generalization ability of the NLM in reasoning. IE methods, in particular Relation Extraction (RE), are the building blocks of KBs and KGs. However, they are currently limited by their low recall [7], which is important because it directly affects the expressiveness and contextual information (very controlled syntactic structure and vocabulary) provided by the resulting KB [8, 9, 10, 11, 12]. The low recall of simple RE methods is due to the fact that the subject and object of a triple are limited to a predefined set of named entities, which are very special cases of noun phrases, and the predicate to a small set of predefined attributional or defining verbs.

NLMs based on Recurrent Neural Networks (RNNs) have shown limitations in reasoning tasks [13, 14], particularly encoder-decoder architectures [15, 16]. These models have been improved with contextual alignment (i.e. attention) between source and target sequences [17, 18], but recent developments have shown even better improvements, i.e. the Transformer encoder-decoder architecture [19], which is a self-attention-based Neural Language Model. From these developments, pre-trained Transformer-based NLMs are becoming ubiquitous in multiple IE and reasoning tasks [20, 21, 22], however the need to prune and tweak them to make more efficient and specialized applications is also becoming apparent [23, 24, 25, 26]. Moreover, they only consider general purpose vocabulary, which makes difficult their transferability to open domains. Openness for new domains allows selecting an arbitrary specific target (application) domain.

In this work, we address the limited flexibility of the current reasoning methodology based on KBs. We apply the obtained improvements to a reasoning task in the context of the research literature on Chronic Diseases (also known as Non-Communicable Diseases, NCDs). For this, we use an open vocabulary chronic disease KB, that we call OpenNCDKB, built using a state-of-the-art OpenIE on abstracts of articles retrieved from PubMed. Consequently, we trained a Transformer-based NLM to solve a set of similar source reasoning tasks (KBC) that involve general purpose vocabulary: the ConceptNet knowledge Graph and an openKB we built from multiple sources of Open Information Extractions (OpenIE [27, 28]). Using the models obtained for the source tasks, we observed their reasoning performance and capabilities on our OpenNCDKB. Our results showed that a significant number of SPO structures used as ground truth were predicted by the NLM. Other predictions were incorrect from the point of view of loss function and accuracy. However, we note that, rather than being due to ground-truth errors, from the point of view of language meaning, the view of these metrics could be biased towards accuracy (which is not peculiar to natural language), although this was effective in learning. We subsequently confirmed this using an STS-based hypothesis test of the predicted and true object sentence embeddings in the source and target tasks [29, 30]. This helped us to analyze by inspection the semantic regularities of some predictions taken at random. The apparent robustness of these regularities showed the potential benefits of the new knowledge inferred by the model from our OpenNCDKB, which suggested that such knowledge (not in the training data) needs to be validated through further research, rather than being directly discarded as a set of senseless predictions.

The rest of this paper is organized as follows. In Section 2 we expose our methodology. In Section 3 we show the theoretical background. In Section 4 we describe the data and the experimental setup. In Section 5 we report our results, together with the corresponding discussion. In Section 6 we describe prior work related to ours, and in Section 7 we present our conclusions.

DL and NLP researchers recently tested Transformer-Based NLMs in general purpose CSR tasks [31, 32]. In sight of this progress, our research uses a methodological approach that can be especially useful for the semantic analysis of documents dealing with arbitrary but specialized topics, such as open domain scientific literature. Figure 1 shows the general methodology used in this work.

First, we trained Transformer-based encoder-decoder NLMs to solve combinations of two similar source reasoning tasks that involve general purpose domains: the ConceptNet KG in its Common Sense KB version, and an openKB we built from multiple sources using OpenIE. The target specialized domains were included in an open vocabulary Chronic Disease KB (OpenNCDKB) we built using a state-of-the-art OpenIE method that extracts open vocabulary triples from a dataset of paper abstracts retrieved from the PubMed.

Although this open vocabulary approach can add complexity to the modeling of semantic relationships (and therefore to the learning problem), it also adds expressiveness to the resulting KBs. Such expressiveness can improve the contextual informativeness of semantic structures and thus allow the NLM to discriminate useful patterns from those that are not for the target reasoning task [33]. To show the effectiveness of our approach in the target reasoning task from multiple points of view, we evaluated and analyzed the results in terms of:

1. 1.

   Performance metrics in source tasks. Accuracy considers the reasoning quality as an exact prediction, with respect to the predicted and ground truth tokens. Cross Entropy quantify the degree of dissimilarity between the probability distribution predicted by the model and the ground truth one. These are effective at learning time because the inner representations of the NLM acquire knowledge on what are the specific words (the object phrase) that probably should be next to the input ones (the subject and predicate).

2. 2.

   STS-based hipothesis testing. When it comes to reasoning, specificity of performance metrics may not be characteristic of natural language, especially from the point of view of the open vocabulary inherent in natural language and semantic change (probably due to logical inference and/or synonymy). To account for these points of view, we performed hypothesis testing based on Semantic Textual Similarity (STS) in source and target tasks, which measures reasoning quality in the sense of semantic relatedness [34, 29]. We compared to the distribution of STS measurements between predicted and shuffled ground truth object phrases (a random baseline that simulates perturbation of the actual correspondence between subject-predicate and object).

3. 3.

   Inspection of the inferences in the source and target tasks, which has the aim of showing the semantic regularities learned by the NLMs and how it holds from source to target tasks.

Note that from the point of view of the ConceptNet KB as a source task, this has been mostly compiled by hand, and therefore, the task represents a distantly supervised learning problem. The target task represented by the OpenNCDKB was built unsupervised, so this task represents the target domain of a transfer learning problem.

Regarding systems with IE, and semantic reasoning capabilities, we found common traits shared with the seminal work in Chronic Disease KGs [48]. The authors proposed a data model to organize and integrate the knowledge extracted from text into graphs (ontologies, in fact), and a set of rules to perform reasoning via first-order predicate logic over a predefined dictionary of entities and relations [49]. More recently, [50] proposed association rule learning for relation extraction for KG construction, and neural network-based graph embedding for entity clustering from EMRs. In [51], the authors constructed a KG of gene-disease interactions from the literature on co-morbid diseases. They predicted new interactions using embeddings obtained from a tensor decomposition method. The authors of [52] proposed a KG of drug combinations used to treat cancer, which was built from OpenIE triples filtered using different thesaurus [53, 54]. The drug combinations were inferred directly from the co-occurrence of different individual drugs with fixed predicate and disease. The authors created their resource from the conclusions of clinical trial reports and clinical practice guidelines related with antineoplastic agents.

An EMR-based KG was used as part of a feature selection method for a support vector machine to successfully diagnose chronic obstructive pulmonary disease [55]. Deep Learning has been used in recent work to predict heart failure risks [56]. The authors used a medical KG to guide the attentional mechanism of a Recurrent Neural Network trained with event sequences extracted from EMRs. Previously the authors of [57] also predicted disease risk, but for a broader spectrum of NCDs, and using Convolutional Neural Networks in a KG of EMR events. Medical entity disambiguation is an NLP task aimed at normalizing KG entity nodes, and the authors of [58] approached this problem as one of classification using Graph Neural Network. Overall, multiple classical NLP methods have been applied to biomedical KGs, including biomedical KG forecasting from the point of view of link prediction (also known as literature-based discovery) [59].

We trained different Transformer-based encoder-decoder NLMs to perform general-purpose reasoning as source task, including one-block and two-block NLMs. On the one hand, the performance results provided that one-block models showed the best metrics (accuracy and loss), however the divergence between train and test measurements was interpreted as overfitting at first glance. The two-block models, on the other hand, showed much more stable and less divergent (although decreasing) performance measurements, which was interpreted as higher generalization at first glance.

Nevertheless, hypothesis testing based on STS revealed with high confidence, on the one hand, that one-block models showed high generalization in source tasks, and from source to target tasks. These models managed to successfully distinguish themselves from simulated adversarial perturbation (the random baseline) without even learning directly from it ($p\ll 0.001$). On the other hand, two-block models ended by memorizing the source task, thus generating nearly random object phrases in the target task (only one two-block model barely reached $p<0.05$). This also confirmed that the NLMs’ reasoning generalization may be manifested in semantic displacements and logical implications (as well as the abstraction and simplification of meanings), rather than in the ground truth distribution of output tokens.

We analyzed by inspection the semantic regularities shown by NLMs during source tasks and the NCD-related target task. The main behaviors observed as transferred were: $(i)$ the model generally takes concrete subjects and predicates, and associates with them the (semantically) most probable object, i.e. it learned semantic agreement; $(ii)$ the model “objectifies” and/or simplifies those meanings that could be considered subjective and/or complex (i.e., in terms of generalization, it is more likely to construct simpler than abstract and/or complex ideas). The two-block NLMs failed to acquire these skills, even the one that barely reached $p<0.05$ in the STS-based hypothesis testing.

Finally, our results suggest with significance that the predictions obtained by the proposed method can help answer research questions whose answers give rise to potential new knowledge, as we demonstrated for NCDs.

We thank to PRODEP-SEP (project number UTMIX-PTC-069) for the support given. Also thanks to Dr. J. Anibal Arias Aguilar (Postgraduate Studies Division, UTM) for giving access to his computing nodes.