Bayesian Networks for Named Entity Prediction in Programming Community Question Answering

As shown in Figure 1, we need to predict the semantic meaningful classes of questions with BN as a multilabel classification problem. For this problem we have textual data, presented as vectors.

More formally, assume we are given two sets of Questions $Q=\langle Q_{1},Q_{2},\dots,Q_{N}\rangle$ and Titles $T=\langle T_{1},T_{2},...,T_{N}\rangle$, where $N$ - is the number of samples in our dataset. For each title $T_{i}\in{T}$ we have $k=25$ dimension vector, $T_{i}=\langle t_{1}^{i},t_{2}^{i},...,t_{k}^{i}\rangle$, where $t_{k}^{i}$ represents the $k_{th}$ entity class of the $i_{th}$ title and $t_{k}^{i}\in{\{0,1\}}$, where $t_{k}^{i}=0$ corresponds to the absence of the $k_{th}$ class entity in title, and $t_{k}^{i}=1$ corresponds to the existence of the $k_{th}$ class entity in title. For the questions it is the same, for each question $Q_{i}\in{Q}$ there are 25 dimension vectors, $Q_{i}=\langle q_{1}^{i},q_{2}^{i},...,q_{k}^{i}\rangle$, where $q_{k}^{i}$ represents the $k_{th}$ entity class of the $i_{th}$ question and $q_{k}^{i}\in\{{0,1\}}$, where $q_{k}^{i}=0$ corresponds to the absence of the $k_{th}$ class entity in question, and $q_{k}^{i}=1$ corresponds to existence of the $k_{th}$ class entity in question. We solve the multi label classification problem by predicting for each $i_{th}$ question its entity classes by $i_{th}$ titles entity classes.

The dataset we use is based on 10% of the Stack Overflow¹¹1https://stackoverflow.com Q&A 3 years ago²²2https://www.kaggle.com/datasets/stackoverflow/stacksample. For the set of questions we apply the following filtering operations: select questions with tag "android", select questions with a length less than 200 words and related to the API Usage category proposed by Stefanie et al. [10]. Moreover, we selected questions without links and images, because information from those types of content is unavailable for Bayesian networks. Thus, we received $N=707$ pairs of title and question $(T_{i},Q_{i})$.

For extracting domain specific entities from text content we used the open source CodeBERT [11] realization trained for the NER problem [12] on Stack Overflow data, since this is the most popular resource for programmers to find answers to questions. The model was tuned to detect 25 entity classes defined by Jeniya et al. [13]. They represent the following classes: ALGORITHM, APPLICATION, CLASS, CODE BLOCK, DATA STRUCTURE, DATA TYPE, DEVICE, ERROR NAME, FILE NAME, FILE TYPE, FUNCTION, HTML XML TAG, KEYBOARD IP, LANGUAGE, LIBRARY, LICENSE, OPERATING SYSTEM, ORGANIZATION, OUTPUT BLOCK, USER INTERFACE ELEMENT, USER NAME, VALUE, VARIABLE, VERSION, WEBSITE. Each class is domain specific and defines context semantics [14].

Declared precision of the open-source model is 0.60³³3https://huggingface.co/mrm8488/codebert-base-finetuned-stackoverflow-ner, hence markup could not be ideal because of model mistakes. Figure 2 shows the Hugging face model inference example. So, annotation models sometimes break a word into several parts and define for each its own class. To smooth out these inaccuracies, we decided to combine parts of words into one entity according to the class of the first defined part. While, entities detected by the model might be ambiguous, testing the key words of sentences mostly results in correct detection. All pairs are vectorized as one-hot encoding, thus each title and question is represented by a k-dimension vector, as there are $k=25$ defined classes.

A Bayesian network is a probabilistic model that encodes a joint probability distribution over a set of variables $V=\{X_{1},…,X_{n}\}$, which, in our case, presents entity classes. We consider only discrete variables. Formally, a Bayesian network $B$ is a pair $\{G,\Theta\}$, where $G$ is a directed acyclic graph called "structure". Each node corresponds to one of the variables from $V$. $\Theta$ is a set of probabilities defined on $G$. It specifies the conditional probability distribution $P(X_{i}|PA_{i})$, where $PA_{i}$ are the parents of the $X_{i}$ variable. The lack of edge between the variables encodes the conditional independence. With BN, it is possible to get the joint probability distribution of all variables, given as:

We used a common train-test split for evaluation. With the dataset described above, we composed the test dataset as random 30% samples of the whole set. The random seed is defined in a specific way whereby classes from the test set are in the train set as well.

Table 1 shows the main evaluation results according to the selected classification metrics. We prefer to accentuate precision, because precision of individual classes is most important for information extraction and context prediction, and wrong class predictions caused context misunderstanding.

Our approach shows better precision metrics than the baseline - CatBoost model [26], 0.56 vs 0.41 macro precision and 0.66 vs 0.58 weighted precision, comparing the BIC score-based network and baseline.

We observe the highest precision in the BIC score-based model, while the K2-based model shows the better recall metric and comparable precision, hence the best network from the F1-score perspective is the K2-based one. As expected, the BIC regularizes the log-likelihood stronger than the BDeu and K2-specific penalty terms. As a result, BDeu and K2-based DAGs detect more relationships that allow to classify more instances of each class correctly, hence the growth of recall.

We see that the Chow-Liu tree-based networks are comparable to other models if Variable Elimination is used as a sampling algorithm. This causes the limitation that each node has exactly one parent, except parents root nodes, and it is non-redundant for the DAG to fit the data. Other sampling algorithms approximating solutions to the inference problem show worse results.

We visualized DAGs from each Bayesian network to see relationships that a BN allows to detect. A BN has causal structure, and we use this property to analyze connections of different semantic entities, describing the context. Figure 4 shows structures learned by described methods. The graphs provide information about the relations between significant parts of the context.

As expected, graphs of K2 (3(b)) and BDeu (4(a)) -based networks detect more relationships and are more complete, in contrast to the BIC (3(a)) -based graph. At each DAG there are semantic links between the same title and question entity classes. The structure of the Chow-Liu trees (4(b)) shows this very well.

Additionally, analysis shows different clusters of semantic entities. This way, DATA STRUCTURE and ALGORITHM separated in each of the four graphs. Furthermore, there is a link between FILE NAME and FILE TYPE, CODE BLOCK and OUTPUT BLOCK. This indicates the logic and validity of BN DAGs structures.

It is noteworthy that the tree structured DAG defines causation from Question ALGORITHM to Title USER NAME and from Title ALGORITHM to Question CLASS without establishing a causal relationship between entities of the same names. More likely, these are outliers, as the NER model is not ideal.

Finally, we compared semantic entities detected by the NER model and predicted by BN based on the K2 metric. Below in Table 2, there are several examples of predictions. Matched to the DAG described above, we observe that overall, predicted entities coincide to target one in the first example. In some cases, BN could not detect semantic instances, as in the second and third rows of Table 2. On Graph (3(b)) VERSION and USER NAME have consequence relations from APPLICATION at both question and title. Similarly, OPERATING SYSTEMS connected to APPLICATION and LANGUAGE in the graph. It is likely that the value of conditional probability was not enough to consider whether these entities would be in the question context.