Distantly Supervised Relation Extraction via Recursive Hierarchy-Interactive Attention and Entity-Order Perception

Given a sentence, the goal of relation extraction (RE) is to identify the relation between a pair of entities in this sentence. To facilitate this in distant supervision scenario, we split all sentences into multiple entity-pair bags $\{B_{1},B_{2},...\}$ . Each bag $B_{i}$ contains some sentences $\{s_{1},s_{2},...,s_{m}\}$ mentioning the same entity pair $<{e_{h}}_{i},{e_{t}}_{i}>$. Each sentence is a sequence of words, i.e., $s=[w_{1},w_{2},\dots,w_{n}]$, and the maximum length is set to $n$. Besides, we have a set of pre-defined relation classes $\mathcal{R}=\{r_{1},r_{2},...\}$. In this case, the goal of DSRE is to distinguish the relation between two given entities based on an entity-pair bag.

In this part, three kinds of features are taken into account, including word embedding [34], position embedding [19] and entity embedding [35]. For each sentence $s_{j}$ in bag $B=\{s_{1},s_{2},...,s_{m}\}$ ¹¹1“sentence” and “instance” are semantically identical., we remove the index $j$ for brevity in the following narrative.

To fully exploit the taxonomic structure of relations and relation embeddings, an attention network with a recursive structure along the relation hierarchies, called RHIA, is proposed. RHIA consists of several RHI cells. Each cell completes the calculation for one relation level. See Figure 3 for the details of the RHI cell.

For a bag $B=\{s_{1},s_{2},...,s_{m}\}$, the sentence representations $U=\{u_{1},u_{2},...,u_{m}\}$ are derived via PCNNs encoder. For each relation $r\in\mathcal{R}$, its hierarchical chain of relations $\{r^{0},r^{1},...,r^{k}\}$ can be generated, where $r^{0}$ is the root relation node and $r^{k}$ is identical to $r$. Connected by the root, all chains compose the relation hierarchies, a kind of tree-like structure. Then, for the $i$-th level of relation hierarchies ($i\in[1,2,...,k]$), we define a learnable relation embedding matrix $R^{(i)}\in\mathbb{R}^{d_{f}\times N^{i}}$, where $N^{i}$ denotes the number of relations at level $i$.

For a sentence representation $u\in U$ ²²2For convenience, we remove indices in the remaining parts., we aim to augment it with the above relation embeddings for each level, and all levels’ augmented representations are concatenated as the relation-augmented sentence representation $u^{r}$. Then an attention-pooling module is employed to generate the bag-level representation. For $i$-th level, it is assumed that the augmented representation of $i$-th level is determined by the input sentence representation $u$ and the heuristic information about relations $h_{i-1}$ from the higher/previous level .

Based on this, we can build the network in a recursive structure. In details, the sentence-to-relation (sent2rel) attention [15] is employed firstly. The sentence representation $u$ and the heuristic information $h_{i-1}$ are used as the query to calculate attention scores by dot product with the relation embedding matrix $R^{(i)}$, respectively,

where $h_{i-1}\in\mathbb{R}^{d_{f}}$, $softmax(\cdot)$ is an activation function for the last dimension, $c^{(i)}$ and $c^{(i)}_{hier}$ are the relation-aware information.

Since the importance of $u$ and $h_{i-1}$ are different, then, we leverage an element-wise gate mechanism to integrate the relation-aware information $c^{(i)}$ and $c_{hier}^{(i)}$:

where $W^{g1}\in\mathbb{R}^{2d_{f}\times d_{f}}$, $\hat{c}^{(i)}$ is the resulting relation-aware representation corresponding to $i$-th level.

After that, to prevent the information loss of the original representation $u$, we leverage an element-wise gate to inject the information of $\hat{c}^{(i)}$ into $u$. In this process, residual connection [36] and layer normalization [37] are also applied. Then, the augmented representation $u^{(i)}$ at level $i$ is generated,

where $W^{g2}\in\mathbb{R}^{2d_{f}\times d_{f}}$, $MLP(\cdot)$ is a multi-layer perceptron that aims to increase nonlinearity.

Finally, we update $h_{i-1}$ to obtain the current heuristic information about relations $h_{i}$ for the next level’s calculation, which is achieved by merging relation-aware information $\hat{c}^{(i)}$ into $h_{i-1}$ with an element-wise gate,

where $W^{g3}\in\mathbb{R}^{2d_{f}\times d_{f}}$.

During the calculation, $h_{0}$ is randomly initialized. Along the relation hierarchies, all levels’ augmented representations are generated, i.e., $\{u^{(1)},u^{(2)},...,u^{(k)}\}$. Then the relation-augmented sentence representation $u^{r}$ is generated by concatenation operation,

For the bag $B$, all relation-augmented sentence representations can be denoted as $B^{r}=[u_{1}^{r},u_{2}^{r},...,u_{m}^{r}]$. Next, to alleviate wrong labeling problem, we use the attention-pooling [38, 39] to select the correctly labelled sentences from the bag in order to facilitate the generation of an accurate bag-level representation. The attention score for each sentence is calculated from its original representation $u$ and its final hidden state $h_{k}$ (i.e., the $k$-th level’s heuristic information about relations). Then a weighted sum over the bag is employed,

where $H$ is the matrix consisting of the final hidden state $h_{k}$ of all sentences in $B$.

Finally, a softmax classifier based on the MultiLayer Perceptron is used to classify the bag representation $b$,

where $o_{b}\in\mathbb{R}^{|\mathcal{R}|}$, $|\mathcal{R}|$ is the number of pre-defined relations.

To retain more entity-order information, we design a classification sub-task in the multi-task paradigm. Specifically, a MultiLayer Perceptron is employed to bicategorize relation-augmented representations $u^{r}$, i.e., whether entity $e_{h}$ appears before entity $e_{t}$ or not. That is,

To optimize our model, three objectives are introduced: 1) The main objective is bag-level classification and is defined as minimizing cross-entropy loss,

where $D$ is the train set consisting of sentence bags. 2) The hierarchical auxiliary objective is designed to guide RHIA module in choosing appropriate relation embeddings to augment each sentence representation. That is,

where $[\cdot]$ denotes indexing operation. 3) The entity-order perception objective is introduced to take entity-order information into account:

Eventually, these three objectives are integrated into a whole. The final loss function can be represented as:

where $\mu$, $\xi$ and $\delta$ are weight scores of different losses. $||\theta||_{2}^{2}$ is the $L_{2}$ regularizer.

For the initialization of word embeddings, we employ the pre-trained embeddings from Lin et al. [7] ⁵⁵5https://github.com/thunlp/OpenNRE. The relation embedding matrices are initialized randomly as Li et al. [15]. Besides, the dropout strategy [43] is applied to the bag representations to prevent overfitting. During training phase, for optimization, we employ mini-batch SGD [44] with the initial learning rate $\gamma$.

The detailed parameter settings are shown in Table 1. For Entity-Aware Embedding module and PCNN module, the parameters keep consistent with those in previous work [15], i.e., $d_{w},d_{p},n,\lambda,\omega,d_{c}$. The learning rate and dropout rate are also the same as Han et al. [13]. For parameters of three objective functions, all are consistent with Li et al. [15] except for the weight of $L_{ord}$. To better exploit Entity-Order Perception, we choose the weight of $L_{ord}$ (i.e., $\xi$) from the list $[0.4,0.5,0.6,0.7,0.8,0.9,1.0,1.5,2.0,3.0]$. Here we use three-fold cross-validation on the train set. Each value is trained three times since cross-validation, and the best value (i.e., 1) is determined according to the average of AUC.

For the following experiments, we apply the held-out evaluation to evaluate our model. The entire train set is used for the training phase, while the test set is used for evaluation. The results in Section 4.5 are only intended to illustrate the stability of our model, and are not relevant to cross-validation for hyper-parameter selection.

We compare RHIA-EOP with competitive previous baselines that are summarized as follows:

• 1.

  PCNN_ATT:  Lin et al. [7] propose a selective attention among training sentences to mitigate wrong labeling problem, which is the most classical approach in DSRE task.

• 2.

  HNRE:  Han et al. [13] design a hierarchical attention network to enrich bag-level representations, which is the first hierarchical relation extraction baseline.

• 3.

  ToHRE:  Yu et al. [16] introduce a top-down classification strategy and a method to enhance the bag representation in different relation levels.

• 4.

  CoRA:  Li et al. [15] enhance sentence representations in a collaborative way across all relation levels.

• 5.

  HNRE w/ Ent. Emb.: The model of Han et al. [13] directly introduces the Entity-Aware Embedding layer in Eqs. 1-3.

• 6.

  HNRE w/ Aux. Obj.: The model of Han et al. [13] directly introduces the hierarchical auxiliary objective in Eq. 21.

• 7.

  HNRE w/ Ent. Emb. w/ Aux. Obj.: The model of Han et al. [13] introduces both the Entity-Aware Embedding layer in Eqs. 1-3 and the hierarchical auxiliary objective in Eq. 21.

The comparison results of different models are displayed in Table 2, Table 3 and Figure 4 (a). Note that, the results in Table 2 are obtained using the full test set of the NYT dataset. While the results in Table  3 are obtained using the remaining test set with all single-sentence bags removed as [15], because for each single-sentence bag, the result is the same whether one, two or all sentences are retained for evaluation. Here the single-sentence bag means a bag consisting of only one sentence.

It can be observed that RHIA-EOP significantly outperforms many baseline models in all metrics at the same time. Our approach achieves the AUC of 0.56, which outperforms the strong baseline CoRA (0.53) by 0.03. For Max_F1, we improve baseline approaches by at least 2.1%. For Top-N precision metric, we take relatively more unique $N$ values in Table 2, and find that RHIA-EOP sets the best scores on all $N$ values. And in another setting, i.e., randomly retaining one, two or all sentence(s) in each bag, RHIA-EOP achieves the best results despite the randomness of retained sentences in Table 3 (Almost all values exceed 90%). Besides, the curve of RHIA-EOP is significantly higher than PCNN_ATT, HNRE, ToHRE and CoRA. The results of setups HNRE w/ Ent. Emb. and HNRE w/ Aux. Obj. indicate that both the Entity-Aware Embedding layer and the hierarchical auxiliary objective bring substantial performance improvements. However, the results of HNRE w/ Ent. Emb. w/ Aux. Obj. are still worse than our RHIA-EOP because the heuristic effects between relation levels and the entity order information are still not considered.

To measure RHIA-EOP’s ability to handle long-tail relations, we conduct a model comparison on all long-tail relations, and the results are shown in table 4. Here Hits@K is employed to measure this ability, which indicates whether the ground-label’s probability of a bag ranking in the top-K relations. During the calculation, we use macro average regarding different relations. Very inspiringly, RHIA-EOP achieve the highest results at all values of $K$. These results confirm the heuristic influence of higher-level relations on the lower-level ones and the advantage of relation hierarchies.

In addition, we briefly analyze the computational overhead. Both our RHIA-EOP and the CORA of Li et al. [15] employ multiple attention mechanisms. We further introduce the interaction effects between relation levels and entity order information, and the number of parameters grow from 13M to 18M. It is still much smaller compared to the pre-trained language models since the smallest BERT-Base has 110M parameters). The parameter growth is not really huge (i.e., 5M). The required training time dose not increase dramatically, both models can converge within two hours, while the performance gains are relatively huge and substantial. Therefore, although some computational overhead is added, it is worthwhile.

In order to validate the effectiveness of each module in RHIA-EOP and to analyze the sources of performance improvement, we evaluate the following ablation experimental setups:

• 1.

  $\sim$ w/o EOP (RHIA): The Entity-Order Perception subtask is removed. It is equivalent to having only the Recursive Hierarchy-Interactive Attention (RHIA) module.

• 2.

  $\sim$ w/o RHIA (EOP): The module RHIA is replaced by CoRA [15]. It is equivalent to combining CoRA and EOP.

• 3.

  $\sim$ w/o Sent2rel Attention: The sentence-to-relation attention (i.e., Eqs.5-10) is replaced by average pooling.

• 4.

  $\sim$ w/o Attention Pooling: The attention pooling (i.e., Eq.17) is replaced by average pooling.

• 5.

  $\sim$ w/o Gating in Eqs.11-12: Eqs.11-12 are removed. Feature concatenation $[u;\hat{c}^{(i)}]$ is directly passed into MultiLayer Perceptron (MLP).

• 6.

  $\sim$ w/o Aux. Obj. in Eq.21: The hierarchical auxiliary objective $L_{hier}$ is removed (i.e., Eq. 21).

• 7.

  $\sim$ w/ uncased BERT-Base: The Entity-Aware Embedding layer is replaced by uncased BERT-Base model.

The results are shown in the bottom of Table 2 and Table3, and Figure 4 (b). And the evaluation results reflect consistent declines in P@N, Max_F1 and AUC. For the two main modules of RHIA-EOP, in Table 2, compared to RHIA-EOP, both RHIA and EOP drop almost 0.015 on AUC, and these two models decrease by 2.5% and 1.4% on the mean of P@N, respectively. While in Table 3, the performance drop is similar obviously, the mean of P@N of RHIA decreases by 2.2%, 1.5% and 1.1%, respectively. For EOP, the values are 0.8%, 2.4% and 1.4%.

To further analyze whether the performance improvement comes from the new attention mechanism or from the full utilization of relation hierarchies, we set up more experimental settings. The setups $\sim$ w/o Sent2rel Attention and $\sim$ w/o Attention Pooling argue for the importance and validity of the attention mechanism. As for the relation hierarchies, firstly, for each level of the relation hierarchies, we construct the corresponding feature representation for each bag and classify it. This is the hierarchical auxiliary objective (i.e., Eq.21). The setup $\sim$ w/o Aux. Obj. in Eq.21 has demonstrated that not using the relation hierarchies leads to a significant performance drop. Secondly, along the relation hierarchies, relational information is propagated in a recursive form (i.e., RHIA). The setup $\sim$ w/o EOP (RHIA) confirms its effectiveness. To sum up, the relation hierarchies are indeed contributing and the attention mechanism is indeed effective.

Interestingly, despite the Pre-trained Language Models (PLMs) are so powerful, the results achieved by the BERT-based implementation are not satisfactory. Except for the values of AUC and Max_F1, the values of all metrics have decreased. The reason may be that BERT cannot highlight two entities as the Entity-Aware Embedding layer does, i.e., it ignores the importance of the entity-pair itself and the position of entities. After all, the essence of DSRE task lies in the classification of entity pairs.

We reran our model five times to calculate the respective means and standard deviations in terms of AUC, Max_F1 and the mean of P@N. Here for P@N, the randomness of the selected samples when retaining one/two/all sentence(s) in each bag at random leads to large fluctuations in its value, so we only report the mean and standard deviation of “the mean of P@N”, i.e., P@N (Mean). Detailed results can be found in Table 5 and Table 6. These results demonstrate that the performance gains are stable and convincing.

To visualize the heuristic effect between relation hierarchies, two example sentences from NYT are selected. And then we list/analyze the Top-3 attention scores at all relation levels of these two sentences in Table 7. Our RHIA-EOP, compared to CoRA, better handles the highly unbalanced category (i.e., NA), and gives greater scores to NA for the true instances of NA and less scores to NA for other instances. This capability can also improve the recognition of noisy sentences from the side, and alleviate the wrong labeling problem. These findings demonstrate the effectiveness of RHIA-EOP.