ConstGCN: Constrained Transmission-based Graph Convolutional Networks for Document-level Relation Extraction

Due to the agnostic nature of relational edges, we need to broadcast the information of entities along with all relation spaces. Therefore, we need to measure the probability that there is a specific relationship between any two nodes to control the information flow.

Specifically, a simplified NCE objective Gutmann and Hyvärinen (2012); Mikolov et al. (2013) can be define as:

where it differentiate positive sample pairs $(v_{i},v_{j})$ from noisy sample pairs $(v_{k}^{\prime},v_{i})$ drawing from the noise distribution $p_{n}$ by means of logistic regression, and $\phi$ is a specific measure function. Inspired by the KGE ideas, we naturally use the logistic term of NCE by replacing the measure function $\phi$ as the KGE scoring functions to calculate the transmitting score from entity $e_{i}$ to entity $e_{j}$ through relation space $r_{k}$:

where the score indicates the probability that there is a relational edge $r_{k}$ from $e_{i}$ to $e_{j}$.

At $t+1$-th layer, by performing the computation of equation (6) for all entity pairs at each relation space, we obtain the matrices of transmitting scores within all relation spaces

where $\mathcal{A}^{(t)}\in\mathbb{R}^{|\mathcal{R}|\times|\mathcal{E}|\times|\mathcal{E}|}$ refer to the tensor of transmitting scores corresponding to all relation spaces at $t$-th convolution layer.

Note that the equation (6) can be used as an objective for KGE learning, and thus it can be used to optimize the representations of entities and relations with a specific KGE approach simultaneously.

For a heterogeneous graph consisting of entity nodes and relational edges in a document, an entity has multifaceted structural semantic information based on its neighbors of particular relations, and such multifaceted information is difficult to model by the hindered message passing on binary associative edges. We propose to update each entity representation by transmitting the representations of its relation-specific neighbors along with the relation spaces to enhance the multifaceted structural information.

Due to the agnostic nature of relation edges in the real world document, we further constrain the propagation with the transmitting scores learned from the previous step to maintain the original relation-aware structure. For each entity, the update of entity representation is defined as $\mathbf{e}^{(t+1)}_{i}=\sum_{k=1}^{|\mathcal{R}|}\sum_{j=1}^{|\mathcal{E}|}f_{r}(\mathbf{e}^{(t)}_{j},\mathbf{r}^{(t)}_{k},\mathbf{e}^{(t)}_{i})(\mathbf{e}^{(t)}_{j}\oplus\mathbf{r}^{(t)}_{k})$, where the operation $\oplus$ denote the transmitting operation contingent upon specific KGE methods introduced above. For the scale of all entities, we can rewrite this comprehensive transmission in an elegant way by using tensor multiplication, and promote computational efficiency:

where $\mathbf{E}^{(t+1)}_{k}\in\mathbb{R}^{|\mathcal{E}|\times d}$ is the entity representations corresponding to the $k$-th relation space, $\tilde{\mathcal{A}}^{(t)}$ refers to the transposed transmitting scores that transpose the matrices consisting of the values of the second and third dimensions of $\mathcal{A}^{(t)}$, and $\mathbf{M}_{k}^{(t)}=[\mathbf{r}^{(t)}_{k},...,\mathbf{r}^{(t)}_{k}]^{\top}\in\mathbb{R}^{|\mathcal{E}|\times d}$ is the matrix composed of $|\mathcal{E}|$ identical vectors $\mathbf{r}^{(t)}_{k}$. The operation $\otimes:(\mathbb{R}^{|\mathcal{R}|\times|\mathcal{E}|\times|\mathcal{E}|},\mathbb{R}^{|\mathcal{E}|\times d})\rightarrow\mathbb{R}^{|\mathcal{R}|\times|\mathcal{E}|\times d}$ refers to the tensor multiplication. Similar to the pooling functions that aggregate feature maps along channel dimension in the computer vision, the function $f_{pool}(\mathcal{X}):\mathbb{R}^{d_{1}\times d_{2}\times d_{3}}\rightarrow\mathbb{R}^{d_{2}\times d_{3}}$ aggregates the matrices comprised of the values with $d_{2}$ and $d_{3}$ dimensions of $\mathcal{X}$ along the $d_{1}$ dimension. There are multiple pooling operations can be used, such as $f_{pool}^{mean}$, $f_{pool}^{max}$ and $f_{pool}^{sum}$ for mean, max, sum.

We introduce an attentive pooling function $f^{att}_{pool}$ that aggregates the representations of entities according to their importance in each relation space:

where $\mathbf{z}_{k}^{(t)}\in\mathbb{R}^{|\mathcal{E}|}$ refers to the $k$-th row of $\mathbf{Z}^{(t)}$ indicating the importance of entities with respect to the relation space $r_{k}$.

By performing $T$ layers of the graph convolution, the representations of entities $\mathbf{E}^{(T)}$ and relations $\mathbf{R}^{(T)}$ that modeled the multi-relational spatial information are obtained.

The proposed methods on a widely-used human-annotated dataset, DocRED Yao et al. (2019), which is built based on Wikipedia and Wikidata, with 5,053 documents, 132,375 entities, 56,354 relational facts, and 96 relation types. More than 40.7% relations can only be extracted across sentences, and 61.1% relations require inference skills such as logical inference. We follow the standard split of the dataset, i.e., 3,053 documents for training, 1,000 for development and 1,000 for test.

We compare ConstGCN with sequential methods that adopt sequential encoders as the main architectures, including CNN/LSTMYao et al. (2019), BERT-2Phase Wang et al. (2019), HIN-GloVe/HIN-BERT Tang et al. (2020), CorefBERT Ye et al. (2020), MIUK Li et al. (2021) and ATLOP Zhou et al. (2021), and graphical methods leveraging GNNs as the backbone, including GCNN Sahu et al. (2019), GAT Veličković et al. (2017), AGGCN Guo et al. (2019), EoG Christopoulou et al. (2019), GAIN Zeng et al. (2020), LSR Nan et al. (2020) and HeterGSAN Xu et al. (2021).

For the foundational components, we use cased BERT_base or RoBERTa_large Liu et al. (2019) as the encoder to validate the efficiency for all BERT-based methods. For ConstGCN, we empirically set the number of relation basis vectors $\mathcal{B}=56$, and the pooling function as att tuned on the dev set. We investigate three implementations with different transmitting operations, ConstGCN(TransE), ConstGCN(DistMult) and ConstGCN(ComplEx), where the fixed margin of ConstGCN(TransE) is set to $\gamma=20.0$. The number of graph convolutional layers are set to 2, 2, and 1 for the three implementations, respectively, and the negative sampling proportion in transmitting score’s learning is set to $1/|\mathcal{T}_{N}^{q}|=1/40$ consistently for all implementations. All hyperparameters are tuned based on grid search with the performance on dev set, and more training details are available in Appendix A.

Figure 3 shows the learning curves of classification performance and transmission costs for all three implementations on the development set during the training. As seen, the $F1$ scores strictly improve with the decreasing of transmitting costs for all three models, and the model achieves the optimal classification performances when the transmission costs converge. This shows that the informative entity representations learned by optimizing the constrained transmission-based graph convolution can lead to significant improvements in classification performance. We also notice that the model using ComplEx transmitting operation performs the best in terms of convergence speed and optimal values compared to the other models, and it suggests that the model learns effective representations especially in the complex space. The visualization of the transmitting scores learned between entities in a specific document are shown in Appendix B.

We further investigate the effectiveness of ConstGCN on documents of different relation-aware complexities to demonstrate its applicability. As the third sub-figure 3 shows, we split the development set of DocRED into 6 disjoint subsets by the number of relations types, and evaluate models trained with or without ConstGCN on each subset. Overall for both models, we found that their F1 performances tend to decrease when the number of relation types keeps growing. However, when the number of relation types is larger than 10, the model w/ ConstGCN consistently exhibit significant better performance compared to the model w/o ConstGCN. This result indicates that the broadcasting of knowledge-based transmission in all relation spaces can learn relation-aware structural information and is more applicable to complex environment. Further, we can assume that when the complexity of the document exceeds that of DocRED, we can still get good performance by adjusting the transmitting operation in ConstGCN.