TripleRE: Knowledge Graph Embeddings via Tripled Relation Vectors

Knowledge graphs store huge amounts of structured data in the form of triples, with projects such as WordNet Miller (1995), Freebase Bollacker et al. (2008), YAGO Suchanek et al. (2007) and DBpedia Lehmann et al. (2015). They have gained widespread attraction from their successful use in tasks such as question answering Bordes et al. (2014), semantic parsing Berant et al. (2013), and named entity disambiguation Zheng et al. (2012) and so on.

In recent years, knowledge representation is one of the hottest topics in natural language processing as it can be applied to various tasks based on knowledge graphs, including semantic parsing, named entity disambiguation, question answering, etc. A knowledge graph contains finite number of facts represented by triplets. A triplet (h, r, t) contains three components, where h, r, t represent head entities, relations and tail entities, respectively. Knowledge representation is one of the most essential parts in research of knowledge graphs. The mission for knowledge representation is to leverage an appropriate mechanism to represent three components of triplets in numeric vectors. An ideal mechanism for representing knowledge is capable to map the components of triplets to the pre-defined vector space, such as Euclidean space, along with accuracy and some degrees of uncertainty.

The process of knowledge representation learning could be simply summarized into two phases: the encoding phase and the decoding phase. (i) The encoding phase: The purpose of the encoding phase is to set the embeddings of three components. (ii) The decoding phase: This phase usually includes scoring functions, which are equations include three components, to measure the plausibility of triplets. In other words, the decoding phase is expected to discriminate factual triplets (positive samples) and noisy triplets (negative samples) which are generated according to a selected sampling strategy.

Although many distinctive signs of progress have been achieved in recent years, from our perspectives, one of the most inspired attempts is the blending of information from the structure itself and the logical information embedded in the structure. Knowledge graphs can be viewed as a special kind of directed graphs, where nodes represent entities and edges represent relations. Based on the properties of directed graphs, structural information can be extracted by the positional information of entities in knowledge graphs. The positional information of entities generally represents centrality of the entity nodes and the distance to other nodes. Furthermore, the logical information embedded in the structure can be extracted by the pairs of connected nodes, where the connections indicate the correlation of entities, and edges and relations have equivalence.

Previously, most researches leveraged only the information from the structure of triplets and the logical information embedded in the structure in triplets to learn the knowledge representation. TransE Bordes et al. (2013) is the fundamental work of translation-based models. Inspired by TransE Bordes et al. (2013), many variants, including TransHWang et al. (2014), TransRLin et al. (2015), TransAt Qian et al. (2018), RotatE Sun et al. (2019) and PairRE Chao et al. (2020), handled some issues of TransEBordes et al. (2013), and improved the representing capacities. Translation-based models are reputational in semantics translation. However, all those approaches do not leverage any diversified information of graph structures in either the encoding phase or the decoding phase.

Structural information can be leveraged in knowledge representation by two different strategies. The first strategy is to extract structural information in the encoding phase. NodePiece Galkin et al. (2021) introduces an anchor-based approach to fix the size of entity vocabulary. Every entity can be represented by combining of a set of anchors – selected entities. The second strategy is to extract structural information in both the encoding scheme and the decoding scheme. Approaches under this strategy are highly dependent on sub-graph sampling, including GraIL Teru et al. (2020) and RedGNN Zhang and Yao , are usually based on Graph Neural Networks (GNN) Kipf and Welling (2016), and follow message passing scheme Gilmer et al. (2017). However, in addition to training complexity, the interpretability of inference is also unsatisfactory. Hence, from our perspectives, the first strategy is more valuable as it reduces the parametric size, and solves the problems of scalability.

Our approach is inspired by NodePiece Galkin et al. (2021), TransAt Qian et al. (2018) and pairRE Chao et al. (2020). Our approach chooses to take advantage of NodePiece Galkin et al. (2021) in the encoding phase as it can significantly reduce the parameters of models. Then, we introduce a novel decoding method, which is inspired by the sub-relations from PairRE Chao et al. (2020) and the intuitions from TransAt Qian et al. (2018). TransAt Qian et al. (2018) believes when humans consider triple link prediction tasks, they usually make decisions based on context, not just single node and relation. We argue that the scoring function of PairRE Chao et al. (2020) is not complete, and do not follow the hypothesis of translation-based models: $\textbf{h}+\textbf{r}\approx\textbf{t}$. Therefore, we introduce a new decoding phase (scoring function) by adding one more sub-relation term. Since our approach uses three sub-relation terms to build the semantics of entire relations, we followed the naming logic of PairRE Chao et al. (2020) and named our method TripleRE.

Our approach achieved a competitive performance in large scale datasets - ogbl-wikikg2 and ogbl-biokg Hu et al. (2020). On the benchmark ogbl-wikikg2, our method achieved the best performance. Compared with other methods, the amount of parameters is significantly reduced. In addition, experiments show that our work also has better results on small datasets such as FB15k Bordes et al. (2013) and FB15k237 Toutanova and Chen (2015).

This section is going to review several approaches to knowledge representation.

1) Translation distance models: Based on the translation invariance of the word embedding they found, TransE Bordes et al. (2013) expresses the relation as the translation of the entity vectors in $h+r\approx t$, which is the ground-breaking work in the translation distance model. TransE Bordes et al. (2013) has the advantages of simplicity and scalability, but it cannot model complex relations and multiple relations patterns, such as symmetric relations. The core of a translation distance model is the scoring function. A better scoring function will have a better performance in modeling complex relations such as 1-N, N-1, N-N and relation patterns, such as symmetric/non-relations, inverse relations, combination relations, and sub-relations. TransX series (TransE Bordes et al. (2013), TransH Wang et al. (2014), TransD Ji et al. (2015), TransR Lin et al. (2015)) have made up for the shortcomings of TransE’s inability to express complex relationships and symmetrical relationships, but they still have many flaws, including the complexity of models and the insufficient expression of the relations. TransAt Qian et al. (2018) describes the implementation of an attention mechanism for prediction by focusing on only a part of the entity’s dimensions. RotatE Sun et al. (2019) takes inspiration from Euler’s formula and uses the rotation of the vector to express relations. At the same time, it uses complex embedding to model inverse relations. PairRE Chao et al. (2020) uses two-stage vectors to express the relations. It is capable of modeling more complex relations, such as sub-relations.

2) Bilinear models: The bilinear model is also known as the semantic matching model. For most bilinear models, their core is also the scoring function. Their score functions are based on similarity scores to measure the possibility of the triplets by matching the latent semantics of entities and relations in the embedding vector space. RESCAL Nickel et al. (2011) uses vectors to represent the embedding of the head entity and the tail entity. The relation is expressed as a matrix to model the interaction of the triple. DistMult Yang et al. (2014) imposes constraints on the relation matrix, and simplifies the calculation. ComplEX Trouillon et al. (2016) embeds the head entities, tail entities, and relations into the complex space so that it can better model the antisymmetric relations. HolE Nickel et al. (2016) combines the expressiveness of RESCAL Nickel et al. (2011) with the efficiency of DistMult Yang et al. (2014), and defines an operation called circular correlation to calculate the rationality score of fact triples. This calculation can be regarded as compressing the interaction between paired entities so that more interactive information can be captured with less calculation. Circular correlation does not satisfy the exchange law, and HolE Nickel et al. (2016) can model asymmetric relations. There is also a bilinear model based on AutoML. AutoSF Zhang et al. (2020) uses a determined search algorithm to search the score function with the best performance in the search space. It first learns a set of the converged model parameters on the training set and then searches for a better scoring function on the validation set. Compared to the translation distance models, the bilinear models have shortcomings in encoding relational patterns and leveraging logical information.

3) Neural network-based models: With the development of deep learning and the popularity of neural networks, there are many knowledge representation models based on neural networks. ConvE Dettmers et al. (2018) and R-GCN Schlichtkrull et al. (2018) can show high performance under desirable parameters, but they are also challenging to manipulate and analyze due to the many parameters and difficulty in explain.

4) NodePiece: NodePiece Galkin et al. (2021) has a remarkable ability to access structural information in the knowledge graph. By filtering the anchors and encoding other nodes, the model can learn the relative position and centrality of each node in the graph. Anchor entity nodes and context relations represent each entity node, and a vocabulary is constructed for model training, which dramatically reduces the number of parameters and improves the effectiveness of the model. In addition, NodePiece has excellent cooperation with other models that leverage the logical information in a knowledge graph.

The intuition of our approach is similar to the intuition of TransAt Qian et al. (2018), which tends to focus on a subset of features for link predictions according to the hierarchical routine of human cognition. Suppose there is a triplet $<$ Yao Ming, married with, $?>$, Yao Ming has multiple identities: a retired basketball player, the chairman of the Chinese Basketball Association, a husband, etc. To identify who is Yao Ming’s wife, it is supposed to pay more attention to Yao Ming’s identity on being a husband. Therefore, we need a mechanism to concentrate on Yao Ming’s features related to being a husband, and reduce attention to irrelevant features. TransAt Qian et al. (2018) follows PCA Dunteman (1989); it argues that if a feature is related to a relation r, then the variance of this feature between candidate entities and r should be large, and vice versa. Therefore, TransAt Qian et al. (2018) sets a threshold to determine whether to retain or drop out a feature from the whole feature set. In other words, TransAt Qian et al. (2018) leverages a subset of all features. Inspired by PairRE Chao et al. (2020), every relation is represented as the projections of the node, and entities are mapped to different feature spaces by performing the dot product with the relation, which is equivalent to feature filtering. After mapping entities to a specific feature space with the involvements of relations, the decision space becomes smaller, which could lead to better predictions.

where $r^{H}=r_{h}$, $r^{T}=r_{t}$ and $r^{M}=r_{m}$.

The scoring function of TripleREv1 is similar to the scoring function of PairREChao et al. (2020). However, according to Figure LABEL:fig:1, PairREChao et al. (2020) is essentially a special case of TripleREv1.

TripleREv2: TripleREv2 is a better version optimized based on TripleREv1. We believe that the segments of relations $r_{h}$ and $r_{t}$ play different roles in the decoding phase. For learning better representations of $r_{h}$ and $r_{t}$, TripleREv2 takes advantage of the concept of residual. The original model needs to learn $r^{H}=r_{h}$, $r^{T}=r_{t}$, we made the following changes: $r^{H}=r_{h}+u*e$, $r^{T}=r_{t}+u*e$, The candidate value of $u$ is heuristically chosen, and we took 1, 0.5, 0.25 and 0.125 for u in our experiments, taking the best result as the value for $u$. Now the model can be regarded as using TransEBordes et al. (2013) as the baseline, using $r^{H}$ and $r^{T}$ to learn the transformation of the residuals. Experiments proved that the involvement of residual mechanism does improve the performance. The scoring function of TripleREv2 is as follows:

Two parameters $u$ and $e$ are added to the scoring function, where $u$ is a constant and set default to 1, and $e$ is a unit vector.

TripleRE+NodePiece: NodePiece Galkin et al. (2021) is an anchor-based method for learning a fixed-size vocabulary for any connected multi-relational graph. In NodePiece Galkin et al. (2021), an atomic set consists of anchors and all relation types, which allow a combined number of sequences to be built from a limited vocabulary of atoms. NodePiece Galkin et al. (2021) has several advantages. First, NodePieceGalkin et al. (2021) can significantly reduce the number of parameters; second, NodePiece Galkin et al. (2021) can extract certain sub-graph features; third, NodePiece Galkin et al. (2021) can be used in the encoding phase to connect with other score functions seamlessly. We use NodePiece Galkin et al. (2021) in the encoding phase and TripleRE in the decoding phase.

We followed the NodePiece Galkin et al. (2021) ablation experiment and used three different methods to encode the nodes. First, nodes are represented by K-nearest anchors and the translation part information of TripleRE; second, K-nearest anchors represent nodes, and third, nodes are represented by K-nearest anchors and full relation information.

As can be seen from Tables 3 and  4, TripleRE has significantly improved the performance on the large-scale datasets ogbl-wikikg2 and ogbl-biokg. On ogbl-wikikg2, TripleRE+NodePiece achieved state-of-the-art performance, and parametric size was reduced by more than three times compared with the approach in second place.

From Table 1, we can see that on the small datasets FB15k and WN18, our work has an obvious advantage over TransAt Qian et al. (2018). Table 2 shows the comparison between TripleRE and other models. The results show that our model has obtained the best performance in MR and achieved competitive results in other metrics.

Table 5 shows the various experiments we performed at TripleRE+NodePiece. We conduct three different experiments on encoding node representations for TripleREv1 and TripleREv2. The experiments in Table 5 show that the ”anchor” method works best on TripleRE+NodePiece, both Test MRR and Valid MRR achieve the best performance, and the number of parameters is also less than the lowest model (NodePiece+AutoSF), which means we have achieved the state-of-the-art with the least amount of parameters on wikikg2.

The contributions of this paper can be summarized as follows:

• •

  This paper proposes a novel approach, TripleRE, in the decoding phase of knowledge representation learning. As a translation-based approach, TripleRE is advantageous in efficiency (in both training and inferencing stages) and simplicity.

• •

  In TripleREv2, it utilizes the residual mechanism to optimize knowledge representation. Considering residual mechanism has made a tremendous difference in many fields of deep learning, it is also promising to apply it to knowledge representation.

• •

  TripleRE achieved competitive performance on large-scale datasets: ogbl-wikikg2 and ogbl-biokg.

• •

  The combination of NodePiece and TripleRE achieved state-of-the-art performance in the benchmark ogbl-wikikg2.

The future work can be summarized in two directions. The first direction is to explore the methods to leverage structural information. One of the potential proposals is to improve the anchor selection method of NodePiece. Another direction is to conduct more experiments on other large-scale knowledge graphs to test the capacities of TripleRE and TripleRE+NodePiece further.