A Temporal Knowledge Graph Completion Method Based on Balanced Timestamp Distribution

Given an input knowledge graph $\mathcal{G}$ containing time information, in which the fact can be expressed in the form of a quadruple $(h,r,t,[t_{s},t_{e}])$, we aim to complete the missing entities and relations under the condition of time. Similar to the existing model, we represent the head entity as $h$, the tail entity as $t$, and the relation as $r$, where $h,r,t\in\mathcal{R}^{d\times 1}$, $d$ is the dimension of the embedding vector.

The overall architecture of our proposed method will consist of the following three steps:

• $\bullet$

  Balanced distribution of the timestamps is completed by proposed balanced timestamp distribution method in the embedding of the model, which is detailed in Section 3.2.

• $\bullet$

  Based on bt-HyTE and tr-HyTE, the entities and relations are projected onto all the time-specific hyperplanes covered by a particular triple, and then the embedding of entities and relations is learned on all the projected hyperplanes. Learning temporal knowledge graph embedding representation is described in Section 3.3.

• $\bullet$

  To complete entity and relation, we compute the scores for each triple in the test set. And for a triple, the scores of elements in all entity set or relation set are arranged in the ascending order. And then the effectiveness of the temporal KGC is evaluated based on the evaluation metrics. See Section 3.4 for details.

The quadruple $(h,r,t,[t_{s},t_{e}])$ of temporal knowledge graph $\mathcal{G}$ indicates that the triple $(h,r,t)$ is valid in the period $[t_{s},t_{e}]$. Assuming that the timestamps are distributed in some way, a fact is valid on the timestamp spans occupied by its time range $[t_{s},t_{e}]$. So the rationality of the timestamp distribution will directly influence the performance of the method. For distributing $N$ timestamps, the dynamic temporal knowledge graph $\mathcal{G}$ will be partition into $N$ static knowledge graphs $\mathcal{G}_{i}$, where $i\in\{1,2,\cdots,N\}$:$\mathcal{G}=\mathcal{G}_{1}\cup\mathcal{G}_{2}\cup\cdots\mathcal{G}_{t}\cdots\cup\mathcal{G}_{N}$.

We consider dealing with timestamp distributing directly in the embedding of the model to mitigate the effects of the problems described above. We take a period of time as the the finest granularity, and collect the most fine-grained time period for each fact, in which the fact is valid. If a year is used as the most fine-grained time period, the triple $(h,r,t)$ is valid in every year within $[t_{s},t_{e}]$, where $\{t|t_{s}\leq t\leq t_{e},t\ is\ the\ most\ fine\_grained\ time\ slice\}$ , which is included in the set of effective time spans. Then, the number of facts contained in each of the most fine-grained times is counted in the embedding, and the threshold for distributing the timestamps is set, thereby the balanced distribution of the timestamps is done. As shown in Alg. 1, we consider not only the start time and the end time of a fact, but also the process time between them, thus taking into account all the times when the fact is valid. For the $(-,hasWonPrize,Royal\_Medal)$ series triple illustrated above, according to the number of facts of the most fine-grained time slice, the later time will accumulate more fact numbers as time advances, and then when the threshold is reached, it is able to be distributed into $time\_class$, by which we distribute the timestamps. With this newly formed timestamp, the prediction of missing entities or relations prior to the year will be conditionally completed.

In the above temporal knowledge graph $\mathcal{G}$, $\mathcal{T}$ is the set of timestamps distributed in the embedding of the model, then the set of timestamps covered by the time period $[t_{s},t_{e}]$ is expressed as $\{\tau|\tau\in\mathcal{T}\}$. Project entity and relation vectors onto the hyperplane of a particular timestamp $n_{\tau}$ under the balanced distribution of timestamps as shown in Fig. 1,

For a valid fact, there should be $P(h,\tau)+P(r,\tau)\approx P(t,\tau)$, where $\tau$ is a timestamp evenly distributed in the embedding of the model using Alg. 1. For a fact triple in a negative sample, $|P\left(h,\tau\right)+P\left(r,\tau\right)-P\left(t,\tau\right)|$ should be a number much greater than 0. Here, we learn the normal vector of each timestamp hyperplane, and the embedding of projected entities and relations on each timestamp hyperplane.

We performed link prediction, relation prediction tasks, and some fine-grained investigation to prove the effectiveness of proposed bt-HyTE and tr-HyTE. The addition of negative sampling strategy for temporal relations can greatly improve the performance of relation prediction, but it can also inhibit the performance of head and tail entity prediction to a certain extent.

The temporal KGC is entity prediction and relation prediction tasks based on the embedding of entities, relations, and times. At present, the widely accepted task of temporal KGC takes time as the condition of the traditional completion task. We also use time as the condition of entity prediction and relation prediction to further improve the performance of KGC. In this paper, the temporal KGC task is divided into the following subtasks:

• $\bullet$

  Head prediction: for an incomplete fact $(?,r,t,\tau)$, predict the head entity $h$ from the entity set at period $\tau$.

• $\bullet$

  Relation prediction: for an incomplete fact $(h,?,t,\tau)$, predict the relation $r$ from the relation set at period $\tau$.

• $\bullet$

  Tail prediction: for an incomplete fact $(h,r,?,\tau)$, predict the tail entity $t$ from the entity set at period $\tau$.

We use the subgraphs containing rich time information of Wikidata and YAGO knowledge graph datasets to carry out the experiment. The subjects of the experiment included bt-HyTE, tr-HyTE and some baselines.

• $\bullet$

  YAGO11k. This is a temporal knowledge graph from YAGO3 [26]. The extracting method was proposed by HyTE [6]. It consists of 20.5k triples, 10,623 entities, and 10 relations.

• $\bullet$

  Wikidata11k. This is a temporal knowledge graph extracted from Wikidata by the method proposed in HyTE [6]. It contains 27.5k triples, 10623 entities and 24 relations.

As following, we compare these five methods to evaluate the performance of our method:

• $\bullet$

  TransE [8]: TransE is the pioneering work of translation-based models, which embeds entities and relations in the same vector space. It is simple but efficient.

• $\bullet$

  TransH [9]: TransH put forward on the basis of TransE. It projects entities onto the relation-specific hyperplane, and regards the relation on the hyperplane as the translation from the head to the tail.

• $\bullet$

  HyTE [6]: HyTE is a dynamic KGE method. Inspired by TransH’s modeling, it projects entities and relations onto the time-specific hyperplane, and regards the projected relation as the translation of the projected head to the projected tail. It has achieved remarkable results in dealing with the completion of temporal KGs.

• $\bullet$

  bt-HyTE: On the basis of HyTE, we consider directly dealing with unbalanced timestamp distribution in the model embedding, and propose bt-HyTE.

• $\bullet$

  tr-HyTE: On the basis of bt-HyTE, we add negative sampling for the temporal relations, and the model is named tr-HyTE, which effectively improves the performance of relation prediction.

We use the traditional evaluation metrics Mean Rank and HIT@k in the field of KGC. The vectors embedded in the low-dimensional space are first projected onto the time-specific hyperplanes by formula (1), and then the scores are calculated and ordered by formula (2). The rankings of the missing entities are recorded, and the following metrics are calculated:

• $\bullet$

  Mean Rank: For each triple in the test set, scores calculated are sorted from small to large, then the missing entity or relation ranking of each triple is counted. The average value of rankings of all the triples in test set is called Mean Rank. For Mean Rank, a smaller value is better.

• $\bullet$

  HIT@k: For each triple in the test set, scores calculated are also sorted from small to large, and the proportion of triples whose missing entity or relation is ranking in front of k to the total number of the test sets is counted. For HIT@k, a larger value is better.

Our proposed models bt-HyTE and tr-HyTE are better than other compared models in performance, which may be due to the effectiveness of our contributions.

As shown in Table 4 and Table 4, the best results are bolded. bt-HyTE are qualitatively improved compared to not only the static KGC models TransE and TransH, but also the current state-of-the-art temporal knowledge graph embedding model HyTE. Based on Wikidata11k dataset, bt-HyTE is better than HyTE on MR of tail prediction (an increase of 26.01) and relation prediction (an increase of 0.04). As for HIT@k metric, bt-HyTE has an increase of 0.39% on head prediction, 1.76% on tail prediction and 0.24% on relation prediction compared with HyTE. Based on YAGO11k dataset, bt-HyTE is better than HyTE on MR of head prediction, relation prediction and HIT@k of head prediction, tail prediction. There is a slight fluctuation in other metrics, which may be due to the experimental error. It illustrates the effectiveness of balanced timestamps.

Compared with bt-HyTE, tr-HyTE has further improved performance. Based on YAGO11k dataset, tr-HyTE is better than bt-HyTE on MR of head prediction (an increase of 3.88) and relation prediction (an increase of 0.05). In HIT@k metrics, tr-HyTE has an increase of 1.46% on head prediction and 2.73% on relation prediction. Based on Wikidata11k dataset, tr-HyTE is also better than bt-HyTE on MR of relation prediction and HIT@k of head prediction, tail prediction, relation prediction. And tr-HyTE is obviously better than HyTE. It demonstrates the effectiveness of balanced timestamps again and illustrates the effectiveness of negative sampling for temporal relations.

However, we also notice that MR-based head entity prediction on the Wikidata11k dataset and MR-based tail entity prediction on the YAGO11k dataset are not very well. We attribute this to the problem that a certain year contains a very large number of facts. Although our timestamp fact number threshold has been set, the number of facts included in some of the finest-grained time slices is much more than the threshold, which obviously weakens the balance of timestamp distribution.

Through the fine-grained investigation method described in Section 4.4, we select a certain number of specific quadruples based on the test set of the YAGO11k dataset, and prove the effectiveness of our proposed model from the fine-grained perspective.

As shown in Table 7 and Table 7, on the entity prediction task, our proposed bt-HyTE and tr-HyTE can complete the missing entities that rank top 10 ahead of the static KGC method TransE and the temporal model HyTE that ignores the balanced distribution of timestamps. As shown in Table 7, in the relation prediction task, for some specific quadruples, HyTE confuses the similar relations between “dieIn” and “wasborn” like TransE, but our proposed bt-HyTE and tr-HyTE can distinguish these relations. The performance of tr-HyTE is better than that of bt-HyTE for relation prediction, which is because of the negative sampling for temporal relations. But in our method, the number of negative samples for temporal relations should be strictly controlled to avoid affecting the entity prediction.