Inter-domain Multi-relational Link Prediction

RESCAL [22] formulates a multi-relational data as a three-way tensor $\mathbf{X}\in\mathbb{R}^{n\times n\times m}$, where $n$ is the number of entities and $m$ is the number of predicates. $\mathbf{X}_{i,j,k}=1$ if the fact $(e_{i},r_{k},e_{j})$ exists and $\mathbf{X}_{i,j,k}=0$ otherwise. In order to find proper latent embeddings for the entities and the predicates, RESCAL performs a rank-$d$ factorization where each slice along the third mode $\mathcal{X}_{k}=\mathbf{X}_{\cdot,\cdot,k}$ is factorized as

Here, $\mathbf{A}=[\mathbf{a}_{1},...,\mathbf{a}_{n}]^{\top}\in\mathbb{R}^{n\times d}$ contains the latent embedding vectors of the entities and $\mathbf{R}_{k}\in\mathbb{R}^{d\times d}$ is an asymmetric matrix that represents the interactions between entities in the $k$-th predicate.

Originally, it is proposed to learn $\mathbf{A}$ and $\mathbf{R}_{k}$ with the regularized squared loss function

where

and reg is the following regularization term

$\mu>0$ is a hyperparameter.

It is later proposed by the authors of RESCAL to learn the embeddings with pairwise loss training [21], i.e. using the following margin-based ranking loss function

where $\mathcal{D}^{+}$ and $\mathcal{D}^{-}$ are the sets of all positive triplets (true facts) and all negative triplets (false facts), respectively. $f_{ijk}$ denotes the score of $(e_{i},r_{k},e_{j})$, $f_{ijk}=\mathbf{a}_{i}^{\top}\mathbf{R}_{k}\mathbf{a}_{j}$ and $\mathcal{L}$ is the ranking function

The negative triplet set $\mathcal{D}^{-}$ is often generated by corrupting positive triplets, i.e. replacing one of the two entities in a positive triplet $(e_{i},r_{k},e_{j})$ with a randomly sampled entity.

The pairwise loss training aims to learn $\mathbf{A}$ and $\mathbf{R}_{k}$ so that the score $f^{+}$ of a positive triplet is higher than the score $f^{-}$ of a negative triplet. Moreover, the margin-based ranking function is more flexible and easier to optimize with stochastic gradient descent (SGD) than the original squared loss function. In the proposed method, the pairwise loss training is adopted.

Given two probability vectors $\bm{\pi}_{1}\in\mathbb{R}_{+}^{n_{1}}$ and $\bm{\pi}_{2}\in\mathbb{R}_{+}^{n_{2}}$ that satisfy $\bm{\pi}_{1}^{\top}\mathbbm{1}_{n_{1}}=\bm{\pi}_{2}^{\top}\mathbbm{1}_{n_{2}}=1$, a matrix $\mathbf{P}\in\mathbb{R}_{+}^{n_{1}\times n_{2}}$ is called a transport plan between $\bm{\pi}_{1}$ and $\bm{\pi}_{2}$ if $\mathbf{P}\mathbbm{1}_{n_{2}}=\bm{\pi}_{1}$ and $\mathbf{P}^{\top}\mathbbm{1}_{n_{1}}=\bm{\pi}_{2}$. Here, $\mathbbm{1}_{n}$ indicates a $n$-dimensional vector of ones. Let’s denote the supports of $\bm{\pi}_{1}$ and $\bm{\pi}_{2}$ as $\mathbf{A}^{1}=[\mathbf{a}^{1}_{1},...,\mathbf{a}^{1}_{n_{1}}]^{\top}\in\mathbb{R}^{n_{1}\times d}$ and $\mathbf{A}^{2}=[\mathbf{a}^{2}_{1},...,\mathbf{a}^{2}_{n_{2}}]^{\top}\in\mathbb{R}^{n_{2}\times d}$, respectively. A transport cost $\mathbf{C}\in\mathbb{R}_{+}^{n_{1}\times n_{2}}$ can be defined as

Given a transport matrix $C$, the transport cost of a transport plan P is computed by

A transport plan $\mathbf{P}^{*}$ that gives the minimum transport cost, $\mathbf{P}^{*}=\arg\min_{\mathbf{P}}\langle\mathbf{P},\mathbf{C}\rangle$, is called an optimal transport plan and the corresponding minimum cost is called the Wasserstein distance. The optimal transport plan $\mathbf{P}^{*}$ gives a reasonable “soft” matching between the two distributions $(\bm{\pi}_{1},\mathbf{A}_{1})$ and $(\bm{\pi}_{2},\mathbf{A}_{2})$ while the Wasserstein distance provides a measurement of how far the two distributions are from each other.

In the scope of multi-relational graphs, $\bm{\pi}_{1}$ and $\bm{\pi}_{2}$ are predefined over the sets of entities, normally being set to be uniform and the supports $\mathbf{A}_{1}$ and $\mathbf{A}_{2}$ can be seen as embeddings of the entities.

The computational complexity of computing the optimal transport plan and Wasserstein distance is often prohibitive. An efficient approach to compute an approximation has been proposed by Cuturi et al. [9]. Instead of the exact optimal transport $\mathbf{P}^{*}$, they compute an entropic-regularized transport plan $\mathbf{P}^{\lambda}$ via minimizing a cost $M$ as follows,

where $\lambda>0$ is a hyperparameter controlling the effect of the negative entropy of matrix $\mathbf{P}$. With large enough $\lambda$, emperically when $\lambda>50$, $\mathbf{P}^{*}$ and the Wasserstein distance can be accurately approximated by $\mathbf{P}^{\lambda}$ and $M(\mathbf{P}^{\lambda})$.

$\mathbf{P}^{\lambda}$ has a unique solution of the following form

where diag(u) indicates a diagonal matrix whose diagonal elements are elements of u. The matrix $\mathbf{K}=e^{-\lambda\mathbf{C}}$ is the element-wise exponential of $-\lambda\mathbf{C}$. Vectors $\mathbf{u}$ and $\mathbf{v}$ can be initialized randomly and updated via Sinkhorn iteration

Maximum Mean Discrepancy (MMD) is originally introduced as a non-parametric statistic to test if two distributions are different [12, 13]. It is defined as the difference between mean function values on samples generated from the distributions. If MMD is large, the two distributions are likely to be distinct. On the other hand, if MMD is small, the two distributions can be seen to be similar. Formally, let $\bm{\pi}_{1}$ and $\bm{\pi}_{2}$ be two distributions whose the supports are subsets of $\mathbb{R}^{d}$, and $\mathcal{F}$ be a class of functions $f:\mathbb{R}^{d}\rightarrow\mathbb{R}$. Usually, $\mathcal{F}$ is selected to be the unit ball in a universal RKHS $\mathcal{H}$. Then MMD is defined as

From sample sets $\mathbf{A}^{1}=\{\mathbf{a}_{1}^{1},...,\mathbf{a}_{n_{1}}^{1}\}$ and $\mathbf{A}^{2}=\{\mathbf{a}_{1}^{2},...,\mathbf{a}_{n_{2}}^{2}\}$, $\mathbf{a}^{t}_{i}\in\mathbb{R}^{d}$, sampled from the two distributions, MMD can be unbiasedly approximated using Gaussian kernels $k(\cdot,\cdot)$ as follows [23, 12].

When $\mathbf{A}^{1}$ and $\mathbf{A}^{2}$ are the embeddings of entities in two domains, MMD represents a dissimilarity between the domains’ entity distributions.

The formal problem setting considered in this study is stated as follows. Given two multi-relational graphs $G^{1}$ and $G^{2}$, each graph $G^{t}$ is defined with a set of entities (nodes) $\mathcal{E}^{t}=\{e^{t}_{1},...,e^{t}_{n_{t}}\}$, a set of predicates (types of links) $\mathcal{R}^{t}=\{r^{t}_{1},...,r^{t}_{m_{t}}\}$, and a set of true facts (observed links) $\mathcal{T}^{t}=\{(e^{t}_{i},r^{t}_{k},e^{t}_{j})\}$ for $t\in\{1,2\}$. For simplicity, this study only considers the case where the two graphs share the same set of predicates, i.e. $\mathcal{R}^{1}\equiv\mathcal{R}^{2}\equiv\mathcal{R}$. The goal is to predict if an inter-domain fact $(e^{1}_{i},r_{k},e^{2}_{j})$ or $(e^{2}_{i},r_{k},e^{1}_{j})$ holds true or not.

The entity embeddings of the two graphs are assumed to follow the same distribution, i.e. there exists a distribution $\bm{\pi}$ such that $\mathbf{a}^{t}_{i}\sim\bm{\pi}$ for embedding $\mathbf{a}^{t}_{i}$ of entity $e^{t}_{i}\in\mathcal{E}^{t}$. In the experiments, the entity sets $\mathcal{E}^{1}$ and $\mathcal{E}^{2}$ are controlled so that they are completely disjoint or partially overlapped with only a small amount of common entities. The common entities are known in overlapping settings.

The proposed method’s objective function consists of two components. The first component is for learning embedding representations of the entities and the predicates of each multi-relational graph, which is based on an existing tensor-factorization method. RESCAL [22] is specifically chosen in the proposed method. The second component is a regularization term for enforcing the entity embedding distributions of the two graphs to become similar.

For each graph $G^{t}$, lets denote the entity embeddings as $\mathbf{A}^{t}=[\mathbf{a}^{t}_{1},...,\mathbf{a}^{t}_{n_{t}}]^{\top}\in\mathbb{R}^{n_{t}\times d}$, where $d$ is the embedding dimension. If the entity sets $\mathcal{E}^{1}$ and $\mathcal{E}^{2}$ overlap, the embeddings of common entities are set to be identical in both domains, i.e. $\mathbf{A}^{t}=[\mathbf{A}^{{}^{\prime}t},\mathbf{A}_{c}]^{\top}$ where $\mathbf{A}_{c}\in\mathbb{R}^{d\times|\mathcal{E}^{1}\cap\mathcal{E}^{2}|}$ is the embeddings of common entities. The embedding of predicate $r_{k}\in\mathcal{R}$ is denoted as $\mathbf{R}_{k}\in\mathbb{R}^{d\times d}$ for $k\in\{1,...,m\}$. The objective function of the proposed method is given as

In (4), the first two terms $L(\mathbf{A}^{t},\mathbf{R}_{k})$ are the loss functions of RESCAL and are defined as in (1). The third term $M(\mathbf{A}^{1},\mathbf{A}^{2},[\mathbf{P}])$ is the entropic-regularized Wasserstein distance (WD) or the MMD discrepancy between the entity distributions of the two graphs. In the case of WD regularizer, $M=M(\mathbf{A}^{1},\mathbf{A}^{2},\mathbf{P})$ is defined as in (2) with $\mathbf{P}\in\mathbb{R}_{+}^{n_{1}\times n_{2}}$. In the case of MMD regularizer, $M=M(\mathbf{A}^{1},\mathbf{A}^{2})$ is defined as in (3).

Via $L(\mathbf{A}^{t},\mathbf{R}_{k})$, the underlying embedding distribution over each entity set $\mathcal{E}^{t}$ is learned and characterized into $\mathbf{A}^{t}$, while $M(\mathbf{A}^{1},\mathbf{A}^{2},[\mathbf{P}])$ helps to drive these two distributions to become similar. Through the objective function $F$, similar entities of $G^{1}$ and $G^{2}$ are expected to lie close to each other on the latent embedding space, which encourages similar entities to involve in similar relations/links. Specifically, if $e^{1}_{i}\in\mathcal{E}^{1}$ and $e^{2}_{i}\in\mathcal{E}^{2}$ have similar embeddings $\mathbf{a}^{1}_{i}$ and $\mathbf{a}^{2}_{i}$, the inter-domain fact $(e^{1}_{i},r_{k},e^{2}_{j})$ is likely to exist if the intra-domain fact $(e^{2}_{i},r_{k},e^{2}_{j})$ exists thanks to their similar scores ${\mathbf{a}^{1}_{i}}^{\top}\mathbf{R}_{k}\mathbf{a}^{2}_{j}\approx{\mathbf{a}^{2}_{i}}^{\top}\mathbf{R}_{k}\mathbf{a}^{2}_{j}$.

The objective function $F(\mathbf{A}^{1},\mathbf{A}^{2},\mathbf{R}_{k})$ (MMD regularizer) is directly optimized with SGD. On the other hand, $F(\mathbf{A}^{1},\mathbf{A}^{2},\mathbf{R}_{k},\mathbf{P})$ (WD regularizer) is minimized iteratively. In each epoch, the transport plan $\mathbf{P}$ is fixed and the embedding vectors $\mathbf{A}^{1}$ and $\mathbf{A}^{2}$ are updated with SGD. At the end of each epoch, $\mathbf{A}^{1}$ and $\mathbf{A}^{2}$ are fixed and the plan $\mathbf{P}$ is sequentially updated via Sinkhorn algorithm [9].

The datasets used in the experiments are created from four popular knowledge graph datasets, namely FB15k-237 [30], WN18RR [10], DBbook2014, and ML1M [5]. The FB15k-237 dataset contains $272k$ facts about general knowledge. It has $14k$ entities and $237$ predicates. The WN18RR dataset consists of $86k$ facts about $11$ lexical relations between $40k$ word senses. The other two datasets represent interactions among users and items in e-commerce. The ML1M (MovieLens-1M) dataset composes of $434k$ facts with $14k$ users/items and $20$ relations, while the DBbook2014 has $334k$ facts with $13k$ users/items and $13$ relations. To create $G^{1}$ and $G^{2}$ for each dataset, two smaller sub-graphs of around $2k$ to $3k$ entities are randomly sampled from the original graph. The two graphs are controlled to share some amounts of common entities. Different levels of entity overlapping are investigated, from $0\%$ (non-overlapping setting) to around $1.5\%,3\%$, and $5\%$ (overlapping setting). Moreover, different predicates are removed so that $G^{1}$ and $G^{2}$ share the same predicate set, i.e. $\mathcal{R}^{1}\equiv\mathcal{R}^{2}\equiv\mathcal{R}$.

Intra-domain triplets $(e_{i},r_{k},e_{j})$ whose both entities $e_{i},e_{j}$ belong to the same graph are used for training. Inter-domain triplets $(e_{i},r_{k},e_{j})$ whose entities $e_{i},e_{j}$ belong to different graphs are used for validating and testing inter-domain performance. The validation and test ratio is $20:80$. Even though the goal is to evaluate a model’s ability to perform inter-domain link prediction, both inter-domain and intra-domain link prediction performances are evaluated. This is because the proposed method should improve inter-domain link prediction while does not harm intra-domain link prediction. Therefore, $5\%$ of intra-domain triplets are further spared from the training data for monitoring intra-domain performance.

The details for the case of $3\%$ overlapping are shown in Table 1. In other cases, the datasets share similar statistics.

In the experiments, Hit@10 score and ROC-AUC score are used for quantifying both inter-domain and intra-domain performances.

The experimental results are shown in Tables 2, 3, 4, and 5. Note that a random predictor has a Hit@$10$ score of less than $0.004$ and a ROC-AUC score of around $0.5$.