Structure Enhanced Graph Neural Networks for Link Prediction

Graphs are ubiquitous in the real world and have a wide range of applications in many fields, including social networks, biological networks, the co-authorship network and the World Wide Web. Over the past few years, Graph Neural Networks (GNNs) have achieved great success on many tasks, including node classification, link prediction, graph classification, and recommender systems(Kipf and Welling, 2016; Hamilton et al., 2017; Zhang and Chen, 2018; Zhang et al., 2018; Ying et al., 2018). GNN uses deep learning methods to solve graph-related problems, which can be seen as an application of deep learning on graph data. One of the most popular methods of GNNs called Graph Convolutional Networks (GCNs). GCNs originated with the work of Bruna et al. (Bruna et al., 2013), which develops a version of graph convolutions based on spectral graph theory. Bruna et al. (Bruna et al., 2013) generalize the convolution operation from Euclidean data to graph data by using the Fourier basis of a given graph. And then Defferrard et al. (Defferrard et al., 2016) propose ChebNet, which utilizes Chebyshev polynomials as the filter to simplify the calculation of the convolution. Kipf et al. (Kipf and Welling, 2016) further simplify ChebNet by using its first-order approximation which is the well-known GCN model. Some approaches define graph convolutions in the spatial domain (Hamilton et al., 2017; Veličković et al., 2017). Hamilton et al. (Hamilton et al., 2017) introduce GraphSAGE which first samples a fixed-sized of neighborhood nodes and then aggregates neighbor information using different strategies. DGCNN (Zhang et al., 2018) proposes a SortPooling layer that sorts graph vertices in a consistent order and then uses a traditional 1-D convolutional neural network for graph classification. Veličković et al. (Veličković et al., 2017) proposes the graph attention network (GAT) model which uses the attention mechanism to assign different weights to different neighborhood nodes when aggregating information.

More recently, there has been a lot of works try to improve the training efficiency of GNNs. FastGCN (Chen et al., 2018) performs layer-wise sampling to solve the neighbor explosion problem of node-wise sampling methods like GraphSAGE (Hamilton et al., 2017). ClusterGCN (Chiang et al., 2019) proposes graph clustering based training method. GraphSAINT (Zeng et al., 2019) proposes the subgraph sampling based training method.

Link prediction is a fundamental problem for graph data. It focus on predicting the existence of a link between two target nodes (Liben-Nowell and Kleinberg, 2007). Classical approaches for link prediction are topology-based heuristic methods. Topology-based methods use graph structural features to calculate nodes’ similarity as the likelihood of whether they are connected. Some representative methods include Common Neighbors (CN) (Newman, 2001), Jaccard measure (Chowdhury, 2010), Adamic/Adar (AA) measure (Adamic and Adar, 2003), and Katz measure (Katz, 1953). CN computes the number of common neighbors as similarity scores to predict the link between two nodes. Jaccard measure computes the relative number of neighbors in common. CN and Jaccard only use one-hop neighbors, while the AA method captures two-hop similarities instead. The Katz measure further uses longer paths to capture a larger range of local structures.

Other common approaches are graph embedding methods, which first embed the nodes into low-dimensional space and then calculate the similarity of the nodes’ embeddings to predict the existence of a link. DeepWalk (Perozzi et al., 2014) first generates random walks and then uses Word2vec (Mikolov et al., 2013) to get the embeddings of nodes. LINE (Tang et al., 2015) introduces concepts of first-order and second-order similarity into weighted graphs. Node2vec (Grover and Leskovec, 2016) extend DeepWalk with a more sophisticated random walk strategy that combines the DFS and BFS.

More recently, GNNs (Kipf and Welling, 2016; Hamilton et al., 2017) have been used for link prediction. GNN-based methods typically use supervised learning to model the link prediction as a classification problem. Zhang and Chen (Zhang and Chen, 2017) propose WLNM to extract the enclosing subgraph around the target link and then encode the subgraph as a vector for subsequent classification. Zhang and Chen (Zhang and Chen, 2018) propose SEAL which first labels the nodes in each enclosing subgraph according to their distances to the source and target nodes, and then applies DGCNN (Zhang et al., 2018) to learn a link representation for link prediction. Li et al. (Li et al., 2020) introduces another node labeling trick similar to SEAL which directly uses the shortest distances to target nodes. You et al . (You et al., 2019) propose Position-aware GNN (P-GNN) (You et al., 2019). P-GNN uses anchor nodes to introduce relative location information of nodes into GNNs. The model proposed in this paper also uses the labeling trick introduced in SEAL (Zhang and Chen, 2018). However, SEAL simply concatenates structural features with original features of the nodes as the input of GNNs, which will be improved in this paper.

We first focus on the question that how to design and encode structural features. There are many local structural patterns around target nodes, such as common neighbors and triangles. To measure the correlation between two target nodes, we need to examine the topology of these nodes and edges connect them at the same time. Graph structures like Common Neighbors (CN) (Newman, 2001), Jaccard measure (Chowdhury, 2010), Adamic/Adar (AA) measure (Adamic and Adar, 2003), and Katz measure (Katz, 1953) are shown to be useful for link prediction. These methods can be unified as follows:

where $s(i,j)$ denotes similarity between node $v_{i}$ and $v_{j}$, $p^{l}_{i,j}$ denotes number of length-$l$ paths between target nodes $v_{i}$ and $v_{j}$, and $N(i)$, $N(j)$ means neighbor nodes of $v_{i}$ and $v_{j}$, respectively. For example, suppose $f(l,N(i),N(j))=\alpha^{l}$ and $\phi(p_{i,j}^{l})=p_{i,j}^{l}$, Equation 2 is the same as the Katz measure $\text{Katz}_{i,j}=\Sigma^{\infty}_{l=1}\alpha^{l}p_{i,j}^{l}$. If $l=1$, the equation is the same as neighbor based methods like CN and AA.

Two key components of Equation 2 are paths of target nodes $p^{l}_{i,j}$ and their neighbor nodes $N(i),N(j)$. Since information of neighbors is already considered in GNNs, paths are especially important to be an extra input of GNN models. Here we propose path between two targets nodes as a kind of topological feature, defined as follows:

To numericalize these paths as usable topological features for nodes, we propose a Path Labeling(PL) method to label each node $u$ with a discrete value $c_{u}$ based on the path it belongs to. Basically, nodes in different paths of different lengths are labeled with different discrete values. The fringe nodes that are not part of the path are labeled with a default value. For example, all 1-hop fringe nodes have an identical label value. The target nodes are also labeled with a default value. For a node that appears in multiple paths at the same time, we choose the shortest path to label the node.

Figure 2 shows examples of path labeling in the subgraph which consists of 1-hop neighbors of two target nodes $a$ and $b$. We label two target nodes $a$ and $b$ with a discrete value $c_{0}$; The node $c$ in (a) is the common neighbor of target nodes, and we label it with $c_{1}$; Nodes $d$ and $e$ in (b) are located on the path of length 3, we label them with $c_{2}$; Nodes $d$ and $e$ in (c) are located in both the path of length 3 (a-d-e-b) and length 4 (a-f-d-e-b), so we choose the path of length 3 to label $d$ and $e$ with value $c_{2}$, and label $f$ in (c) with $c_{3}$. Node $g$ is a fringe node, so we label it with a default value $c_{4}$.

Given these topological features extracted from paths of each pair of target node, we further propose a structure encoding method to learn representation vectors from them. The encoding method is defined as follows:

This encoding process can be implemented with neural networks. In Algorithm 1, we introduce our structure encoding method. We first extract the paths $p_{i,j}$ and then generate structural features $c_{u}$ for each node $u\in G_{S}$ according to path labeling(PL). Then we use one-hot method to generate the initial structural embedding of node $u$ according to $c_{u}$, notes that $c_{u}$ is first truncated with a constant $\lambda$ to prevent too long paths. Finally a GCN layer and a MLP is applied to get the final node structural embeddings.

In Algorithm 1, we use a GCN layer and MLP to fit $f$ and $\phi$ in Equation 2. With appropriate parameters and enough layers, we hope the error of approximation can be ignored.

By the way, SEAL (Zhang and Chen, 2018) also uses labeling method to generate structural features. It’s Double-Radius Node Labeling(DRNL) method labels each node according to its radius with respect to the two target nodes. The difference is that DRNL is much more strict than PL, because it distinguishes different nodes in the same length path. For example, given a path (a-b-c-d-e), where $a$ and $e$ are target nodes, the labeling result of DRNL is ($c_{0}$, $c_{1}$, $c_{2}$, $c_{1}$, $c_{0}$) and the result of PL is ($c_{0}$, $c_{1}$, $c_{1}$, $c_{1}$, $c_{0}$). In addition, we use a constant $\lambda$ to truncate too long paths, these difference prevent PL from overfitting thus more robust than DRNL.

To learn both node attributes and structural information, one straightforward way is to directly concatenate node features and structural features as the input of GNNs like SEAL (Zhang and Chen, 2018). However, these two features express different meanings. The node attributes are usually constructed by semantic information, while the structural features are directly extracted from graph topology. How to learn both semantic information and structural information at the same time is also a challenging problem.

We design a framework to fuse these two features called SEG. The detail architecture is shown in Figure 3. The SEG architecture contains two parts, one is a structure encoder and the other is a deep GNN. In the structure encoder, we start at encoding node structure features ($c_{u}$) with a single layer GCN. The reason for choosing one layer GCN is that the multi-hop information is already included in the path, so only one layer of aggregation is needed to update two target nodes we focus on. The output of the GCN layer are used as the input of a MLP to fit $f$ and $\phi$ in Equation 2. We denote the output vectors $z_{u}$ of MLP as the structural embeddings of $v_{u}$. Another MLP is used to predict the structure similarity score $s_{structure}$ based on the Hadamard product of structural embeddings of target nodes $z_{i}$ and $z_{j}$.

For the deep GNN model, structural embedding $z_{u}$ and original node attributes $x_{u}$ are firstly fused by a MLP. The module of feature fusion is to map $x$ and $z$ into the same feature space. With fused features as the input, a deep GNN model can learn from both structural features and original node features. Here we choose a multi-layers GraphSAGE (Hamilton et al., 2017) as our backbone GNN model and use a SortPooling (Zhang et al., 2018) layer to get the final representation output of two targets nodes. Finally, we use a MLP to predict the high-level semantic similarity score, denoted as $s_{semantic}$. We combine the structure similarity score and semantic similarity score as the final link existence probability:

In this section, we introduce the SEG training algorithm. The GNN updating procedure can be formed in Equation 1. However, in practice, considering the efficiency of training, especially for real-world large graphs, we do not perform the above update process directly on the whole graph (Hamilton et al., 2017; Chiang et al., 2019; Zhu et al., 2019). Instead, we extract a $k$-hop enclosing subgraph $G_{S}$ of two target nodes $i,j$ from the whole graph $G$. A $k$-hop enclosing subgraph is a graph consisting of $k$-hop neighbors of nodes $i$ and $j$. Then, we apply structure encoding described in Algorithm 1 on $G_{S}$. Finally, two parts of SEG is jointly training using a sigmoid cross entropy loss,

where $s_{t}$ refers to the probability score that the link $t$ is predicted to be true, $y_{t}$ is the label of the link $t$, and $N$ is the number of training edges. Algorithm 2 details how to generate the final predict score. During training, we randomly sample two nodes that are not connected as negative samples.

We conduct experiments on three public OGB datasets to answer the first question. It is worth mentioning that the SEAL method has achieved the best performance on these three datasets so far. Experimental results on three OGB datasets are reported in Table 2. Results show that the proposed method SEG beats SEAL and achieves the best performance on all three datasets.

On both ogbl-ppa and ogbl-collab datasets, SEG achieves 3$\%$ improvement over the SEAL method. The topology-based method CN achieves higher Hits@K than the Node2vec and GNNs which indicates that the structural information is more important than the original node features in these graphs. Besides, the MLP baseline performs extremely poorly on all three datasets, which shows that the attributes of the nodes in these datasets do not accurately reflect the interaction between the nodes.

As a matter of fact, in many real-world graphs, it is difficult to design high-quality semantic features for target tasks. For these graphs, it is better to effectively use the structural information from graph itself.

For larger scale ogbl-citation2, SEG still outperforms all baselines, which shows the effectiveness of the proposed method. DE-GNN, SEAL and SEG achieve a huge improvement over other GNN models on both ogbl-ppa and ogbl-collab, which illustrates the effectiveness of adding structural information to GNNs. Note that validation edges are also allowed to be included in training (Hu et al., 2020). So we conduct experiments to use the validation edges as training as well. The results in Table 3 show that SEG is still better than all other methods.

We next attempt to answer the second and third questions raised at the beginning of this section. We first only use structure encoding part in SEG denoted as SEG-SE, and compare the SEG-SE with SEAL-PL which uses path labeling instead of DRNL in SEAL on the ogbl-collab dataset. The result in Table 4 shows that SEG-SE achieves 0.5503 Hits@50, which is better than SEAL-PL. It demonstrates the effectiveness of our structure encoding. We then remove the final predictor MLP and $s_{structure}$ of structure encoding in SEG, denoted as SEG-GNN, and compare it with SEG-SE and SEG. As shown in Table 4, within these three methods, SEG achieves the best results, which illustrates the benefits of joint training. By joint training, structural embedding can better represent the original structural features.

It is worth noting that the result of SEG-GNN is still better than SEAL, which means our structure encoding method is more effective than DRNL of SEAL (Zhang and Chen, 2018).

This also proves our proposition, which is that the structural information and the original node features are in different spaces and cannot be directly and simply concatenated. Instead, we use a feature fusion module to map these two features to the same space.

To further address the difference between our Path Labeling(PL) and DRNL of SEAL, we conduct an experiment that uses SEAL’s DRNL features as the structural features in the SEG framework, denoted as SEG-DRNL in Table 4. The result shows that SEG outperforms SEG-DRNL. In addition, we replace the DRNL in SEAL with PL named SEAL-PL, and compare it with the original SEAL. The result shows that SEAL-PL is better than original SEAL. It means under the same training framework and settings, the PL are more effective than DRNL. Our analysis of the label values reveals that for the 1-hop subgraph, DRNL has a total of 17 different values of node labels, much more than the 4 of PL. This indicates that more label values introduce noise, so we do not distinguish between labels of nodes of the same path and place a limit on the length of the path to further prevent overfitting. Besides, the Hits@50 of SEG-DRNL is higher than Hits@50 of SEAL on ogbl-collab. It also illustrates the effectiveness of our SEG framework.

There is an ogbl-ddi graph (Wishart et al., 2017) in OGB link prediction datasets which is much denser than the other datasets introduced above. The ogbl-ddi dataset is an undirected graph. Each node represents a drug and the edge represents the interaction between drugs. It has 1,334,889 edges but only has 4,267 nodes. The average degree of the nodes in this graph is 500.5 and the density is 14.67$\%$, which is much larger than the other three graphs (see statistics of all datasets in Table 5). Nodes in ogbl-ddi do not contain any feature, so we exclude the MLP method in comparison. All learning-based methods use free-parameter node embeddings as the input. In the SEG method, we only use GNN and exclude structure encoder.

Table 6 shows experimental results. The results on ogbl-ddi indicate that DE-GNN, SEAL and SEG, all three GNN methods using structural features do not perform well on dense graphs. Although SEG still beats SEAL, it falls behind ordinary GNNs methods like GCN and GraphSAGE. This result shows the local structural information is no longer distinguishable in this dense graph. Note that the topology-based CN method has the worst performance, which also indicates that on this dense graph it is hard to learn meaningful structural patterns. However, most large-scale real-world graphs (e.g., citation graphs, internet graphs, etc.) are power-law (Clauset et al., 2009; Faloutsos et al., 2011), so local graph structure is useful for these graphs. For dense graph, we need to consider a larger range of global structural features, this is left to our future work.