HMSG: Heterogeneous Graph Neural Network based on Metapath Subgraph Learning

The purpose of GNN is to apply deep neural network model to graph representation learning, and map graphs to low-dimensional vector spaces for downstream tasks, such as link prediction [35, 4], node classification [23, 1], recommendation [22, 36, 7], etc. The notion of graph neural networks is initially outlined in [10], then Franco et al. [24] extended recursive neural networks to graph learning task and Li et al. [18] treated the neighborhood information as the time step input of gated recurrent units. Recently, how to apply convolutional neural networks to graph data has become a hot topic. Related works can be divided into two categories: spectral-based methods and spatial-based methods. The main idea of spectral-based methods is to perform convolution operations in the Fourier domain. Defferrard et al. [5] used Chebyshev polynomials to approximate the graph filters, which reduced the complexity of graph convolution in spectral domain. Kipf et al. [17] further restricted the filters only on the first-order neighborhood of each node. Due to the reason that spectral-based methods input the whole graph to perform related operations, these methods suffer poor scalability and stability. The idea of spatial-based method is to directly aggregate information from neighbors of each node. Atwood et al. [2] regarded graph convolution as a diffusion process and assumed that information is transferred from a node to one of its neighboring nodes with a certain transition probability. Hamilton et al. [12] utilized aggregator functions to aggregate the information from sampled neighbors. Benefiting from the wide application of attention mechanism which have achieved good performance in various fields [29], Veličković et al. [30] used the self-attention mechanism to aggregate node information based on neighbors’ importance. The graph neural network models mentioned above are mostly only for homogeneous graphs, and cannot be directly applied to heterogeneous graphs.

Heterogeneous graph embedding aims to embed heterogeneous graphs into low-dimensional vector spaces. Dong et al. [6] designed metapath-guided random walk to generate sequences as the input of skip-gram [19] model to obtain the embedding of each node. Fu et al. [8] performed multiple prediction training tasks to learn embedding of both nodes and metapaths in heterogeneous information networks. Shi et al. [25] designed a metapath-based random walk to generate the same type of node sequences and apply DeepWalk model to learn node representation. Chen et al. [3] decomposed the heterogeneous graph into several bipartite graphs, and then applied the LINE [28] model to learn the embedding of each bipartite graph. Zhang et al. [34] performed joint optimization of heterogeneous skip-gram and deep semantic encoding to capture semantic-aware representation of heterogeneous graph. The above researches are traditional graph representation learning models, which only considered the graph structure and ignored node attributes. There are also many models base on deep learning. Wang et al. [32] used the attention mechanism to aggregate information on metapath-based homogeneous graphs, and then use semantic attention to aggregate multiple metapath information. Zhang et al. [33] jointly considered the heterogeneous content information and heterogeneous structural information. However, in their models, the information aggregation process was carried out only within the same type of nodes. Fu et al. [9] proposed intra-metapath aggregation to aggregate all node information in each metapath instance, but they indiscriminately treated different types of nodes in the same way.

The are some other heterogeneous graph neural network models that used different methodologies instead of metapath. Hong et al. [14] designed a type-aware attention layer, which embedded each node of the whole heterogeneous graph by jointing different types of adjacent nodes and associated linkages. Hu et al. [16] proposed a subgraph sampling method and designed graph transformer to directly aggregate features from heterogenous neighbors. Hu et al. [15] employed adversarial learning to capture rich semantics on heterogeneous graph, but the node attributes were not considered in this model. Since these studies used different methodologies, they can be regarded as parallel works to ours.

This part mainly aims to project attributes of different types of nodes into the same latent space, so that attribute information can be transferred among them. In a heterogeneous graph, different types of nodes are associated with different attributes, such as paper node with keywords, abstracts and author node with research fields. Those attributes are usually in represented in different feature spaces. Therefore, for each type of node, we design a type-specific linear transformation matrix to project their attributes into the same latent space. For a node $v\in\mathcal{V}_{A}$ of type $A$ in $\mathcal{A}$, we have

where ${\rm h}_{v}^{A}\in\mathbb{R}^{d_{A}}$ is the original feature vector, and ${{\rm h}}^{\prime}_{v}\in\mathbb{R}^{{d}^{\prime}}$ is the projected feature vector of node $v$. ${\rm W}_{A}\in\mathbb{R}^{{d}^{\prime}\times{d_{A}}}$ is the linear transformation matrix of node type $A$.

Through the transformation of attributes, the heterogeneity between different types of nodes is addressed, that facilitates the information transfer and aggregation between nodes in the graph.

Different metapaths represent different semantics. According to the starting and ending node types of metapaths, we can divide metapaths into two categories for convenience:

where $ho$ means the starting and ending nodes are of the same type in metapaths, otherwise is $he$.

In order to fully learn the information of each metapath, we generate the corresponding subgraphs according to Definition 4 and then apply aggregation methods to each subgraph. Following the types $t$ of metapaths, the generated subgraphs can be divided into homogeneous subgraphs $\mathcal{G}^{ho}$ and heterogeneous subgraphs $\mathcal{G}^{he}$ (as shown in Figure 1(c)):

For a specific type of nodes, their connections to the neighbors in different subgraphs carry different semantic information, so each subgraph can be regarded as an interaction graph with specific semantic information. Since the subgraphs are independent, learning tasks can be carried out on each subgraph $G\in\mathcal{G}^{t}$ in parallel, which results in more efficient learning. Moreover, by learning from homogeneous and heterogeneous subgraphs independently, useful information can be retained as much as possible.

In this step, the information in each subgraph will be transmitted between nodes. For homogeneous graph learning, there are many excellent works such as GCN [17], GAT [30] which can be used accordingly.

Each heterogeneous subgraph is in the form of a bipartite graph, because there are only two types of nodes in the subgraph and connections only exist between node pairs with different types. In bipartite graph aggregation, we are mainly concerned with the information of first-order heterogeneous neighbors, because the information of second-order neighbors, i.e., homogeneous neighbors, can be obtained from the homogeneous graph aggregation. Here we give three candidate aggregators inspired by GraphSAGE [12].

Mean. The mean operation averages the element-wise features of heterogeneous neighbor nodes as the features of target node:

where $N_{v}$ is the neighbors of node $v$.

Pooling. Element-wise pooling operation aggregates information across heterogeneous neighbors in the following way:

where $\sigma\left(\cdot\right)$ is the non-linearity activation function (ReLU in this paper), and $\rm max(\cdot)$ demotes the element-wise maximize operator which can be replaced by mean as well, ${\rm W}_{\rm pool}\in\mathbb{R}^{{d}^{\prime}\times{d}^{\prime}}$ and ${\rm b}_{\rm pool}\in\mathbb{R}^{{d}^{\prime}}$ are learnable parameters.

Attention. The self-attention mechanism has been proven to be an effective information aggregator [12]. Since different neighbors may have different influence on the target node, we use attention to learn the importance of each neighbor.

Next, we will use the self-attention mechanism to aggregate information from both homogeneous and heterogeneous graphs. Specifically, for a target node $v\in\mathcal{V}_{A}$, given a node pair $\left(v,u\right)$ in $G\in\mathcal{G}_{A}^{t}$, which is generated by a metapath $P_{A}^{t}$ that starts from node type $A\in\mathcal{A}$, we adopt a graph attention layer to learn the importance $e_{vu}^{G}$ which measures how node $u$ would contribute to the target node $v$. In homogeneous subgraph the importance of node $u$ can be formulated as follows:

while in heterogeneous subgraph the importance of node $u$ is:

where ${\rm a}_{G}\in\mathbb{R}^{2{d}^{\prime}}$ in homogeneous graph and ${\rm a}_{G}\in\mathbb{R}^{{d}^{\prime}}$ in heterogeneous graph are the parameterized attention vector for graph $G$ and $||$ denotes the concatenate operation.

Then we calculate $e_{vu}^{G}$ for nodes $u\in N_{v}^{G}$, where $N_{v}^{G}$ denotes the neighbors which are directly connected to node $v$ in $G$. We apply softmax function to obtain the normalized weight coefficient $\alpha_{vu}^{G}$:

Finally, the embedding of node $v$ in subgraph $G$ can be aggregated by the neighbors’ projected features with the corresponding coefficients as follows:

where ${\rm z}_{v}^{G}$ is the output of node $v$ for subgraph $G$, and $\sigma\left(\cdot\right)$ is the activation function.

In order to reduce the variance introduced by the heterogeneity of graphs and make the learning process more stable, we extended the self-attention to multiple heads. Specifically, we repeat $K$ independent attention mechanisms, and then concatenate the learned embeddings, resulting in the following formulation:

In summary, given $X$ metapath-based graphs $\mathcal{G}_{A}^{t}=\left\{G_{1},\cdots,G_{X}\right\}$ with node type $A\in\mathcal{A}$ and projected node features ${\rm h}^{\prime}$, $X$ groups of embeddings $\left\{{\rm z}_{v}^{G_{1}},\cdots,{\rm z}_{v}^{G_{X}}\right\}$ of the target node $v\in\mathcal{V}_{A}$ are generated, as shown in Figure 2(b).

After node aggregation step, we obtain the embedding of each node in different subgraphs. In order to get more semantic information, attention mechanism is applied to assign different weights to different subgraphs according to their importance.

First, the embeddings we obtained from node aggregation are transformed through a nonlinear transformation. Then we sum up each subgraph $G_{i}\in\mathcal{G}_{A}$ by averaging the node embeddings for all nodes $v\in\mathcal{V}_{A}$,

where ${\rm q}_{A}\in\mathbb{R}^{d_{x}}$ is the parameterized attention vector for node type $A$, ${\rm M}_{A}\in\mathbb{R}^{d_{x}\times{d}^{\prime}}$ and ${\rm b}_{A}\in\mathbb{R}^{d_{x}}$ are learnable parameters.

To make coefficients easily comparable across different subgraphs $G_{i}$, we normalize them using softmax function and then weighted summing all subgraphs:

In this step, we get the final embeddings for all nodes $v\in\mathcal{V}_{A}$, Figure 2(c) gives the illustration.

Through the above sections, we obtain the node representations of each node, which can be used for different downstream tasks. Different loss function can be defined depend on specific tasks. We train HMSG in two major learning paradigms: semi-supervised learning and unsupervised learning.

For semi-supervised learning, there are only a small number of nodes in the graph with label information. We can minimize the cross entropy of labeled node data and apply back propagation and gradient descent methods to optimize the parameters of all nodes:

where $\mathcal{V}_{L}$ is the set of labeled nodes, $y_{v}$ and ${y}^{\prime}_{v}$ are the ground truth and predicted probability vector of node $v$, respectively.

For unsupervised learning, label information is invisible, we can optimize the model weights by minimizing the following reconstructive loss function through negative sampling [20]:

where $\sigma\left(\cdot\right)$ is the sigmoid function, $V^{+}$ is the set of connected (positive) node pairs, $V^{-}$ is the set of negative node pairs randomly sampled from all unconnected node pairs. The overall process of HMSG is shown in Algorithm 1.

We use four commonly used public datasets to evaluate the performance of our proposed model and compare with state-of-the-art baselines. The detailed data description is summarized in Table 2.

ACM²²2http://dl.acm.org/. This is an academic network. Similar to HAN [32], we extract a subset of ACM that contains 4025 papers (P), 7167 authors (A) and 60 subjects (S) according to the conferences (KDD, SIGMOD, VLDB, SIGCOMM, MobiCOMM) where papers published in, and divide the papers into three classes (Data Mining, Database, Wireless Communication). Each paper is described by a bag-of-words representation of terms. The metapaths we selected are {PAP, PSP} and {PA, PS}. For semi-supervised learning tasks, the paper nodes are divided into training, validation, and testing ratio of 1:1:8.

IMDB³³3https://www.imdb.com/. This is an online database about movies and television programs. We extract a subset of IMDB that contains 4181 movies (M), 5257 actors (A), and 2081 directors (D). Each movie is labeled as one of three classes (Action, Comedy, Drama) based on their genre. Each movie is described by a bag-of-words representation of its plot keywords. The metapaths we selected are {MAM, MDM} and {MA, MD}. For semi-supervised learning tasks, the movie nodes are divided into training, validation, and testing ratio of 1:1:8.

Amazon Review⁴⁴4http://jmcauley.ucsd.edu/data/amazon/index.html. This dataset is about product reviews from Amazon which is used for the link prediction task, and no label or feature is included in it. We extract two datasets, i.e., Review-I (Video Games category) with 6259 users and 20145 items, Review-II (Movies category) with 12453 users and 32395 items. The metapaths we selected are {UI, IU}. For unsupervised learning tasks, the user-item pairs are divided into training, validation, and testing ratio of two groups, i.e., 5:1:4 and 7:1:2 for both Review-I and Review-II. The node features of training dataset are pretrained by DeepWalk algorithm [21].

We compare HMSG with other graph embedding methods to verify the performance of our proposed method. The detail description of baselines are as follows.

• •

  Deepwalk [21]: This is a random walk-based homogeneous graph representation learning method. We ignore the heterogeneity of the graph and input the entire graph into the DeepWalk model.

• •

  metapath2vec [6]: This is a heterogeneous graph representation learning method based on metapath guided random walk and skip-gram model. We test all the metapaths for metapath2vec and report the best performance.

• •

  HERec [25]: This is a heterogeneous graph embedding method which designs a metapath-based random walk to generate the same type of node sequences and feed into DeepWalk model. We test all the metapaths for HERec and report the best performance.

• •

  GCN [17]: It is a homogeneous graph convolutional network which aggregates information from immediate neighbors. We test GCN on metapath-based homogeneous graphs and report the results from the best metapath on semi-supervised learning tasks. On unsupervised learning tasks, we input the entire graph by ignoring the heterogeneity of the graph.

• •

  GAT [30]: It is a homogeneous graph convolutional network which calculates the importance of different neighbors by attention mechanism. We test GAT on metapath-based homogeneous graphs and report the results from the best metapath on semi-supervised learning tasks and input the entire graph by ignoring the heterogeneity of the graph on unsupervised learning tasks.

• •

  HAN [32]: It is a heterogeneous graph neural network which designs node attention for aggregating metapath-based homogeneous graph and utilize semantic attention to aggregate the information of multiple metapaths.

• •

  MAGNN [9]: It is a heterogeneous graph neural network which ultilizes intra-metapath aggregation and inter-metapath aggregation to aggregate node information.

For random-walk-based methods, including DeepWalk, metapath2vec and HERec, we set window size to 5, walk length to 100, walk per node to 40, and negative samples to 5. For GCN, GAT, HAN and HMSG, we use the same splits of training, validation, and testing sets. We train them for 1000 epochs and apply early stopping with patience of 30. We employ the Adam optimizer with the learning rate set to 0.005 and the weight decay (L2 penalty) set to 0.001. We set the dropout rate to 0.6. In the HMSG model, we use the self-attention mechanism to aggregate homogeneous and heterogeneous subgraphs, and other variants of HMSG are analyzed in the ablation study section. For GAT, HAN and HMSG, we set the number of attention heads to 8. For HAN, HMSG, we set the dimension of attention vector in subgraph aggregation to 128. For a fair comparison, we set the embedding dimension of all the algorithms to 64. Our model is implemented via the Deep Graph Library (DGL) [31] package of PyTorch frameworks.

We conduct experiments on ACM and IMDB datasets to compare the node classification performance of different models. We use the embedding of labeled nodes (paper nodes in ACM and movie nodes in IMDB) as the input of the SVM classifier, and divide the train/test ratio into different proportions. Similar to [9], only the nodes in the testing set of HMSG are used for the downstream training and evaluation of SVM. This strategy is also used for node clustering and link prediction tasks. In order to eliminate the variance caused by the unbalanced data division, we repeat each experiment for 10 times and report the average $Macro\text{-}F1$ and $Micro\text{-}F1$ as evaluation metrics. The node classification results are shown in Table 3.

It can be seen from the Table 3 that, compared to the state-of-the-art models HAN and MAGNN, HMSG achieves the best performance on all experimental groups, showing the significant advantage of fusing the information from both homogeneous and heterogeneous subgraphs. In addition, the performance of metapath2vec method which considers the heterogeneous structure outperforms other random walk-based methods, but it is still not as good as GCN and GAT, which are based on graph convolution model.

We conduct clustering experiments on ACM and IMDB datasets to evaluate the quality of embeddings learned by HMSG. For the clustering experiment, similar to node classification, we take the embeddings of labeled nodes in the testing set as the input of K-Means model, and use NMI and ARI metrics to evaluate the performance. Similarly, the results are average of 10 executions.

The results are shown in Table 4, from which we see that HMSG significantly outperforms all baselines, showing its strong capability of learning effective node representations. Besides, there is a significant performance gap between graph convolution-based methods (GCN, GAT, HAN, MAGNN and HMSG) and random walk-based methods, which again shows the power of graph convolutional model.

Link prediction is used to test our algorithm’s effectiveness in unsupervised learning task. We use two Amazon Review datasets Review-I and Review-II to evaluate the performance in link prediction task. We regard connected user-item pairs as positive node pairs and all unconnected links as negative node pairs. In the training phase, we randomly select the same number of positive and negative node pairs to training, validation and testing, and use formula (15) to optimize the model.

Then, we use the dot product of user and item embeddings obtained from the model to calculate the link probability between two nodes: $p_{ui}=\sigma\left({\rm z}_{u}^{\rm T}\cdot{\rm z}_{i}\right)$, and $\sigma\left(\cdot\right)$ is the sigmoid function. We conduct 10 repeated experiments and report the average AUC and AP metrics on the testing set. The results of link prediction task are presented in Table 5.

As shown in Table 5, HMSG still performs the best on all the dataset groups evaluated by all the metrics. Again, graph convolution-based methods perform much better than random-walk-based methods. At the same time, we can see that heterogeneous graph models such as HAN and MAGNN perform better than homogeneous neural networks GCN and GAT in most of the time.

In sum, the above results validate the effectiveness of our HMSG model. In the following subsection, we will conduct ablation study to compare different variants of HMSG.

To validate the effectiveness of different components in our model, we conduct experiments on different variants of HMSG. Here ${\rm HMSG_{\rm ho}}$ means only homogeneous subgraphs are used in the model; ${\rm HMSG_{\rm mean}}$ means we use an average strategy in heterogeneous subgraph aggregation; ${\rm HMSG_{\rm pool}}$ means max pooling strategy in heterogeneous subgraph aggregation; ${\rm HMSG_{\rm atten}}$ means self-attention mechanism are used in heterogeneous subgraph aggregation. We present the average results with different training ratios in Table 6.

As can be seen, except for the clustering task on IMDB where the ${\rm HMSG_{\rm ho}}$ model (which uses homogeneous subgraphs only) performs the best, on all the other experimental groups, HMSG models considering both homogeneous and heterogeneous subgraphs give the best results, among which ${\rm HMSG_{\rm atten}}$ model exhibits the best overall performance, which again validates the effectiveness and necessity of our model design.