Local Structure-aware Graph Contrastive Representation Learning

Let $\mathcal{G}=(\mathcal{V},\mathcal{E})$ indicates an undirected attributed graph, where $\mathcal{V}$ and $\mathcal{E}$ denote the set of nodes and edges, respectively. We denote the adjacency matrix of the graph and the initial feature matrix of the nodes as $\textbf{A}\in\left\{0,1\right\}^{N\times N}$ and $\textbf{X}\in\mathbb{R}^{N\times F}$, where N and F represent the number of nodes in the graph and the dimension of the initial attribute feature, respectively. $\textbf{A}_{ij}=1$ if ($i$, $j$) $\in\mathcal{E}$. $\textbf{X}_{i}\in\mathbb{R}^{1\times F}$ denotes the initial feature of node $i$.

In the GCL framework, the positive and negative samples for graph-structured data are constructed in terms of the graph structure or the attribute features of nodes, which affect the learning ability of the encoder[24]. The DGI obtains the corrupted graph by shuffling the node order of the feature matrix and utilizes the corrupted graph embedding and original graph embedding as negative sample and positive sample separately [18]. The GGD[25] obtains the corrupted graph by the edge and feature dropout, and assigns different manual labels to the nodes in the original graph and corrupted graph to train the GNN encoder[25]. However, global graph embedding, which compresses too much structural information into a simple fixed-length vector, approximates the constant vector[25], and the edge dropout methods ignore the information of the substructure.

In this paper, to explore the local graph structural information, we apply the semantic subgraph for data augmentation. Considering the structural information of nodes mainly depends on local neighbors, we introduce the personalized pagerank algorithm (PPR) [26], which searches for the top $K$ related nodes to target nodes, to construct the semantic subgraphs.

Given the adjacency matrix A of original graph $\mathcal{G}$, we firstly obtain the importance score matrix M, which is calculated as shown in Eq.1:

where $p$ is an optional parameter which is set as 0.15 and I denotes the identity matrix. $\hat{\textbf{A}}=\textbf{A}\textbf{D}^{-1}$ represents the column-normalized adjacency matrix, where D indicates the diagonal matrix with $\textbf{D}_{ii}=\sum_{j}\textbf{A}_{ij}$. And $\textbf{M}_{i}$ denotes the importance score vector of node $i$, where each value represents the relative importance between node $i$ and another node.

For the target node $i$, we obtain the top $K$ important node set $ID(i)$ via the importance score vector $\textbf{M}_{i}$ of node $i$. The calculation process is shown below:

where $K$ indicates the count of nodes in the semantic subgraph and $Rank(\cdot)$ is a function that returns the index of the top $K$ related nodes. We construct semantic subgraphs via the obtained $K$ nodes and the relationships between them:

where $\textbf{X}^{S}_{i}$ and $\textbf{A}^{S}_{i}$ denote the attribute information and structural information of the semantic subgraph $S_{i}$ for target node $i$, respectively. The process of generating subgraphs can be achieved as a pre-task to reduce memory usage and running time compared to the online subgraph generation methods.

In the LS-GCL, each contrastive objective represents the local structural and semantic information of the target nodes in different levels. In this paper, we use a single GCN layer as the encoder to learn the node embeddings, and add a shared MLP layer for enhancing the expressiveness of the GNN encoder. The calculation process is as follows:

where $\tilde{\textbf{A}}=\textbf{A}+\textbf{I}_{N}$ and $\textbf{I}_{N}$ denotes the identity matrix. $\tilde{\textbf{D}}_{ii}=\sum_{j}\tilde{\textbf{A}}_{ij}$ and W represents the weight matrix to be trained in the GNN encoder. $\sigma(\cdot)$ denotes the activation function (PReLU). In our LS-GCL framework, the GNN encoder can be changed, such as the GAT, GraphSAGE and SGC which have different message aggregation methods.

For the semantic subgraph $S_{i}$ of target node $i$, we employ the GNN-based encoder to learn the target node embeddings. The calculation process is as follows:

where $\vec{h}^{S}_{i}$ represents the embedding of target node $i$ at the subgraph-level. The purpose of constructing semantic subgraph $S_{i}$ is to maximize the local structural information of nodes. Therefore, we apply a pooling function to generate the subgraph-level graph embedding for target node $i$:

where $Pool(\cdot)$ is the pooling function and $\vec{h}^{i}_{S}$ represents the embedding representation of the semantic subgraph $S_{i}$ for target node $i$. In our experiments, we apply the mean-pooling function to learn the subgraph embeddings, which averages the feature vectors of nodes in the subgraph $S_{i}$.

Furthermore, we add the target node embeddings from the global-graph view as the contrastive objective.

where $\vec{h}^{G}_{i}$ represents the embedding representation of target node $i$ at global graph-level.

Finally, we obtain the structural information of target nodes from multiple perspectives, including the node embedding at the global graph-level, the node embedding at the subgraph-level and the corresponding subgraph-level graph embedding, which can explain the semantic information of nodes from different levels.

In the paper, the process of LS-GCL is to define a pre-task that constructs positive and negative samples for training the GNN encoder without using label information. For the target node $i$, the LS-GCL combines the node embedding at the global graph-level $\vec{h}^{G}_{i}$ and the node embedding at subgraph-level $\vec{h}^{S}_{i}$ as well as the subgraph-level graph embedding $\vec{h}^{i}_{S}$. The LS-GCL then adopts the margin triplet loss function[27] to train the GNN encoder. To reduce the training time of the LS-GCL model, we conduct contrastive learning for any two perspectives in parallel. The contrastive loss function $\mathcal{L}_{NS}$ between the node embeddings $\vec{h}^{S}_{i}$ at subgraph-level and the subgraph-level graph embeddings $\vec{h}^{i}_{S}$ for target node $i$ is as follows:

where $\sigma(\cdot)$ denotes the sigmoid function. $\alpha$ is the margin value. ${\vec{h}}^{j}_{S}$ is the negative sample embedding where $j$ is one of the other nodes.

The contrastive loss function $\mathcal{L}_{NG}$ between the node embeddings $\vec{h}^{S}_{i}$ at subgraph-level and the node embeddings $\vec{h}^{G}_{i}$ at global graph-level for target node $i$ is as follows:

The contrastive loss function $\mathcal{L}_{SG}$ between the node embeddings $\vec{h}^{G}_{i}$ at the global graph-level and subgraph-level graph embeddings $\vec{h}^{i}_{S}$ for target node $i$ is as follows:

To capture the comprehensive structural and semantic information, the overall contrastive loss function $\mathcal{L}$ of the proposed LS-GCL can be defined as follows:

We adopt the Adam optimizer to adjust the parameters of the GNN encoder through the back propagation mechanism for maximizing the common information of node embeddings at three levels. The pseudocode of the LS-GCL model is summarized in Algorithm 1.

In the experiments, we utilize five public datasets, such as Cora[28], Citeseer[29], Pubmed[30], Cora_ML[31] and DBLP[32] datasets. The five datasets are the citation graphs, where nodes represent scientific papers and edges represent the citation relationships between papers. Each node in the datasets has the initial features and a unique category label. The statistical information of the five datasets is shown in Table 1.

For the tasks of node classification and link prediction, we use two classical semi-supervised learning algorithms and four state-of-art graph contrastive learning algorithms as baseline algorithms. For the baseline methods, we report their experimental results on downstream tasks based on their open codes.

• 1.

  GCN: The graph convolutional network (GCN) is a classical graph neural network algorithm that captures the structural information of target nodes by aggregating the feature information of neighbors to learn the node embeddings. The training process requires the participation of label information.

• 2.

  GAT: The graph attention network (GAT) is one of the classical graph neural network algorithms. It introduces the attention mechanism to aggregate the feature information of neighbors to extract the structural information of nodes. The training process of the GAT requires label information.

• 3.

  DGI: The deep graph infomax (DGI) is an unsupervised learning model that maximizes the common information between the target nodes and the global graph by establishing a contrastive perspective between the node level and the global graph level to generate the node embedding representations.

• 4.

  GMI: The graphical mutual information (GMI) is an unsupervised method that maximizes the mutual information of both features and edges between input graph and output graph of the encoder to learn node embeddings.

• 5.

  SUBG-CON: The sub-graph contrast (SUBG-CON) is a self-supervised learning paradigm, which maximizes the common information at the node-level and subgraph-level by constructing the subgraphs for nodes. The SUBG-CON method alleviates the memory problem of the GCL paradigm.

• 6.

  GGD: The graph group discrimination (GGD) is a self-supervised learning model, which trains the model by assigning different artificial labels to positive and negative samples to achieve the consistency of feature information between similar samples and then obtains the embedding representations of nodes.

In the experiments, for the graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$, we employ the personalized pagerank (PPR) algorithm to obtain the top $K$ important node sets of target nodes for constructing the semantic subgraphs. We set $K=20$ on five datasets in the experiments. For the parameter $\alpha$ in the margin triplet loss function, we set $\alpha=0.5$ on the four datasets except that the value of $\alpha$ in the Pubmed dataset is 0.35. In the GNN encoder, the input dimension is the initial feature dimension of the nodes, and the output dimension is 1000, except for 450 in the Pubmed dataset. We use the Adam optimizer with an initial learning rate of 0.001 to adjust the parameters of the encoder. The experiments are conducted on an eight-core Intel i7 2.50 GHz processor and 16 GB of RAM and a NVIDIA RTX 3090 24G GPUs.

Compared to supervised learning methods, the proposed LS-GCL model exploits structural feature information of nodes using known graph structures (e.g. adjacency matrices), without the node labels, to learn the node embeddings. We then use a small number of labels, such as 20 labels for each category of nodes, to fine-tune[33] the node embeddings for specific node classification task.

Experimental results of node classification in terms of accuracy on five real-world datasets are shown in Table 2. All experimental results are the mean and standard deviation of ten repeated node classification experiments. We can find that the proposed LS-GCL model obtains optimal node classification accuracy on all five datasets. Our LS-GCL model mines the semantic information of nodes from both local and global perspectives. Nodes with similar structural features in the graph generate similar node embeddings, which helps to predict the unknown labels of nodes. In addition, the node classification accuracy of the GCL models is higher than that of the semi-supervised learning methods on five datasets, demonstrating that the GCL methods are useful for capturing latent structural features of nodes.

The task of link prediction is to predict potential node-pair relationships according to the structural features between nodes in the graph. Traditional GNN models, which are the supervised learning methods, require a large number of labels to train the parameters of the model for capturing the relationships between node pairs for link prediction experiments[34]. The LS-GCL learns target node embeddings in the link prediction task is similar with the node classification experiments. Our model migrates the node embeddings obtained from pre-training process to the downstream tasks by fine-tuning. The link embeddings are generated by concatenating the embedding vectors of the node pairs. The training and test sets containing positive and negative samples as well as the corresponding link labels are fed into a classifier (e.g. random forest[35]) to evaluate the effectiveness of the node embeddings generated by the LS-GCL model. In the five datasets, the proportion of test set is 40%.

Experimental results of link prediction for the AUC index on five real-world datasets are shown in the Fig. 2. To illustrate the robustness of the proposed LS-GCL, we use three other metrics, namely Recall, Precision and F1-score(F1)[36], which are shown in Table 3 and Table 4. All experimental results are the mean and standard deviation of ten repeated link prediction experiments. We can see that our LS-GCL model achieves outstanding and stable experimental results on four evaluation indices for all five datasets compared to current GCL and semi-supervised learning models. The node embeddings obtained by the GCL method with fine-tuning achieve an improvement in the link prediction task compared to the traditional GNN model.

The LS-GCL model has achieved better results in both node classification and link prediction tasks. In this section, we discuss three different ways of constructing subgraphs to evaluate the effectiveness of semantic subgraphs and the robustness of our LS-GCL framework. $K$-hop subgraph[37] represents that the subgraphs are constructed by the neighboring nodes which are $K$ hop away from the target nodes. We construct the 1-hop subgraphs to capture the local structural features of target nodes. $K$-RW subgraph denotes that the local subgraphs are constructed by the path generated through the random walk algorithm with $K$ length for the target nodes. We set the length of the random walk as 20 to obtain the same size subgraph. $K$-rank subgraph denotes the semantic subgraph which contains top $K$ related nodes.

Node classification results of three methods for extracting subgraphs are shown in Table 5. Our LS-GCL method with semantic subgraphs has higher accuracy than the LS-GCL model based on $K$-hop subgraphs and $K$-RW subgraphs, which indicates that the semantic subgraphs are capable for capturing potential structural information of nodes. The semantic subgraphs not only focus on the first-order neighboring nodes of target nodes, but also consider the higher-order related nodes, which enriches the semantic information of nodes.

The semantic subgraphs of our LS-GCL model are constructed with the top $K$ relevant nodes. We investigate how subgraph size affects the actual performance of our model on node classification task, which is shown in the Fig.3. It has been shown that the node classification accuracy of the LS-GCL model steadily improves as the subgraph size $K$ increases. For the five datasets, the accuracy gradually improves as the subgraph size $K$ increases from 0 to 20, indicating that GNN can extract more structural information from semantic subgraphs with more nodes. The node classification accuracy of the LS-GCL model reaches a plateau when the number of nodes exceeds 20. Compared to traditional GCL methods such as the SUBG-CON, the LS-GCL model considers the global graph structure, which improves the robustness of our model in downstream tasks.

In this paper, the proposed LS-GCL uses the GNN model as an encoder to learn node embeddings. We perform node classification experiments with different GNN models as feature encoders to test the impact of the encoders on our LS-GCL framework. For the GAT, the GraphSAGE[38], the SGC model[39], we apply a single GNN layer as the encoders. The experimental results are shown in Table 6. Our LS-GCL framework using a GCN-based encoder achieves higher accuracy on four datasets except the DBLP dataset. The node classification accuracy of the GNN-based encoders varies slightly between the five datasets, which demonstrates that our LS-GCL framework is robust. We can adapt the different GNN encoders to the actual situation and design a specific LS-GCL model for the downstream tasks.

In the paper, we propose a new multi-level contrastive loss function $\mathcal{L}$. To reflect the semantic information of nodes at different levels, we set the node embeddings at the global graph-level, the node embeddings at the subgraph-level and the subgraph-level graph embeddings corresponding to target nodes as the contrastive objectives. To evaluate the effectiveness of the multi-level contrastive loss function, we use $\mathcal{L}_{NG}$ and $\mathcal{L}_{NS}$ as the contrastive loss functions to learn the target node embeddings and perform node classification experiments, respectively. $\mathcal{L}_{NG}$ denotes that the contrastive objectives are node embeddings at subgraph-level and global graph-level. $\mathcal{L}_{NS}$ denotes that the contrastive objectives are node embeddings at subgraph-level and subgraph-level graph embeddings corresponding to target nodes. Experimental results of node classification experiments corresponding to different contrastive loss functions on five datasets are shown in Fig.4. The LS-GCL model using the multi-level contrastive loss function $\mathcal{L}$ achieves better results than $\mathcal{L}_{NG}$ and $\mathcal{L}_{NS}$. Compared with the models using $\mathcal{L}_{NG}$ and $\mathcal{L}_{NS}$ as the contrastive loss functions, the LS-GCL model captures different semantic information at three levels and each contrastive object has an essential role in the maximisation of the feature information for target nodes.