GaGSL: Global-augmented Graph Structure Learning via Graph Information Bottleneck

Spectral-based methods aim to identify graph patterns in the frequency domain, leveraging the sound mathematical precepts of Graph Signal Processing (GSP) [33, 34]. Bruna et al. [35] first extended the convolution to general graphs using a Fourier basis, treating the filter as a set of learnable parameters and considering graph signals with multiple channels. but eigendecomposition requires $O(n^{3})$ computational complexity. In order to reduce the computational complexity, Defferrard et al. [36] and Kipf et al. [10] made several approximations and simplifications. Defferrard et al. [36] defined fast localized convolutional filters on graphs based on Chebyshev polynomials. Kipf et al. [10] further simplified ChebNet via a localized first-order approximation of spectral graph convolutions. Despite being spectral-based, GCN can also be viewed through a spatial perspective. In this context, GCN operates by aggregating feature information from the local neighborhood of each node. Recent studies have progressively improved upon GCN [10] by exploring alternative symmetric matrices. For example, The adaptive Graph Convolution Network (AGCN) [37] proposed a Spectral Graph Convolution layer with graph Laplacian Learning (SGC-LL), which efficiently adapts the graph topology according to the data and learning task context. Dual Graph Convolutional Network (DGCN) [38] hamilton2017inductivedesigned a dual neural network structure to encode both local and global consistency. Other spectral graph convolutions have also been proposed. Graph Wavelet Neural Network (GWNN) [39] defines the convolution operator via wavelet transform, which avoids matrix eigendecomposition and provides good interpretability with its local and sparse graph wavelets. Simple Spectral Graph Convolution (S²GC)[40] derives a variant of GCN based on a modified Markov Diffusion Kernel. It achieves a balance between low- and high-pass filter bands to capture both global and local contexts of each node.

Conversely, spatial-based methods draw inspiration from the message passing mechanism employed in Recurrent Graph Neural Network (RecGNN) [41, 42]. They directly define graph convolution in the spatial domain as transforming and aggregating local information. The Neural Network for Graphs (NN4G) [43] is the first work towards spatial-based convolutional graph neural networks (ConvGNNs). Unlike RecGNNs, NN4G utilizes a combinatorial neural network architecture where each layer has independent parameters to learn the mutual dependencies of the graph. This method allows the extension of a node’s neighborhood through the progressive construction of the architecture. To identify a particularly effective variant of the general approach and apply it to the task of chemical property prediction, the message passing neural network (MPNN) [8] provides a unified model for spatial-based ConvGNNs. In this framework, graph convolutions are treated as a message-passing process, where information is directly transmitted between nodes along the edges. The graph isomorphism network (GIN) [14] identifies that previous methods based on MPNNs lack the ability to differentiate between distinct graph structures based on the embeddings they generate. To address this limitation, GIN proposed a theoretical framework for analyzing the expressive capabilities of GNNs to capture different graph structures. Obtaining the full size of a node’s neighborhood is inefficient since the number of neighbors a node has can span a wide range, from as few as one to possibly more than a thousand or more. Hamilton et al. [44] generated representations by sampling and aggregating features from a node’s local neighborhood. Velivckovic et al. [5] assumed that the contributions of neighboring nodes to the central node are neither identical as in GraphSage nor predetermined as in GCN. Velivckovic et al. [5] assigned distinct edge weights based on node features during aggregation process. GCN is primarily inspired by the latest deep learning methods and therefore may inherit unnecessary complexity and redundant computation. Wu et al. [20] reduced complexity and computation by successively removing nonlinearities and collapsing weight matrices between consecutive layers.

Many other graph neural network models can be reviewed in recent surveys [6, 45]. But almost all these GNNs assume that the observed structure accurately reflects the underlying node relationships, which considerably constrains their ability to manage uncertainty in the graph topology.

Metric learning methods polish the graph structure by learning a metric function that evaluates the similarity between pairs of node representations. For example, Zhang et al. [48] and Wang et al. [9] leveraged cosine similarity to model the edge weights. Zhang et al. [48] detected fake edges with different features and labels between nodes and mitigates their negative impact on prediction by removing these edges or reducing their weight in neural messaging. Wang et al. [9] employed a distinct method to derive the final node embeddings. It diffuses node features across the original graph and integrates the representations from both the generated feature graph and the original input graph using an attention mechanism. Note that Chen et al. [25] constructed the graph via a multi-head self-attention network, which is an end-to-end graph learning framework for iteratively optimizing graph structure and node embeddings. The core idea of IDGL [25] is that better node embedding is beneficial to capture better graph structure.

Probabilistic modeling approaches assume that graph is generated through a sampling process from certain distributions, and they use learnable parameters to model the probability of sampling edges. Franceschi et al. [23] modeled the edges between each pair of nodes by sampling from Bernoulli distributions with learnable parameters, and presented GSL as a two-layer programming problem, which was the first work in probabilistic modeling. The approach proposed by Zhang et al. [49] employed Monte Carlo dropout to sample the learnable model parameters multiple times for each generated graph. Meanwhile, the method developed by Wang et al. [22] processed multi-view information, such as multi-order neighborhood similarity, as observations of the optimal graph structure, and then derived the final graph structure based on Bayesian inference.

Direct optimization approaches consider the graph adjacency matrix as trainable parameters, which are tuned jointly alongside the primary GNN parameters. Yang et al. [50] aimed to leverage a given class label to both refine the graph structure and optimize the parameters of the GNN simultaneously. Jin et al. [19] focused on investigating key graph properties like sparsity, low rank, and feature smoothness, with the goal of designing more robust graph neural network models.

GSL methods can also be roughly split into single view and multiple fundamental views methods. However, these methods are insufficient to explore the theoretical guidance of learning optimal structure. A more comprehensive review of GSL can be found in a recent study [21].

Given an input graph $G=(\bm{A},\bm{X})$ and partial node labels $\bm{Y}$, the objective of GSL for GNNs is to jointly learn an optimal graph structure and the GNN model parameters, with the aim of enhancing the node classification performance for unlabeled nodes. The main notations used in this paper are summarized in Table I.

Intuitively, the first term $-I(\bm{Z};\bm{Y})$ encourages the representations $\bm{Z}$ to be maximally informative about the target (sufficient). The second term $(\bm{Z};G)$ serves to prevent $\bm{Z}$ from obtaining extraneous information from the data that is not pertinent to predicting the target (minimal). The Lagrangian multiplier $\beta$ trading off sufficiency and minimality. Existing works based on GIB primarily focus on graph-level tasks [29, 51, 28]. In this paper, we specifically discuss GIB for node classification tasks.

We can construct a similarity matrix $\bm{S}^{1}$ by Eq. (11). The original graph structure $\bm{A}$ carries relatively rich information. Ideally, the learned graph structure $\bm{S}^{1}$ can complement the original graph structure $\bm{A}$, creating an optimized graph for GNNs that enhances performance on the downstream task [25]. Thus, the matrix $\bm{A}$ is combined with the similarity matrix $\bm{S}^{1}$ to get the redefined structure $\bm{A}_{r1}$:

where $\gamma^{1}\in[0,1]$ is combination coefficient. Similarly, we can obtain the redefined structure $\bm{A}_{r2}$ for graph $G_{af}$:

where $\gamma^{2}$, $\mu\in[0,1]$ are combination coefficients. Note that when using Eq. (11) based on the diffusion matrix, top-$k$ neighbors are selected for each node based on the PPR values.

After employing structure redefinition, we can obtain two corresponding graphs $G_{r1}=(\bm{A}_{r1},\bm{X})$ and $G_{r2}=(\bm{A}_{r2},\bm{X})$.

Please review that we aims to learn minimum sufficient graph structure for node classification tasks. In other words, we want the learned representations $\bm{Z}^{*}$ based on $G^{*}=(\bm{A}^{*},\bm{X})$ to contain only labeled information, while filtering out label-irrelevant noise. To the end, we use GIB to guide the training of $\bm{A}^{*}$ so that it is the minimal sufficient. We presume that no information is lost during this process, following the standard practice of MI estimation [54]. Therefore, we have $I(G^{*};G_{r1})\approx I(\bm{Z}^{*};G_{r1})$ and $I(G^{*};\bm{Y})\approx I(\bm{Z}^{*};\bm{Y})$. For the sake of convenience in the following, we replace $I(\bm{Z}^{*},\bm{Y})$ with $I(G^{*};\bm{Y})$ and $I(\bm{Z}^{*};G_{r1})$ with $I(G^{*};G_{r1})$. The objectives of GaGSL are as follows:

By adding Eq. (15) and Eq. (16), we obtain:

We can simplify Eq. (17) by letting $\beta^{1}=\beta^{2}=\beta$:

where the first term $-I(G^{*};\bm{Y})$ is used to encourage $G^{*}$ to contain maximum information about the labels $\bm{Y}$. The second term $I(G^{*};G_{r1})+I(G^{*};G_{r2})$ encourages $G^{*}$ to contain as little irrelevant information from $G_{r1}$ and $G_{r2}$ as possible, which is label-irrelevant for predicting the target.

The non-Euclidean nature of graph data makes it challenging to estimate the MI in Eq. (18) accurately [55]. Therefore, we introduce a variational upper bound of $-I(G^{*};\bm{Y})$, and use the InfoNCE [56, 57] approximation to calculate $I(G^{*};G_{r1})$ and $I(G^{*};G_{r2})$. First, we examine the prediction term $-I(G^{*};\bm{Y})$.

Proposition 4.1 (Upper bound of $-I(G^{*};\bm{Y})$). Given graph $G$ with label $\bm{Y}$ and $G^{*}$ learned from $G$, we have

where $q_{\theta}(\bm{Y}|G^{*})$ is the variational approximation of the $p(\bm{Y}|G^{*})$. We have the following proof based on Sun et al. [28]:

Since $p(\bm{Y}|G^{*})$ is intractable, let $q_{\theta}(\bm{Y}|G^{*})$ is the variational approximation of the true posterior $p(\bm{Y}|G^{*})$. According to the non-negativity of Kullback Leiber divergence:

Plug Eq. (20) into Eq. (21), then we have

where $H(\bm{Y})$ is the entropy of label $\bm{Y}$, which can be ignored in optimization procedure.

It is not trivial to compute the integral directly. To optimize the objective in Eq. (19), we approximate the integral by Monte Carlo sampling [58] of all training samples, so that we have:

where $\bm{Z}_{i}^{*}$ represents the embedding of node $v_{i}$ and $q_{\theta}(\bm{Y}|G^{*})$ provides the label distribution of the learned graph $G^{*}$, which can be modeled as a classifier. The classification loss, $\mathcal{L}_{cls}$, is chosen to be the cross-entropy loss.

Then, we introduce how to approximate the mutual information $I(G^{*};G_{r1})$ and $I(G^{*};G_{r2})$ using InfoNCE. The InfoNCE loss [32] has been shown to maximize a lower bound of MI. Here, we design a MI calculator. Specifically, for $G_{r1}$ we conduct a layer of GCN [10] followed by a shared two-layer MLP:

where $\bm{H}^{r1}$ and $\bm{H}_{p}^{r1}$ are the node representations obtained from GCN and MLP, respectively. Similarly, we can get the projected embedding $\bm{H}_{p}^{r2}$ and $\bm{H}_{p}^{*}$ of $G_{r2}$ and $G^{*}$, respectively. For readability purposes, we represent the embedding of node $v_{i}$ in graph $G^{*}$ as $\bm{u}_{i}$, and the embedding of $v_{i}$ in graph $G_{r1}$ as $\bm{v}_{i}$. Then, the InfoNCE loss of graph $G_{r1}$ and graph $G^{*}$ is given following the GCA [59]:

where $B$ is the number of nodes that randomly sampled. The pairwise objective for each positive pair $(\bm{u}_{i},\bm{v}_{i})$ is defined as follows:

where $s(\bm{u}_{i},\bm{v}_{i})$ is the cosine similarity and $\tau$ is temperature coefficient. Similarly, we can calculate loss $\mathcal{L}(G^{*},G_{r2})$.

• •

  Compared with other baselines, the proposed GaGSL can obtain better or competitive results on all datasets, which shows that our cleverly designed GSL framework can effectively improve the node classification performance.

• •

  GaGSL demonstrates a significant performance improvement compared to the baseline GCN. Specifically, across all datasets, GaGSL achieves an AUC that is 0.4%-4.8% higher than GCN, an F1-macro score that is 0.4%-11.5% higher than GCN, and an F1-micro score that is 0.5%-11.5% higher than GCN. This observation suggests that GaGSL effectively mitigates the bias caused by a single perspective, enabling it to learn an appropriate graph structure by considering multi-perspective.

• •

  Our performance improvement compared to other GSL methods demonstrates the effectiveness of utilizing GIB for guiding GSL. The learned graph structure contains more valuable information and less noise, making it better suited for node classification tasks. Although GaGSL was the runner-up on the Citeseer dataset, it has an extremely weak disadvantage over the winner, possibly due to data imbalance or the presence of labeling noise.

From the visualization, we can observe that in the perturbed graph structure, there exist noisy connections, as indicated by the higher probability of edges between different communities compared to the original graph structure. These noisy connections degrade the quality of the graph structure, thereby reducing the performance of GNNs. Additionally, we observed that, in contrast to the perturbed graph, the learned graph structure weakens the connections between communities and strengthens the connections within communities.

This observation is expected because GaGSL is optimized based on the GIB principle. The GIB optimization allows GaGSL to effectively capture information in the graph structure that contributes to accurate node classification while constraining information that is label-irrelevant. This optimization process ensures that the learned graph structure is robust against noisy connections and focuses on preserving the most informative aspects of the graph for the task at hand.

As can be observed from Fig. 7, by tuning the value of $\gamma^{1}$ (or $\gamma^{2}$), we can achieve better node classification performance compared to when $\gamma^{1}$ (or $\gamma^{2}$) is set to 0. This precisely demonstrates that our designed structure estimator is capable of capturing the complex correlations and semantic similarities between nodes.

Observing Fig. 8, it is evident that as the value of $\beta$ increases, the performance of GaGSL first rises to a peak and then declines. When $\beta$ is small, $\mathcal{L}_{ML}$ has a minimal impact on the total loss function $\mathcal{L}$, and $\mathcal{L}_{cls}$ dominates the model’s training. At this stage, increasing $\beta$ enhances the influence of $\mathcal{L}_{ML}$, thereby helping the model better constrain the label-irrelevant information from $G_{r1}$ and $G_{r2}$ in $G$, thus improving performance. However, if $\beta$ becomes too large, the model may excessively focus on $\mathcal{L}_{ML}$ and neglect $\mathcal{L}_{cls}$ during training, leading to a decline in performance. Overall, an appropriate value of $\beta$ can effectively balance the influence of $\mathcal{L}_{cls}$ and $\mathcal{L}_{ML}$, thereby enhancing the model’s performance.

From the histograms, it is apparent that the weights of edges between different communities are predominantly concentrated in the first bin (less than 0.1). In contrast, the weights of edges within the same communities are distributed not only in the first interval but also in higher bins. This distribution demonstrates that GaGSL effectively differentiates between inter-community and intra-community edges when assigning weights. Consequently, this ability to assign appropriate weights enhances the performance and robustness of graph representations.