NODE-SELECT: A Graph Neural Network Based On A Selective Propagation Technique

The first step in our method consists of linearly transforming the initial feature vector $x_{i}$ using a matrix $\textbf{W}\in\mathbb{R}^{F^{\prime}\times F}$. The transformed feature vector $\textbf{W}x_{i}$, with reduced dimensionality, has the same cardinality as the embedding output. Subsequently, we estimate a random sensitivity value $\hat{p}_{i}$ as a measure to classify nodes with potentially harmful or useless embedding information. The goal is to use this $\hat{p}_{i}$ value so the network can detect nodes whose omission of information improves the training. We utilize the sum of the neighbours embedding $\mathcal{N}_{(i)}$ to learn this value and then define our selection technique as:

where $\hat{p}_{i}=\sigma\left(\textbf{W}_{0}\left(\sum_{j\in\mathcal{N}_{(i)}}{\textbf{W}x_{j}}\right)\right)$, $\textbf{W}_{0}\in\mathbb{R}^{1\times F^{\prime}}$ is a weight matrix, $T$ a determined threshold, and $\sigma$ a non-linear transformation. Thus, $\hat{p}_{i}$ can be interpreted as a normalized signal, ranging between 0 and 1, allowing the model to make its selection with respect to the entire graph. Namely, the learnable choice of whether or not a node needs to propagate its information is made on a global scale such that an un-selected node can no longer affect any other nodes.

Once a node’s selection $S(v_{i})$ is learned, we allow the layer to aggregate the embedding information only from its selected neighbors. In other words, we want the model to keep learning embeddings even with the cancellation of some nodes’ propagation. The simplified form of this selectively aggregated information is simplified below:

where $\alpha_{i}$ defines the propagation weight for each selected node. This propagation weight is calculated by linearly transforming the concatenated summed and selectively aggregated neighborhood embeddings; such that $\alpha_{i}=\sigma\left(\textbf{W}_{1}\left(\sum_{j\in\mathcal{N}_{(i)}}{\mathcal{S}(v_{j})\cdot\textbf{W}x_{j}}\parallel\sum_{j\in\mathcal{N}_{(i)}}{\textbf{W}x_{j}}\right)\right)$ and $\textbf{W}_{1}\in\mathbb{R}^{1\times 2\cdot F^{\prime}}$ is a weight matrix. Since our model learns the selection $\mathcal{S}(v_{j})$ with respect to the entire graph (global selection), $\alpha_{i}$ thereby corresponds to the hard attention for a selected node $v_{i}$. Also in our experiments, applying this global weight $\alpha_{i}$ resulted in higher ( $\geq$ 3%) accuracy performance than adapting a weight relative to each node’s neighbor (i.e. $\alpha_{ij}$ as in GAT).

Therefore, for layer $l$ in our method, the simplified form of its function $f_{\theta}^{(l)}$ takes the form of:

where $\textbf{W}x_{i}$ is the originally transformed feature vector and $A_{i}$ the pooled embedding information from just the selected nodes.

The last step consists of simply summing the embedding information from all layers. Given $L$ layers, the final embedding output for a given is the summation of all $L$ independently learned embeddings $f^{(l)}_{\Theta}(x_{i})$. This summation is described below.

To assess the performance of our proposed model, we conduct two sets of experiments using a total of 8 benchmark datasets. In the first experiment, we utilize all 8 datasets: Cora, CiteSeer, PubMed, Cora Full, Coauthor CS, Coauthor Physics, Amazon Computers, and Amazon Photo. Cora, CiteSeer, and PubMed contain relational data on academic papers [20, 21]. Datasets Coauthor CS (Co-CS) and Coauthor Physics (Co-P) are co-authorship datasets from the Microsoft Academic Graph [16]. Lastly, Amazon Computers (Amz-C) and Amazon Photo (Amz-P) are graph datasets defining segments of the Amazon product categories graphs. For each dataset, we randomly split the nodes so that the training, validation, and testing sets follow a ratio of 20-20-60 percent. This split is repeated for 10 randomly chosen seeds which are used in each model experiment.

In the second experiment, we only use Cora, CiteSeer, and the PubMed datasets which we modify by adding noise data into their graphs. We increase the size of each graph by 10 and 25% using pseudo vertices. We attribute each pseudo vertex a feature vector from a standard normal distribution, a random label, and random neighbors from the original graph. We follow the same 20-20-60 splitting ratio as in the first experiment but afterwards remove the pseudo vertices from the testing set. This split is repeated for 5 randomly chosen seeds. Details of the Datasets are provided in Table 1.

We compare our proposed method to 6 GNN variants selected for either their robust performance, contrasting node sampling method, or both. These baselines include DropEdge, FastGCN, GAT, GCN, GraphSAGE, and Node2vec [9, 33, 5, 4, 6, 34]. Given that each framework performs differently under various training dynamics, we perform random hyper-parameter and only report results from the best performing models with respect to the validation set. We apply a fixed dropout rate of 0.5 after the GNN layers and use Adam as optimizer[35, 36]. We implement all the models using Pytorch and the library of Pytorch-Geometric [37, 25]. The hyper-parameters from the best performing models are provided in Table 1 of the Appendix.

Table 2 displays the average accuracies over the 10 random splits from the first experiment. As seen, NODE-SELECT consistently matches or outperforms the performance by the baselines by up to 1.4 percentage points. Table 3 lists the average classification accuracy of 5 random splits for the second experiment with noise introduced to the training. Once a GNN model is introduced to a considerably amount of noise information, the results demonstrate that the accuracy significantly drops [31]. Nevertheless, our NODE-SELECT is only marginally affected by the presence of noise information whereas the baselines considerably are. As shown in the results, our proposed method particularly stands out in these noise experiments by outperforming the other baselines by up to 20 percentage points. Simply put, the resilient ability of our network to be affected by noise information is due to the used selection mechanism which allows the network a direct control of blocking nodes propagating them.

Compared to the traditional sequential stacking of GNN layers, NODE-SELECT adopts the approach of stacking its layers in parallel. Based on our experiments, we have found that the parallel stacking of these selective layers yielded better and more stable results. Figure 2 illustrates a comparative study of these two stacking options. As demonstrated, the parallel setting proves to be more beneficial with its results reaching higher accuracy and lower variance. The sequential layout forces a layer $l$ to depend on the set of selected nodes $\mathcal{V}_{s}^{(l-1)}$ of a previous layer $l-1$. In the rare cases that the composition of $\mathcal{V}_{s}^{(l-1)}$ is not completely suitable to the weights of layer $l$, the model may result in a much lower performance; thus observing a higher variance and reduced accuracy. The parallel layout removes this dependence issue by stacking its layers side-by-side and having them learn independently as in the ensemble method [38].

Because of the parallel configuration, any given NODE-SELECT layer operates independently. Each layer separately makes a node selection that yields to a node embedding. Figures  3 and  4 contrast these layers’ differences in terms of their predicted sensitivity value, accuracy score, and proportion of selected nodes for experiments on the Cora and Pubmed Dataset. In Figure  3, the $\hat{p}_{i}$ values of 3 nodes from the training set are predicted by 3 parallel layers on a Cora experiment. Because of the layers’ independence, a node’s $\hat{p}_{i}$ values from distinct layers are also independent. The independence is illustrated with Layer-1 learning to output high values above the threshold for node 1728 but conservative values node 1601, yet Layer-3 does the opposite. In Figure  4, the accuracy results ranging from 75 to 85% for the 10 layers used in a trained model are displayed with the selection percentage ranging between 22 and 100 % (in blue). For instance, layer 6 reached the highest accuracy of 85% with a selection percentage of 84% while layer 3 had the third lowest accuracy score of 80% with 100% selection proportion. As demonstrated in the figure, a layer’s accuracy performance is not correlated to its proportion in size of selected nodes. However, based on our experiments, we found that a layer that more effectively filters the most noise-propagating nodes is more likely to reach a higher accuracy. We also see the reported accuracy (in green) of the final output, obtained by summing the embeddings of the 10 layers, being much higher than any layer’s individual accuracy score.

A direct benefit of stacking our layers in parallel is that our method is very scalable. Because our method is composed of ensembled 1-layer GNN, its memory usage is very effective. For example, the number of trainable parameters can be estimated with $L\left(F^{\prime}(3+F)\right)$; with $L$ describing the number of layers, $F^{\prime}$ the output dimension, and $F$ the input dimension. Figure 5 contrasts how NODE-SELECT scales with larger graphs when compared to some baseline frameworks. As demonstrated, NODE-SELECT scales to larger graphs comparably to GCN and much better than GAT, GraphSAGE, and DropEdge. As the graph gets larger, the amount of memory required for the learning to take place also increases and the challenge of being scalable affects many current GNN [1, 2]. Nonetheless, NODE-SELECT adapts very well to larger graphs.

In contrast to other GNNs, NODE-SELECT only has two configurable parameters that affect the model’s performance. The parameter $L$ guides the model’s fitting behavior while $T$ guides its selection mechanism. In our study, we found that any arbitrary number of layer leads to the a good performance. However, depending on the size and properties of the graph, too few layers may cause the model to under-fit while too many to over-fit. Table 4 provides a simple illustration of the effect of increasing the parameter $L$. Using only 1 layer causes the model to under-fit with an accuracy of 94%. Using 20 or more (100) layers causes the model to over-fit with accuracy scores that are below the well-fit models trained with 5 or 10 layers. Particularly, as a NODE-SELECT model uses more layers (past the optimal number), each individual layer becomes weaker. This decrease in performance results from the fact that each layer learns the patterns pertaining to a specific region in a graph and thus generalizes poorly.

A NODE-SELECT model will adjust its weights so as to retain as much embedding information as possible during training. Therefore, a NODE-SELECT only begins to cancel nodes when the threshold parameter is not too small; for instance above $T$ value of $0.3$. However, using low values for the threshold results in performance that is only marginally lower than with moderate threshold. Also, the selection mechanism is a lot more effective when the graph is not small. In our experiments with smaller graphs (Cora and Citeseer), an effective layer only cancelled a minimal number of nodes (i.e. between 0 and 10%). Figure 6 displays the effect of changing $T$ on the model accuracy. Using the parameter $T$ allows the model to conservatively remove subset of nodes that are not needed to lower its loss. Using a $T$ value in a range of $0.3\leq T\leq 0.49$ gives the best results in terms of both accuracies and node cancellations. However the application of a large $T$ value leads the model to cancel too many nodes, thereby losing crucial information from potentially important nodes.

There is an optimal number of layers that should be included in a NODE-SELECT model. Exceeding this number of layers increases the likelihood the model will over-fit while using less causes it to underfit. Figure  7 displays the impact of surpassing the optimal number of layers (3-5) in the Cora dataset. At the optimal number of layers, the model’s performance (average accuracy for 10 random splits) is at its peak. However, the increase in the number of layers prior to reaching that optimal number increases the performance of the NSGNN model. As more layers are added, NODE-SELECT captures the relationship patterns more efficiently and thus its predictive performance improves. The latter increase in performance is due to the specialization of the layers, which together complement each the other’s inaccuracies. Figure  9 illustrates the effects of the addition of layers (in green) on the overall model (in red) on the Amazon-Computers dataset. In Figure  8, the model’s accuracy reaches an accuracy score of about 82% while best layers obtains accuracy at exactly 70%. As 5 more layers are added in the illustration of figure  9, the highest accuracy reached by any layers drops 62% while the NSGNN accuracy improves to 87%.

Compared to the traditional sequential stacking of layers, NODE-SELECT adopts the well-known ensemble method for which each layer independently learns the nodes embedding based on a primary selection of propagating nodes. In our study, we observed that the average cosine-similarity of these embeddings per node is always high ($\geq 0.85$). Henceforth, by combining these sets of independent yet correlated embeddings, the model subsequently learns a final embedding that is more precise. Figure  10 resulting embeddings from 10 independent layers for the first node on the Co-P experiment. Noticeably, the majority of the embeddings are very similar. Based on our experiments, we deduced that not all of the layers’ embeddings need to match for a good node prediction. As long as there is a general harmony in a sufficient number of layers, the model will depend on these more frequent harmonious embeddings to form its final embedding.

Beyond the simple selection mechanism that we presented in our paper, we also implemented a sequence mechanism that is more appropriate for smaller graphs. Mainly, this additional sequence mechanism allows each selected node to propagate information up to its $q$-hop neighbours. Figure  11 provides an illustration of this sequence mechanism.

Starting with the subset of propagating nodes, a layer sequentially learns a global weight coefficient to perform the message-passing operation across $Q$-hop neighborhoods. For each $q=0,1,..,Q\text{-}1$ depths, a corresponding weight coefficient $\alpha_{i}^{(q)}$ is calculated:

, in which $\textbf{W}_{2}\in\mathbb{R}^{1\times(F+Q)}$ denotes a learnable matrix, $y_{i}^{(q)}$ the $q$th updated feature vector of $v_{i}$, and $q^{*}$ the one-hot encoded vector of the depth $q$. Upon learning this selection-depth adapted coefficient, a node’s feature updates as:

. Lastly, we implement a noise layering mechanism so as to regulate the amount of noise information that is being learned though the message-passing operations. Our motivation for the latter mechanism comes from the assumption that there exists a minority of nodes that do not need to aggregate their neighbors’ information. We adapt our updating operation so that the layer tries to maintain an appropriate balance between learned neighboring information and each node’s own feature. Therefore, after $Q$ updates, the layer then calculates a final global coefficient $c_{i}$ though the use of a matrix $\textbf{W}_{3}\in\mathbb{R}^{1\times(2F)}$,