Spatio-Temporal Sparsification for General Robust Graph Convolution Networks

Given an undirected graph $G=(V,E,X)$, where $V=\left\{v_{1},v_{2},...,v_{n}\right\}$ is a set of nodes with $|V|=n$, $E\subseteq V\times V$ is a set of edges that can be represented as an adjacency matrix $A\in{{\left\{0,1\right\}}^{n\times n}}$, and $X=\left[{{x}_{1}},{{x}_{2}},\ldots,{{x}_{n}}\right]^{T}\in{{\mathbb{R}}^{n\times d}}$ is a feature matrix with $x_{i}$ denoting a feature vector of node $v_{i}\in V$. $\text{C}=\left<{{c}_{1}},{{c}_{2}},\ldots,{{c}_{n}}\right>$ is the class label vector with $c_{i}$ representing the label value of node $v_{i}$.

In the paper, we focus on GCNs for node classification. In particular, we will consider the well established work (Kipf and Welling 2017). As a semi-supervised model, GCN can learn the hidden representation of each node. The hidden vectors of all nodes in the $l+1$ layer can be represented recursively by the hidden vectors of the $l$ layer as follows.

where ${\tilde{A}}={A}+{{I}_{n}}$, $W^{\left(l\right)}\in{\mathbb{R}}^{d^{(l)}\times d^{(l+1)}}$yl denotes the learnable weight matrix at layer $l$, ${{\tilde{D}}_{i}}=\mathop{\sum}_{j}{{\tilde{A}}_{ij}}$, and $\sigma\left(\cdot\right)$ is an activation function, such as ReLu. Initially, $H^{(0)}=X$.

First, we will describe the mapping from the dense space to the high-dimensional space, which can be simply realized through replacing the parameter matrix $W^{(l)}$ in Eq. (1) with a high-dimensional version $W^{(l)}_{h}$, as shown in Eq. (2).

where $W^{\left(l\right)}\in{\mathbb{R}}^{d^{(l)}\times d_{h}}$. Compared to $d^{(l+1)}$ (the second dimension of $W^{(l)}$ in Eq. (1)), $d_{h}$ (the second dimension of $W^{(l)}_{h}$) is much larger. In Section 5, we will illustrate the underlying reason for high dimensional space through experiment evaluation, which will show that the low dimension can significantly reduce the performance of the proposed ST-SparseGCN. Thus, the high-dimensional space is one of the key factors for the effectiveness of the propose ST-SparseGCN. It is worth to note that $d_{h}$ is the same for all layers except the input layer, i.e., $\forall l\geq 1$, $H^{(l)}\in{\mathbb{R}}^{n\times d_{h}}$ and $H^{(0)}=X\in{\mathbb{R}}^{n\times d}$.

Next, we will formally introduce the definition of spatial sparsity as follows.

Def. 1 implies that the non-zero elements of a spatial sparse vector should be much less than the vector dimension. In the following, we will adopt $s_{i}$ to denote the sparse version of $h_{i}$. Also, $S=[s_{1},s_{2},\ldots,s_{n}]^{T}$ represents the sparse matrix consisting of the sparse vectors from all nodes.

Although spatial sparsity can ensure the feature sparsity of individual nodes, it cannot guarantee that individual features are sparse, i.e., the number of nodes activated on any given feature is much less than the total number of nodes. For example, in Fig 2, feature $j$ is not temporally sparsed after spatial sparsification because too many nodes activate feature $j$. Through temporal sparsification, the non-zero elements associated with feature $j$ will be gradually reduced.

This new type of sparsity can be illustrated through simple calculation. If $\forall t\in\left[{1,2,\ldots,{\rm{T}}}\right]$, where $T$ is the total number training epochs, and $\forall v_{i}\in V$, $||s^{t}_{i}||_{0}\leq k$, then $||S^{t}||_{0}\leq n\times k$, where $s^{t}_{i}$ and $S^{t}$ represent $s_{i}$ and $S$ at epoch $t$, respectively, as there are $n$ nodes in total. Thus, on average, each feature will be on duty (i.e., non-zero values) for at most $\frac{n\times k}{d_{h}}$ nodes, because there are $d_{h}$ features in total. Since $k\ll d_{h}$ according to Def. 1, it can be concluded that $\frac{n\times k}{d_{h}}\ll n$, where $n$ is actually the maximal number of non-zero elements for any feature at epoch $t$. Thus, from the feature’s perspective, if the duty cycle (in terms of non-zero elements) for all features needs to be balanced, it also shows the sparsity phenomenon.

The underlying reason for the necessity of the duty-cycle balance lies in that, if a feature is on duty for too many nodes, this feature may show the Mathew effect, i.e., the more a feature is used at the current epoch, the more oftern it will be used in the following epochs. This Mathew effect can be took advantaged by the adversarial attacker through manipulating the heavily used feature.

Thus, it is desirable to introduce temporal spasity so that the duty cycle of features can be balanced along with the training epochs. To formally define temporal sparsity, we introduce $s_{*j}^{t}$ to denote the set of nodes that utilizes the $j$-th feature, which is equal to the $j$-th column of $S^{t}$, i.e., $s_{*j}^{t}=<s_{1j}^{t},s_{2j}^{t},\ldots,s_{nj}^{t}>$.

Based on the above description, the temporal sparsity concerning the $j$-th feature can be formally defined as follows.

In our ST-SparseGCN model, the spatial sparsification is implemented through TopK, which simply selects the top $k_{\alpha}=\lfloor\alpha\cdot d_{h}\rfloor$ features for any $h_{i}\in H$, where $\alpha\in\left(0,1\right)$ is the spatial sparse ratio. TopK can be formalized as follows.

where $z_{i}$ represents the set of features with the largest $k_{\alpha}$ values from $h_{i}$. In another words, TopK will keep the values of those top $k$ features and set all the other features to zero.

Through replacing the activation function in Eq.(2) with TopK, we can implement a spatial sparsed GCN, which can be formalized through the equation shown in Eq.(4).

where $S^{(0)}=X$ initially. It is worth to note that the TopK function in Eq.(4) is a matrix version of the TopK fuction in Eq.(3). This matrix version selects the top $k_{\alpha}$ features for each node $v_{i}$ independently.

The spatial sparse ratio $\alpha$ in the TopK function is a hyperparameter to be adjusted. Intuitively, on one hand, small $\alpha$ implies less non-zero features, which might seriously compromise the generalization capability of the proposed model, because the possible vectors that can be represented in the high-dimensional space will become less along with smaller $k_{\alpha}$. On the other hand, large $\alpha$ may compromise the model robustness. The appropriate value of $\alpha$ will be evaluated in Section 5.

TopK VS. ReLu. In ST-SparseGCN, the ReLu function in GCN has been replaced by the TopK function. The effect of the replacement can be illustrated through Fig. 3(a), where the GCN coupled with TopK and the GCN with ReLu are compared in terms of the ratio of activated neurons during the training process. From Fig. 3(a), it can be observed that TopK greatly reduces the number of activated neurons. TopK and ReLu can also be compared through the funciton curves as shown in Fig. 3(b), from which it can be observed that it is more difficult for a neuron to be activated through the TopK activation function.

The Cost of Robustness. (Xiao, Zhong, and Zheng 2020) has proved that the computational complexiy of TopK is asymptotically $O(N)$, which is the same as ReLu. However, it takes more time for TopK to converge in experiments, which might originate from the spatial sparsity. Due to the spatial sparsity, only a small number of neurons can be activated, which implies that the gradient update covers only a small number of neurons in each epoch. Nevertheless, the computing cost of TopK can be reduced through the computing optimization of sparse matrix.

At the first glance, it seems that temporal spasity can be realized through applying the TopK function as follows: $TopK(s^{t}_{*j},\frac{n\times k}{d_{h}})$. However, this may reduce the number of non-zero features for certain nodes, which might compromise the generalization capability as discussed previously. Furthermore, it may cause a sudden discontinuities in the model output.

To avoid the above issues, we propose an attention-based temporal sparsification mechanism, where at any epoch $t$, for any node $v_{i}$, its feature $j$ is assigned an attention value $b_{ij}^{t}$. This attention value will be adaptively adjusted according to the historical sparsity information of feature $j$, namely, $||s^{t^{\prime}}_{*j}||_{0}$ ($t^{\prime}\in\{1,2,\cdots,t-1\}$). Then, the adjusted attention value $b_{ij}^{t}$ is used as a weight to adjust the corresponding feature $j$ of node $v_{i}$ in the spatial sparsified GCN hidden representation (namely $S^{(l)}$, as defined in Eq.(4)) so that the feature with larger sparsity value will reduce its chance to be selected by the TopK function.

Concretely, in each epoch $t$, the attention mechanism updates the attention value of each node $v_{i}$’s feature $j$ based on the integration of the historical sparsity $\hat{s}_{ij}^{t}$ and the current sparsity of the feature (i.e. $||s^{t}_{*j}||_{0}$). If integrated feature sparsity is higher than the sparsities of the other features, its attention value $b^{t}_{ij}$ will be reduced accordingly.

Formally, the integrated sparsity of any feature $j$ associated with node $v_{i}$ is updated as shown in Eq.(5).

where $\hat{s}_{ij}^{t}$ is the historical sparsity of node $v_{i}$’s feature $j$ before epoch $t$ and $\tau$ is a hyper parameter that controls the decay rate of historical information. Initially, $\hat{s}_{ij}^{0}=0$.

Based on the integrated sparsity $\hat{s}_{j}^{t}$, the attention value $b^{t}_{j}$ is updated through a smooth exponential function as shown in Eq.(6)

where $\gamma$ is a hyper parameter. From Eq.(6), it can be observed that, the larger the integrated sparsity $\hat{s}_{ij}^{t}$, the smaller the updated attention value $b_{ij}^{t}$, because the smooth exponential function $\exp(\cdot)$ is a monotonically decreasing function.

From Eq.( 5), it can be observed that the historical sparsity $\hat{s}_{ij}^{t}$ is actually independent of nodes, and so does the attention value $b^{t}_{ij}$, according to Eq.(6). Thus, Eq.( 5) and Eq.(6) can be computed only once for all nodes, from which we can obtain feature $j$’s historical sparsity and attention values for all nodes, namely vector $\hat{s}_{*j}^{t}$ and vector $b^{t}_{*j}$, respectively.

From $b^{t}_{*j}$, $j\in\{1,\cdots,d_{h}\}$, we can contruct the attention mask matrix ${\cal B}^{t}$ as follows.

Based on the attention mask matrix, the proposed ST-SparseGCN can be formalized through Eq. (8).

Initially, $\mathcal{B}_{t}^{(0)}=0$ and $S^{(0)}=X$. Eq. (8) can be desribed as follows. At epoch $t$, in the $(l+1)$-th layer, sparse matrix $S^{(l)}$ first multiplies the attention mask matrix $\mathcal{B}_{t}^{(l)}$ element-wisely to temporal sparsify the feature space, so as to mitigate the Mathew effect. The sparsified matrix will be fed as input into GCN for information propagation among nodes. The GCN output is further spatially sparsified thourgh TopK activation fucntion. In the end, $S^{(L)}$ is passed to a fully connected layer with the softmax activation function to predict labels $Y$.

Baselines. To evaluate the robustness and effectiveness of ST-SparseGCN, experiments are performed on the deep learning framework PyTorch (Steiner et al. 2019) and the GNN extension library PyG (Fey and Lenssen 2019). The proposed defense model (ST-SparseGCN) is compared with four baselines, three of which are representative graph defense methods, on the task of node-level semi-supervised classification as follows.

• •

  GCN(Kipf and Welling 2017):GCN proposes to simplify the graph convolution using only the first order polynomial, i.e. the immediate neighborhood. By stacking multiple convolutional layers, GCN achieved the state-of- the-art performance in clean datasets.

• •

  GCN-Jaccard(Wu et al. 2019a): GCN-Jaccard utilizes the Jaccard similarity of features to prune perturbed graphs based on the assumption that the connected nodes usually show high feature similarity.

• •

  GCN-SVD(Entezari et al. 2020): GCN-SVD proposes a low-rank representation method, which can approximate the original node representation with a low-rank representation, so as to resist the adversarial attacks.

• •

  RGCN(Zhu et al. 2019): RGCN aims to defend against adversarial edges with Gaussian distributions as the latent node representation in hidden layers to absorb the negative effects of adversarial edges.

We implement the above baseline methods with refering to the implementation in DeepRobust(Li et al. 2020).

Attacker models. To validate the defensive ability of our proposed defender, we choose four representative GCN attacker models.

• •

  DICE(Waniek et al. 2018):DICE randomly selects node pairs to flip their connectivity (i.e., the removal of the existing edges and the connection of non-adjacent nodes).

• •

  Mettack(Zügner and Günnemann 2019): Mettack aims at reducing the overall performance of GNNs via meta learning. We used the attack method Meta-Self.

• •

  PGD(Xu et al. 2019b):PGD is projected gradient descent topology attack to attacking a pre-defined GNN.

• •

  Min-Max(Xu et al. 2019b): Min-max is min-max topology attack to attacking a re-trainable GNN. The minimization is optimized using the PGD method and the maximization aims to constrain the attack loss by retraining $W$.

Parameter Setting. The following common parameters are the same for ST-SparseGCN and the baselines. The number of GCN layer is 2 and the training epochs is 200. The selected optimizer is Adam (Kingma and Ba 2015) with a fixed learning rate of 0.01. The other hyperparameters in baselines are closely followed the benchmark setup. And the hyper parameters in the ST-SparseGCN model are adjusted based on the validation set to achieve the best robust performance. Parameter sensitivity of ST-SparseGCN will be analyzed in Section 5. The final results of all experiments are obtained by averaging 5 repeated experiments. Our experiments are performed on NVIDIA RTX 2080Ti GPU.

Datasets. ST-SparseGCN is evaluated on three well-known datasets: Cora, Citeseer and Polblogs (Sen et al. 2008), where nodes represent documents and edges represent citations. The sparse bag-of-words feature vector associated with each node is the model input. Table 1 enumerates the statistics information of the datasets. The same training set, test set, and validation set from the same data set is used to fairly evaluate the performance of different models.

In order to properly measure the impact of the perturbation, we first evaluated the performance of ST-SparseGCN and all baselines on different clean datasets. The average accuracy with standard deviation is enumerated in Table 2, which indicates that ST-SparseGCN can achieve excellent performance on clean data sets. Compared to four baselines, the superiority of ST-SparseGCN may come from the generalization capability of the temporal sparsification on the feature space.

In the section, we evaluate the overall defense performance of the proposed ST-SparseGCN by comparing it with various defense methods under different adversarial attackers along with different perturbation sizes.

Perturbation Size. For each attacker, we increase the perturbation rate from 0 to 0.25 with a step size of 0.05. In general, the defense performance decreases along with the increase of the perturbation size. In order to concisely present the experiment results, we define a new metric to evaluate the defense performance, termed dropping rate (DR) as shown in Eq.(9).

where $\widehat{Acc}$ is the accuracy of GCN on clean/original graph. Dropping rate characterizes the defense performance by measuring the integration of the performance degeneration caused by attacker models and the performance remedy from the defense methods. The smaller the dropping rate, the better the defense methods. We use the mean dropping rate(mDR) to describe the overall defense performance along with different perturbation sizes.

Hybrid Defense. To illustrate that the proposed ST-SparseGCN defense methods can be complementary to the existing defense methods, we propose to integrate ST-SparseGCN with two existing defense models, namely, GCN_Jaccard and GCN_SVD, which improve GCN robustness through data preprocessing. The two integrated defense models are called ST-SparseGCN_Jaccard and ST-SparseGCN_SVD, respectively.

Experimental Results. The experiment results on the Cora and Citeseer datasets are enumerated in Table 3. Due to space limitation, the experiment results on the Prolblogs dataset are not included, but illustrated in Fig. 4 instead.

From Table 3, we can make the following observations: (i) the proposed ST-SparseGCN defense model or its variants (ST-SparseGCN_Jaccard and ST-SparseGCN_SVD) achieve the best defense performance under varoius attackers on all datasets, as ST-SparseGCN constructs a robust feature space in each GCN layer; (ii) the hybrid defenders (ST-SparseGCN_Jaccard and ST-SparseGCN_SVD) can improve defense performance compared with the corresponding baselines (namely GCN_Jaccard and GCN_SVD) alone in most cases, which implies that our proposed defender ST-SparseGCN is complementary to the existing defenders, because ST-SparseGCN intends defend the adversarial attacks from the perspective of sparsification in feature space, which is complementary to most of the existing defense methods; (iii) none of the existing non-hybrid graph defenders (including the ST-SparseGCN alone) can perform best under all attackers on all datasets. This phenomenon may originate from the fact that the success of adversarial attacks comes from various aspects of the GCN model. Thus, the hybrid defense models may deserve further exploration.

Fig. 4 summaries the performance results under different attackers along with varied perturbation sizes on the Polblogs dataset. The results show that ST-SparseGCN again consistently achieves better performance than all the baselines, which demonstrates the superiority of the proposed ST-SparseGCN model. The experiment results shown in Table 3 and Fig. 4 have illustrated that our defender can improve defense performance under all attackers on all datasets. It is worth to note that ST-SparseGCN does not rely on any prior knowledge of any particular adversarial attack method. The advantage of ST-Sparse might lie in that it can construct a robust feature space, which can be effectively against various adversarial attacks.

In the section, we compare the generalization performance and robustness of ST-Sparse and Dropout through experiments. Table 4 demonstrates both dropout and ST-Sparse can improve the generalization ability of the model, and ST-Sparse is even better. In terms of robustness, ST-Sparse performs better than Dropout in the face of attacker. In addition, Dropout combined with ST-Sparse will damage the performance of the model. This phenomenon may be the random inactivation of Dropout will damage the ability to preferentially select features in ST-Sparse.

We conduct several experiments on datasets-model pairs mentioned above to report the runtime of a whole training procedure for 200 epochs obtained on a single NVIDIA RTX 2080 Ti (cf. Fig. 5). Thanks to the ability to quickly process sparse data in the PYG framework, the time overhead between models is basically the same. Among them, GCN_Jaccard takes the most time because its data preprocessing process is very slow.

In this section, we evaluate the maginal effect of the temporal sparsity, spatial sparsity, and the dimension $d_{h}$ on the accuracy and robustness of ST-SparseGCN. The performance evaluated on clean and perturbed datasets are shown separately. Due to space limitation, we only show the experiment results on the Cora dataset under the Mettack with a perturbation size of 0.05. The experiments on the other datasets exhibit similar patterns, which are included in the supplementary material.

In the experiments, the extent of the spatial sparsity is controlled by the spatial sparse ratio $\alpha$. Fig. 6(a) shows that the influence of the temporal sparsity and the Mettack along with the increasing sparse ratio $\alpha$ from $0.02$ to $0.5$, where T-Sparsity and Perturbed represent the temporal sparsity and the perturbation from the Mettack, respectively. From Fig. 6(a), by comparing the performance of the models with and without temporal spasity, it can be osberved that temporal sparsity can effectively improve the ST-SparseGCN’s classification performance on both the clean graphs and the perturbed graphs. This illustrates the benefits of the temporal sparsity, which can not only increase the model’s generalization capability (from the performance improvement on the clean graphs), but also improve the model’s defense performance (from the performance improvement on the perturbed graphs).

It can also be inferred from Fig. 6(a) that, when the spatial sparsity ratio $\alpha$ varies from a small value (0.02) to a relative larger value (0.08), both the models with and without temporal sparsity show significant performance improvement. Thus, this illustrates the necessity of spatial sparsity. Moreover, when $\alpha$ is larger than 0.08, the accuracy of the ST-SparseGCN remains basically unchanged. However, a very small $\alpha$ will degrade the performance, probably because the number of non-zero features is not enough to distinguish different categories for node classificaiton task.

Fig. 6(b) illustrates the impact of the dimension $d_{h}$ on the performance. It is can be observed that whether it is the clean dataset or the perturbed dataset, the performance is drastically reduced when $d_{h}$ reduced to a small value. On the other hand, when $d_{h}$ increases to a certain level, the performance almost remains stable. This illustrates that there exists an appropriate value for $d_{h}$. The results also illustrate that the high-dimensional space can enable the GCN model to have more robust performance in case of the perturbation incurred by the attackers.