A New Perspective on the Effects of Spectrum in Graph Neural Networks

Based on the generalized formulation of Eq. 6, we conduct correlation analysis on existing graph convolution in the perspective of the graph’s spectrum. We denote $\mathcal{S}=p_{\gamma}(S)$ for simplicity. $\bm{h}\in\mathbb{R}^{n}$ denotes one channel in $f_{\Theta}(H)$. Then the convolution on $\bm{h}$ is represented as $\mathcal{S}\bm{h}$. The cosine similarity between $\bm{h}$ and the $i$-th eigenvector $\bm{p}_{i}$ of $\mathcal{S}$ is

$\alpha_{i}=\bm{h}^{\top}\bm{p}_{i}$ is the weight of $\bm{h}$ on $\bm{p}_{i}$ when representing $\bm{h}$ with the set of orthonormal bases $\bm{p}_{i},i\in[n]$. The cosine similarity between $\mathcal{S}\bm{h}$ and $\bm{p}_{i}$ is

The detailed derivations of Eq. 7 and Eq. 8 are given in Appendix A.

Eq. 8 builds the connection between the cosine similarity and the spectrum of the underlying graph matrix. We say the spectrum is $smooth$ if all eigenvalues have similar magnitudes. By comparing Eq. 7 and Eq. 8, it shows that the graph convolution operation with the unsmooth spectrum, i.e., dissimilar eigenvalues, results in signals correlated (a higher cosine similarity) to the eigenvectors corresponding to larger magnitude eigenvalues and orthogonal (a lower cosine similarity) to the eigenvectors corresponding to smaller magnitude eigenvalues. In the case where 0 eigenvalue is involved in the spectrum, signals would lose information in the direction of the corresponding eigenvectors. In the deep architecture, this problem would further be exacerbated:

We prove Proposition 3.1 in Appendix B. Proposition 3.1 shows that a deeper architecture violates the spectrum’s smoothness, which therefore makes the input signals more correlated to each other. ²²2 Here, nonlinearity is not involved in the propagation step. This meets the case of the decoupling structure where a multi-layer GNN is split into independent propagation and prediction steps (Liu et al., 2020; Wu et al., 2019; Klicpera et al., 2019a; Zhu & Koniusz, 2020; Zhang et al., 2021). The propagation involving nonlinearity remains unexplored due to its high complexity, except for one case of ReLU as nonlinearity (Oono & Suzuki, 2020). Most convergence analyses (such as over-smoothing) only study the simplified linear case (Cai et al., 2021; Liu et al., 2020; Wu et al., 2019; Klicpera et al., 2019a; Zhao & Akoglu, 2020; Xu et al., 2018; Ming Chen et al., 2020; Zhu & Koniusz, 2020; Klicpera et al., 2019b; Chien et al., 2021). Finally, $\textrm{Rank}((\bm{h}_{1},\bm{h}_{2},\dots,\bm{h}_{d}))=1$, and the information within signals would be washed out. Note that all the above analysis does not impose any constraint to the underlying graph such as connectivity.

Revisiting oversmoothing via the lens of correlation issue. In the well-known oversmoothing analysis, the convergence is considered as $\lim_{k\rightarrow\infty}\tilde{A}_{\mathrm{sym}}^{k}H^{(0)}=H^{(\infty)}$ where each row of $H^{(\infty)}$ only depends on the degree of the corresponding node, provided that the graph is $irreducible$ and $aperiodic$ (Xu et al., 2018; Liu et al., 2020; Zhao & Akoglu, 2020; Chien et al., 2021). Our analysis generalizes this result. In our analysis, the convergence of the cosine similarity among signals does not limit a graph to be $connected$ or $normalized$ that is required in the oversmoothing analysis analogical to the stationary distribution of the Markov chain, and even does not require a model to be necessarily $deep$ : it is essentially caused by the bad distributions of eigenvalues, while the deep architecture exacerbates it. Interestingly, inspired by this perspective, the correlation problem actually relates to the specific topologies since different topologies correspond to different spectrum. There exists topologies inherently with bad distributions of eigenvalues, and they will suffer from the problem even with a shallow architecture. Also, by taking the symmetry into consideration, Proposition 3.1(i) shows that the convergence of cosine similarity with respect to $k$ is also $monotonous$. In contrast that existing results only discuss the theoretical infinite depth case, this provides more concrete evidence in the practical finite depth case that a deeper architecture can be more harmful than a shallow one.

Revisiting graph filters via the lens of correlation issue. The graph filter is approximated by a polynomial in the theory of spectral graph convolution (Hammond et al., 2011; Defferrard et al., 2016). Although theoretically, one can approximate any desired graph filter by increasing the order $k$ of the polynomial (Shuman et al., 2013), most GNNs cannot gain improvements by enlarging $k$. Instead, the simple low-pass filter studied by many improvements on spectral graph convolution acts as the practical effective choice (Shuman et al., 2013; Wu et al., 2019; NT & Maehara, 2019; Muhammet et al., 2020; Klicpera et al., 2019b). Although there are studies involving high-pass filters to better process high-frequency signals recently, the low-pass is always required in graph convolution (Zhu & Koniusz, 2020; Zhu et al., 2021; Balcilar et al., 2021; Bo et al., 2021; Gao et al., 2021). This can be explained in the perspective of correlation analysis. As we have shown, the graph convolution is sensitive to the spectrum. A more proper filter to better respond to relevant signal patterns may result in an unsmooth spectrum, making different channels correlated to each other after convolution. In contrast, although a low-pass filter has limited expressiveness, it corresponds to a smoother spectrum, which alleviates the correlation issue.

One can directly apply a smoothing function on the spectrum of $S$, which helps to narrow the range of eigenvalues close to 1 or -1. There can be various approaches to this end, and in this paper, we propose the following eigendecomposition-based method for a symmetric matrix $S=P\Lambda P^{\top}$ ⁴⁴4Although the computation of $S_{\rho}$ requires eigendecomposition, $S$ is always a symmetric matrix and the eigendecomposition on it is much faster than a general matrix.

where $i\in[n]$. $\rho\in(0,1)$, $\lambda^{\rho}=e^{\rho\ln\lambda}$. $S_{\rho}$ serves as the polynomial bases in Eq. 10. Unlike general spectral approaches, $S$ is not required to be a bounded spectrum. It can leverage more bases while alleviating both the diminishing and divergence problems by controlling $\rho\cdot k$ in a small range. Therefore, $S_{\rho}$ can be considered as a basis-augmentation technique as shown in Fig. 1.

There can be other transformations on the spectrum, e.g., $P\bigl{(}\mathrm{Sigmoid}\bigl{(}\Lambda\bigr{)}+\rho\bigr{)}P^{\top}$, which have a similar effect to $S_{\rho}$. Note that the injectivity of $f_{\rho}$ also influences the approximation ability, which is discussed in more details in Appendix C.

Eq. 12 directly operates on the spectrum, which can achieve an accurate control on the range of the spectrum but requires eigendecomposition. To avoid eigendecomposition, we alternatively study the effects of graph normalization on the spectrum. We generalize the normalized adjacency matrix as follows

where $\epsilon\in[-0.5,0]$ is the normalization coefficient and $\eta\in[0,1]$ is the shift coefficient. Widely-used $\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}$ corresponds to $\epsilon=-0.5$ and $\eta=1$.

We prove Proposition 4.1 in Appendix D. Proposition 4.1 extends the results in (Spielman, 2007), showing that the normalization has a scaling effect on the spectrum: a smaller $\epsilon$ is likely to lead to a smaller $\mu_{i}$, while a larger $\epsilon$ is likely to lead to a larger $\mu_{i}$. When $\epsilon=0$, the upper and lower bounds coincide with $\mu_{i}=\lambda_{i}+\eta$.

To further investigate the effects of the normalization on the spectrum, we fix $\eta=0$ and empirically evaluate $\epsilon$ as shown in Fig. 2. When fixing $\epsilon$, $\tilde{D}^{\epsilon}\tilde{A}\tilde{D}^{\epsilon}$ shrinks the spectrum of $A$ with different degrees on different eigenvalues. For eigenvalues with small magnitudes (in the middle area of the spectrum), it has a small shrinking effect, while for eigenvalues with large magnitudes, it has a relatively large shrinking effect. Hence, $\tilde{D}^{\epsilon}\tilde{A}\tilde{D}^{\epsilon}$ can be used as a spectral smoothing method. Also, different $\epsilon$ results in different shrinking effects, which is consistent with the results in Proposition 4.1. Widely-used $\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}$ with the spectrum bounded by $[-1,1]$ may not be a good choice since the diminishing problem. Intuitively, to utilize more bases, we should narrow the range of the spectrum close to 1 (or -1) to avoid both the diminishing and divergence problems in higher-order bases. This can vary from different datasets and we should carefully balance $\epsilon$ and $k$.

Settings. We use the default dataset splits for OGB and ZINC. For TUDatasets, we follow the standard 10-fold cross-validation protocol and splits from (Zhang et al., 2018) and report our results following the protocol described in (Xu et al., 2019; Ying et al., 2018). Following all baselines on the leaderboard of ZINC, we control the number of parameters around 500K. The baseline models include: GK (Shervashidze et al., 2009), RW (Vishwanathan et al., 2010), PK (Neumann et al., 2016), FGSD (Verma & Zhang, 2017), AWE (Ivanov & Burnaev, 2018), DGCNN (Zhang et al., 2018), PSCN (Niepert et al., 2016), DCNN (Atwood & Towsley, 2016), ECC (Simonovsky & Komodakis, 2017), DGK (Yanardag & Vishwanathan, 2015), CapsGNN (Xinyi & Chen, 2019), DiffPool (Ying et al., 2018), GIN (Xu et al., 2019), $k$-GNN (Morris et al., 2019), GraphSage (Hamilton et al., 2017), GAT (Veličković et al., 2018), GatedGCN-PE (Bresson & Laurent, 2017), MPNN (sum) (Gilmer et al., 2017), DeeperG (Li et al., 2020), PNA (Corso et al., 2020), DGN (Beani et al., 2021), GSN (Bouritsas et al., 2020), GINE-VN (Brossard et al., 2020), GINE-APPNP (Brossard et al., 2020), PHC-GNN (Le et al., 2021), SAN (Kreuzer et al., 2021), Graphormer (Ying et al., 2021). Spec-GN denotes the proposed graph convolution in Eq. 9 with the smoothed filter basis by spectral transformation in Eq. 12. Norm-GN denotes the proposed graph convolution in Eq. 9 with the smoothed filter basis by graph normalization in Eq. 13.

Results. Tab. 2 and 3 summarize performance of our approaches comparing with baselines on TUDatasets, ZINC and MolPCBA. For TUDatasets, we report the results of each model in its original paper by default. When the results are not given in the original paper, we report the best testing results given in (Zhang et al., 2018; Ivanov & Burnaev, 2018; Xinyi & Chen, 2019). For ZINC and MolPCBA, we report the results of their public leaderboards. TUDatasets involves small-scale datasets. NCI1 and NCI109 are around 4K graphs. ENZYMES and PTC_MR are under 1K graphs. General GNNs easily suffer from overfitting on these small-scale data, and therefore we can see that some traditional kernel-based methods even get better performance. However, Spec-GN and Norm-GN achieve higher classification accuracies by a large margin on these datasets. The results on TUDatasets show that although Spec-GN and Norm-GN achieve more expressive filters, it does not lead to overfitting on learning graph representations. Recently, Transformer-based models are quite popular in learning graph representations, and they significantly improve the results on large-scale molecular datasets. On ZINC, Spec-GN and Norm-GN outperform these Transformer-based models by a large margin. And on MolPCBA, they are also competitive compared with SOTA results.

We perform ablation studies on the proposed architecture and the filter bases $\tilde{A}_{\rho}$ (by setting $S=\tilde{A}$ in Eq. 12) and $\tilde{D}^{\epsilon}\tilde{A}\tilde{D}^{\epsilon}$ on ZINC. We use “idp” and “shd” to respectively represent the correlation-free architecture (also known as independent filter architecture) in Eq. 9 and the general shared filter architecture in Eq. 6. Both architectures learn the filter coefficients from scratch.

Correlation-free architecture and different filter bases. In Fig. 3, we visualize the learned filters in the correlation-free on three bases, i.e. $\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}$, $\tilde{A}_{\rho}$ and $\tilde{D}^{\epsilon}\tilde{A}\tilde{D}^{\epsilon}$. The visualizations show that each channel indeed learns a different filter on all three bases. $\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}$ has the bounded spectrum $[-1,1]$ that is slightly close to $1$ due to the involvement of self-loop. The filters learn a similar response on all range which corresponds to different frequencies in frequency domain. $\tilde{A}_{\rho}$ and $\tilde{D}^{\epsilon}\tilde{A}\tilde{D}^{\epsilon}$ have the spectrum close to $1$ or $-1$ while the filters learn diverse responses on these areas, which corresponds to more complex patterns on different frequencies. Tab. 4 shows that the correlation-free always outperforms the shared filter by a large margin on all tested bases. Both $\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}$ and $\tilde{D}^{-1}\tilde{A}$ have the bounded spectrum $[-1,1]$ and they have similar performance. $\tilde{A}_{\rho}$ and $\tilde{D}^{\epsilon}\tilde{A}\tilde{D}^{\epsilon}$ narrow the range of the spectrum close to $1$ or $-1$ through completely different strategies, but they have similar performance that is much better than $\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}$ and $\tilde{D}^{-1}\tilde{A}$. This validates our analysis on the filter basis. Meanwhile, $\tilde{A}_{\rho}$ achieves more accurate control on the spectrum, and correspondingly, it slightly outperforms $\tilde{D}^{\epsilon}\tilde{A}\tilde{D}^{\epsilon}$.

Do more bases gain improvements? In Fig. 4, we systematically evaluate the effects of the number of bases on learning graph representations, including $\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}$ and our $\tilde{A}_{\rho}$ with $\rho=1/3,1/6,1/7,1/8$. The shared filter case, i.e. shd$+\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}$ cannot well leverage more bases (a larger $k$) as the MAE stops decreasing at $0.150$ which is also reported by several baselines in Tab. 3. In contrast, both correlation-free cases idp$+\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}$ and idp$+\tilde{A}_{\rho}$ outperform the shared filter case by a large margin and they continuously gain improvements when increasing $k$. The MAE of idp$+\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}$ stops decreasing at the test MAE close to 0.09 and the valid MAE close to 0.11. By replacing $\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}$ with $\tilde{A}_{\rho}$, the best test MAE is below 0.07, and the best valid MAE is close to 0.088. The bases in $\tilde{A}_{\rho}$ are controlled by both $\rho$ and $k$. We use the tuple $(\rho,k)$ to denote a combination of $\rho$ and $k$. By fixing $\rho$, the curves corresponding to $\rho=1/3$ and $\rho=1/6$ show that increasing $k$ gains improvements. By fixing the upper bound of $\rho\times k$ to be 1, $(1/6,6)$ involves 3 more bases than $(1/3,3)$ and outperforms $(1/3,3)$. The same results are also reflected in the comparison of $(1/6,12)$ and $(1/3,6)$. For the comparison of $(1/6,18)$ and $(1/3,9)$, both settings achieve the lowest MAE and the difference is less obvious.

The effects of model depth. Fig.5 shows the performance comparisons between correlation-free and shared filter as depth increases. Each architecture is tested with the default basis $\tilde{D}^{\frac{1}{2}}\tilde{A}\tilde{D}^{\frac{1}{2}}$ and our proposed $\tilde{A}_{\rho}$. We set the same number of bases in all resulting models, and each model is tested with the number of layers (depth) equal to $\{5,10,15,20,25\}$. The results show that the correlation-free can preserve the performance as depth increases. The shared filter cases perform quite unstable and drop dramatically when depth $>20$. Also, across all depths, the correlation-free almost always outperforms the shared filter and has low variance among different runs. In Appendix G, we also test cosine similarities of different layers in a deep model.

Stability. We also found that the correlation-free is more stable in different runs than the shared filter case as reflected in the standard deviation in Tab. 4. This is probably because different channels may pose different patterns, which causes interference among each other in the shared filter case. While the correlation-free well avoids this problem. Also, the results of $\tilde{A}_{\rho}$ and $\tilde{D}^{\epsilon}\tilde{A}\tilde{D}^{\epsilon}$ are more stable than $\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}$ and $\tilde{D}^{-1}\tilde{A}$ in different runs. For $\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}$ and $\tilde{D}^{-1}\tilde{A}$, the difference between the best and the worst runs can be more than 0.02. While for $\tilde{A}_{\rho}$ and $\tilde{D}^{\epsilon}\tilde{A}\tilde{D}^{\epsilon}$, this difference is less than 0.01. More results are given in Appendix H. The instability of $\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}$ and $\tilde{D}^{-1}\tilde{A}$ is probably because learning filter coefficients from scratch without any constraints is difficult to maintain spectrum properties and therefore easily falls into an ill-posed filter (He et al., 2021). In contrast, $\tilde{A}_{\rho}$ and $\tilde{D}^{\epsilon}\tilde{A}\tilde{D}^{\epsilon}$ inherently with smoother spectrum alleviate this problem and make them more appropriate in the scenario of learning coefficients from scratch.