Node-Variant Graph Filters in Graph Neural Networks

Of particular interest is the seminal work by Mallat [15, 16], concerning discrete-time signals and images, that argues that the improved performance of convolutional neural networks (CNNs) over linear convolutional filters is due to the activation functions. Concretely, nonlinear activation functions allow for high-frequency content to be spread into lower frequencies, where it can be processed in a stable manner—a feat that cannot be achieved by convolutional filters alone, which are unstable when processing high frequencies.Leveraging the notion of graph Fourier transform (GFT) [17], these results have been extended to GNNs [10], establishing that the use of functions capable of creating low-eigenvalue frequency content allows them to be robust to changes in the graph topology, facilitating scalability and transferability [18].

While nonlinear activation functions play a key role in the creation of low-eigenvalue frequency content, it is not possible to know the exact shape in which this frequency content is actually generated. Node-variant graph filters (NVGFs) [19], which essentially assign a different filter tap to each node in the graph, are also able to generate frequency content. Different from nonlinear activation functions, this frequency content can be exactly computed, given the filter taps. Thus, by learning or carefully designing these filter taps, it is possible to know exactly how the frequency content is being generated.

The NVGF is a linear filter, which means that replacing the nonlinear activation functions with NVGFs actually renders the whole architecture a linear one. Therefore, by comparing the performance of this architecture to that of a traditional GNN, it is possible to isolate the role of frequency creation in the overall performance of the architecture, from that of the nonlinear nature of mappings. The contributions of this paper can be summarized as follows:

• (C1)

  We introduce NVGFs as a means of replacing nonlinear activation functions, motivated by their ability to create frequencies. We obtain closed-form expressions for the frequency response of NVGFs as a function of the filter taps.

• (C2)

  We prove that NVGFs are Lipschitz continuous with respect to changes in the underlying graph topology.

• (C3)

  We put forth a framework for designing NVGFs.

• (C4)

  We propose a GNN architecture where the nonlinear activation function is replaced by a NVGF. The filter taps of the NVGF can be either learned from data (Learn NVGF) or obtained by design (Design NVGF). The aim of the resulting architecture is to decouple the role of frequency creation from the nonlinear nature of the GNN.

• (C5)

  We investigate the problem of authorship attribution to demonstrate, both quantitatively and qualitatively, the role of frequency creation in the performance of a GNN, and its relationship to the nonlinear nature of traditional architectures.

In essence, we show that nonlinear activation functions are not strictly required for creating frequencies, as originally thought in [15, 10], but that linear NVGF activation functions are sufficient. Furthermore, we demonstrate that this frequency content can be learned with respect to the specific problem under study. All proofs, as well as further simulations, can be found in the appendices. Code can be found online at http://github.com/fgfgama/nvgf.

NVGFs are first introduced in [19] to extend time-variant filters into the realm of graph signals. In that work, NVGFs are used to optimally approximate linear operators in a distributed manner. In this paper, we focus on the frequency response of NVGFs and on their capacity to create frequency content.

Leveraging the notion of GFT, [10] shows that a GNN is Lipschitz continuous to small changes in the underlying graph structure. Likewise, frequency analysis has been used to understand graph scattering transforms, where the filters used in the GNN are chosen to be wavelets (and not learned) [13, 14]. In this paper, we focus on the role of frequency creation that is put forth in [15, 10], and study NVGFs as linear mechanisms for achieving this.

Let $\mathcal{G}=(\mathcal{V},\mathcal{E})$ be an undirected, possibly weighted, graph with a set of $N$ nodes $\mathcal{V}=\{v_{1},\ldots,v_{N}\}$ and a set of edges $\mathcal{E}\subseteq\mathcal{V}\times\mathcal{V}$. Define a graph signal as the function $\mathsf{x}:\mathcal{V}\to\mathbb{R}$ that associates a scalar value to each node. For a fixed ordering of the nodes, the graph signal can be conveniently described by means of a vector $\mathbf{x}\in\mathbb{R}^{N}$ such that the $i^{\text{th}}$ entry $[\mathbf{x}]_{i}$ is the value $x_{i}$ of the signal on node $v_{i}$, i.e., $[\mathbf{x}]_{i}=x_{i}=\mathsf{x}(v_{i})$.

Describing a graph signal as the vector $\mathbf{x}\in\mathbb{R}^{N}$ is mathematically convenient but carries no information about the underlying graph topology that supports it. This information can be recovered by defining a graph matrix description (GMD) $\mathbf{S}\in\mathbb{R}^{N\times N}$ which is a matrix that respects the sparsity pattern of the graph, i.e., $[\mathbf{S}]_{ij}=0$ for all distinct indices $i$ and $j$ such that $(v_{j},v_{i})\notin\mathcal{E}$. Examples of GMDs widely used include the adjacency matrix, the Laplacian matrix, and their corresponding normalizations [27, 28, 29].

The GMD $\mathbf{S}$ can thus be leveraged to process the graph signal $\mathbf{x}$ in such a way that the underlying graph structure is exploited. The most elementary example is the linear map $\mathsf{S}:\mathbb{R}^{N}\to\mathbb{R}^{N}$ between graph signals given by $\mathbf{y}=\mathsf{S}(\mathbf{x})=\mathbf{S}\mathbf{x}$. This linear map is a linear combination of the information located in the one-hop neighborhood of each node:

where $\mathcal{N}_{i}=\{v_{j}:(v_{j},v_{i})\in\mathcal{E}\}$ is the set of nodes that share an edge with node $v_{i}$. The last equality in the above equation is due to the sparsity pattern of the GMD $\mathbf{S}$.

More generally, a graph filter $\mathsf{H}(\cdot;\mathbf{S}):\mathbb{R}^{N}\to\mathbb{R}^{N}$ is a mapping between graph signals that leverages the structure encoded in $\mathbf{S}$ [19]. In particular, linear shift-invariant graph filters (LSIGFs) are those that can be built as a polynomial in $\mathbf{S}$:

where $\boldsymbol{H}^{\text{lsi}}(\mathbf{S})=\sum_{k=0}^{K}h_{k}\mathbf{S}^{k}$ with $h_{k}\in\mathbb{R}$. Note that $\mathsf{H}^{\text{lsi}}(\mathbf{x};\mathbf{S})$ is written in the form of $\boldsymbol{H}^{\text{lsi}}(\mathbf{S})\mathbf{x}$ to emphasize that the function is linear in the input $\mathbf{x}$, i.e., $\mathbf{x}$ is multiplied by a matrix $\boldsymbol{H}^{\text{lsi}}(\mathbf{S})$ that is parametrized by $\mathbf{S}$. LSIGFs inherit their name from the fact that they satisfy the property that $\boldsymbol{H}^{\text{lsi}}(\mathbf{S})\mathbf{S}\mathbf{x}=\mathbf{S}\boldsymbol{H}^{\text{lsi}}(\mathbf{S})\mathbf{x}$. The set of polynomial coefficients $\{h_{k}\}_{k=0}^{K}$ are called filter taps, and can be collected in a vector $\mathbf{h}\in\mathbb{R}^{K+1}$ defined as $[\mathbf{h}]_{k+1}=h_{k}$ for all $k\in\{0,\ldots,K\}$. Note that the term $\mathbf{S}^{k}\mathbf{x}$ is a convenient mathematical formulation, but in practice $\mathbf{S}^{k}\mathbf{x}$ is computed by exchanging information $k$ times with one-hop neighbors, i.e., there are no matrix powers involved. In general, GSP regards $\mathbf{S}$ as given by the structure of the problem, and regards $\mathbf{x}$ as the actionable data [27, 28, 29].

This paper focuses on NVGFs [19], which are linear filters that assign a different filter tap to each node, for each application of $\mathbf{S}$. This can be compactly written as follows:

where $\mathbf{h}^{(k)}\in\mathbb{R}^{N}$ and $\mathsf{diag}(\cdot)$ is the diagonal operator that takes a vector and creates a diagonal matrix with the elements of the vector in the diagonal. Since the NVGF is linear in the input, it holds that $\mathsf{H}^{\text{nv}}(\mathbf{x};\mathbf{S})=\boldsymbol{H}^{\text{nv}}(\mathbf{S})\mathbf{x}$, where $\boldsymbol{H}^{\text{nv}}(\mathbf{S})=\sum_{k=0}^{K}\mathsf{diag}(\mathbf{h}^{(k)})\mathbf{S}^{k}$. The $i^{\text{th}}$ entry of the vector, $[\mathbf{h}^{(k)}]_{i}=h_{ik}$, is the filter tap that node $v_{i}$ uses to weigh the information incoming after $k$ exchanges with its neighbors. The set of all filter taps can be conveniently collected in a matrix $\mathbf{H}\in\mathbb{R}^{N\times(K+1)}$ where the $(k+1)^{\text{th}}$ column is $\mathbf{h}^{(k)}\in\mathbb{R}^{N}$ and the $i^{\text{th}}$ row, denoted by $\mathbf{h}_{i}\in\mathbb{R}^{K+1}$, contains the $K+1$ filter taps used by node $v_{i}$, i.e., $[\mathbf{h}_{i}]_{k+1}=h_{ik}$.

The LSIGF in (2) and the NVGF in (3) are both linear and local processing operators. They depend linearly on the input graph signal $\mathbf{x}$ as indicated by the matrix multiplication notation in (2) and in (3). They are local in the sense that, to compute the output, each node requires information relayed directly by their immediate neighbors. The LSIGF is characterized by the collection of $K+1$ filter taps. The NVGF, on the other hand, is characterized by $N(K+1)$ filter taps. It is noted that while the NVGF requires additional memory to store more parameters, this can be distributed throughout the graph. It is also observed that both the LSIGF and the NVGF have the same computational complexity.

The GMD $\mathbf{S}\in\mathbb{R}^{N\times N}$ can be used to define a spectral representation of the graph signal $\mathbf{x}\in\mathbb{R}^{N}$ [17]. Since the graph is undirected, assume that $\mathbf{S}$ is symmetric so that it can be diagonalized by an orthonormal basis of eigenvectors $\{\mathbf{v}_{i}\}_{i=1}^{N}$, where $\mathbf{v}_{i}\in\mathbb{R}^{N}$ and $\mathbf{S}\mathbf{v}_{i}=\lambda_{i}\mathbf{v}_{i}$, with $\lambda_{i}\in\mathbb{R}$ being the corresponding eigenvalue. Then, it holds that $\mathbf{S}=\mathbf{V}\mathsf{diag}(\boldsymbol{\lambda})\mathbf{V}^{\mathsf{T}}$, where the $i^{\text{th}}$ column of $\mathbf{V}$ is $\mathbf{v}_{i}$ and where $\boldsymbol{\lambda}\in\mathbb{R}^{N}$ is given by $[\boldsymbol{\lambda}]_{i}=\lambda_{i}$ for $i=1,\ldots,N$. We assume throughout this paper that the eigenvalues are distinct, which is typically the case for random connected graphs.

The spectral representation of a graph signal $\mathbf{x}$ with respect to its underlying graph support described by $\mathbf{S}$ is given by

where $\mathbf{\tilde{x}}\in\mathbb{R}^{N}$, see [17]. The spectral representation $\mathbf{\tilde{x}}$ of the graph signal $\mathbf{x}$ contains the coordinates of representing $\mathbf{x}$ on the eigenbasis $\{\mathbf{v}_{i}\}_{i=1}^{N}$ of the support matrix $\mathbf{S}$. The resulting vector $\mathbf{\tilde{x}}$ is known as the frequency response of the signal $\mathbf{x}$. The $i^{\text{th}}$ entry $[\mathbf{\tilde{x}}]_{i}=\mathbf{v}_{i}^{\mathsf{T}}\mathbf{x}=\tilde{x}_{i}\in\mathbb{R}$ of the frequency response $\mathbf{\tilde{x}}$ measures how much the $i^{\text{th}}$ eigenvector $\mathbf{v}_{i}$ contributes to the signal $\mathbf{x}$. The operation in (4) is called the GFT, and thus the frequency response $\mathbf{\tilde{x}}$ is often referred to as the GFT of the signal $\mathbf{x}$.

The GFT offers an alternative representation of the graph signal $\mathbf{x}$ that takes into account the graph structure in $\mathbf{S}$. The effect of a filter can be characterized in the spectral domain by computing the GFT of the output. For instance, when considering a LSIGF, the spectral representation of the output $\mathbf{y}=\boldsymbol{H}^{\text{lsi}}(\mathbf{S})\mathbf{x}$ is

where the eigendecomposition of $\mathbf{S}$, the GFT of $\mathbf{x}$, and the fact that $\mathbf{S}^{k}=\mathbf{V}\mathsf{diag}(\boldsymbol{\lambda}^{k})\mathbf{V}^{\mathsf{T}}$ were all used. Note that $\boldsymbol{\lambda}^{k}$ is a shorthand notation that means $[\boldsymbol{\lambda}^{k}]_{i}=\lambda_{i}^{k}$. The vector $\mathbf{\tilde{h}}\in\mathbb{R}^{N}$ is known as the frequency response of the filter and its $i^{\text{th}}$ entry is given by

where $\mathsf{\tilde{h}}:\mathbb{R}\to\mathbb{R}$ is a polynomial defined by the set of filter taps $\{h_{k}\}_{k=0}^{K}$. The function $\mathsf{\tilde{h}}(\cdot)$ depends only on the filter coefficients and not on the specific graph on which it is applied, and thus is valid for all graphs. The effect of filtering on a specific graph comes from instantiating $\mathsf{\tilde{h}}(\cdot)$ on the specific eigenvalues of that graph. The function $\mathsf{\tilde{h}}(\cdot)$ is denoted as the frequency response as well, and it will be clear from context whether we refer to the function $\mathsf{\tilde{h}}(\cdot)$ or to the vector $\mathbf{\tilde{h}}$ given in (6) that stems from evaluating $\mathsf{\tilde{h}}(\cdot)$ at each of the $N$ eigenvalues $\lambda_{i}$. Note that since $\mathsf{\tilde{h}}(\cdot)$ is an analytic function, it can be applied to the square matrix $\mathbf{S}$ so that $\mathsf{\tilde{h}}(\mathbf{S})=\boldsymbol{H}^{\text{lsi}}(\mathbf{S})$.

In the case of the LSIGF, it is observed from (5) that the $i^{\text{th}}$ entry of the frequency response of the output $[\mathbf{\tilde{y}}]_{i}=\tilde{y}_{i}$ is given by the elementwise multiplication

This implies that the frequency response of the output $\tilde{y}_{i}$ is the elementwise multiplication of the frequency response of the filter $\mathsf{\tilde{h}}(\lambda_{i})$ and the frequency response of the input $\tilde{x}_{i}$. This makes (7) the analogue of the convolution theorem for graph signals. Therefore, oftentimes the LSIGF in (2) is called graph convolution. It is observed from (7) that LSIGFs are capable of learning any type of frequency response (low-pass, high-pass, etc.), but that they are not able to create frequencies, i.e., if $\tilde{x}_{i}=0$, then $\tilde{y}_{i}=0$.

Unlike discrete-time signals, the frequency response of the LSIGF is not computed in the same manner as the frequency response of graph signals. More specifically, the frequency response of the filter $\mathbf{\tilde{h}}$ can be directly computed from the filter taps $\mathbf{h}$ by means of a Vandermonde matrix $\mathbf{\Lambda}\in\mathbb{R}^{N\times(K+1)}$ given by $[\mathbf{\Lambda}]_{ik}=\lambda_{i}^{k-1}$ as follows:

This implies that when processing graph signals, the graph filter cannot be uniquely represented by a graph signal [27, Th. 5], and thus the concept of impulse response is no longer valid. Furthermore, the graph convolution is not symmetric, i.e., the filter and the graph signal do not commute. Finally, it is interesting to note that, when $\mathbf{S}$ is the adjacency matrix of a directed cycle and $K=N-1$, then $\mathbf{\Lambda}=\mathbf{V}^{\mathsf{H}}$ and the GFT of the signal and of the filter taps (in time, known as the impulse response) becomes equivalent, as expected.

When considering NVGFs, as in (3), the convolution theorem (7) no longer holds. Instead, the frequency response of the output is given in the following proposition.

It is immediately observed that NVGFs are capable of generating new frequency content, even though they are linear.

As an example, consider the case where the input is given by a single frequency signal, i.e., $\mathbf{x}=\mathbf{v}_{t}$ for some $t\in\{1,\ldots,N\}$ so that $\mathbf{\tilde{x}}=\mathbf{e}_{t}$ where $[\mathbf{e}_{t}]_{i}=1$ if $i=t$ and $[\mathbf{e}_{t}]_{i}=0$ otherwise. This is a signal that has a single frequency component. Yet, the $i^{\text{th}}$ entry of the frequency response of the NVGF output is $\tilde{y}_{i}=\sum_{j=1}^{N}\mathsf{\tilde{h}}_{j}(\lambda_{t})v_{ji}v_{jt}$, where $v_{ij}=[\mathbf{V}]_{ij}$ and $\mathsf{\tilde{h}}_{j}$ is the frequency response obtained from using the $K+1$ coefficients at node $v_{j}$. Unless $\mathsf{\tilde{h}}_{j}(\lambda_{t})$ is a constant for all $j$, i.e., $\mathsf{\tilde{h}}_{j}(\lambda_{t})=\tilde{h}_{t}$, the $i^{\text{th}}$ entry of the GFT of the output is nonzero even when the $i^{\text{th}}$ entry of the GFT of the input is zero. This also shows that by carefully designing the filter taps for each node, the frequencies that are allowed to be created can be chosen, analyzed, and understood.

To finalize the frequency analysis of NVGFs, we establish their Lipschitz continuity to changes in the underlying graph support, as decribed by the matrix $\mathbf{S}$. In what follows, we denote the spectral norm of a matrix $\mathbf{A}$ by $\|\mathbf{A}\|_{2}$.

Theorem 3 establishes the Lipschitz continuity of the NVGF filter with respect to the support matrix $\mathbf{S}$ (Lipschitz continuity with respect to the input $\mathbf{x}$ is immediately given for bounded filter taps) as long as the graphs $\mathbf{S}$ and $\mathbf{\hat{S}}$ are similar, i.e., $\varepsilon\ll 1$. The bound is proportional to this difference, $\varepsilon$, and to the shape of the frequency responses at each node through the Lipschitz constant $C$. It also depends on the number of nodes $N$, but it is fixed for given graphs with the same number of nodes. In short, Theorem 3 gives mild guarantees on the expected performance of the NVGF across a wide range of graphs $\mathbf{\hat{S}}$ that are close to the graph $\mathbf{S}$.

One of the main roles of activation functions in neural networks is to create low-frequency content that can be processed in a stable manner [15]. However, the way the nonlinearities create this frequency content is unknown and cannot be shaped. One alternative for tailoring the frequency creation process to the specific problem under study is to learn the NVGF filter taps (Sec. 5). However, doing so, implies that the number of learnable parameters depend on $N$ which may lead to overfitting. In what follows we propose one method of designing, instead of learning, the NVGFs.

Problem statement. Assume that each data point $\mathbf{x}$ is a random graph signal with mean $\mathbb{E}[\mathbf{x}]=\boldsymbol{\mu}_{x}$, correlation matrix $\mathbf{R}_{x}=\mathbb{E}[\mathbf{x}\mathbf{x}^{\mathsf{T}}]$, and covariance matrix $\mathbf{C}_{x}=\mathbb{E}[(\mathbf{x}-\boldsymbol{\mu}_{x})(\mathbf{x}-\boldsymbol{\mu}_{x})^{\mathsf{T}}]$. The objective is to estimate a pointwise nonlinear function $\mathsf{\rho}:\mathbb{R}\to\mathbb{R}$ such that $[\mathsf{\rho}(\mathbf{x})]_{i}=\mathsf{\rho}([\mathbf{x}]_{i})$, using a NVGF-based estimator as $\mathbf{\hat{y}}=\boldsymbol{H}^{\text{nv}}(\mathbf{S})\mathbf{x}+\mathbf{c}$ for $\boldsymbol{H}^{\text{nv}}(\mathbf{S})$ as in (3) and $\mathbf{c}\in\mathbb{R}^{N}$. Given the random variable $\mathbf{y}=\mathsf{\rho}(\mathbf{x})$, the aim is to find the filter taps $\mathbf{H}\in\mathbb{R}^{N\times(K+1)}$ that minimize the mean squared error (MSE)

Our particular focus is set on obtaining unbiased estimators.

From Lemma 4, the unbiased estimator is now

Therefore, the objective becomes finding the filter tap matrix $\mathbf{H}\in\mathbb{R}^{N\times(K+1)}$ for some fixed value of $K$ that satisfies

Note that, due to the orthogonal nature of the GFT, minimizing (13) is equivalent to minimizing the difference of the corresponding frequency responses.

The optimal filter taps for each node, i.e., the rows $\mathbf{h}_{i}^{\star}\in\mathbb{R}^{K+1}$ of $\mathbf{H}^{\star}$ that solve (13), can be obtained by solving a linear system of equations as follows.

From Proposition 5, it is immediate that only knowledge of the first and second moments of $\mathbf{x}$, and of the correlation between the input $\mathbf{x}$ and the output $\mathbf{y}=\mathsf{\rho}(\mathbf{x})$, is required to solve for the optimal NVGF. These moments can be estimated from training data. Also observed is that the optimal filter taps for each node can be computed separately at each node.

We now consider the specific case of random graph signals with zero-mean, independent, identically distributed (i.i.d.) entries, to illustrate the form of the optimal NVGF.

Corollary 6 shows that if the elements of the graph signal are zero-mean and i.i.d., then the NVGF boils down to a LSI graph filter. Hence, this optimal unbiased NVGF is not capable of generating frequencies. This is sensible since the elements of the graph signal $\mathbf{x}$ bare no relationship to the underlying graph support, and thus the NVGF does not leverage that support to create the appropriate frequencies. Furthermore, if the distribution of each element of $\mathbf{x}$ is symmetric around zero, then an optimal approximation of the $\mathsf{ReLU}$ nonlinearity amounts to a LSI graph filter that outputs half of the input, as one would expect.

It is observed from Corollary 7 that the effect of the contribution of each frequency component to node $v_{i}$ gets modulated by the power spectral density $\mathbf{q}$. Also, the filter in this case is different for each node, and thus optimal unbiased NVGFs for processing stationary graph processes actually create frequencies.

A graph convolutional neural network (GCNN) $\mathsf{\Phi}$ has a layered architecture [23, 6], where each layer applies a LSIGF as in (2), followed by a pointwise nonlinear activation function $\rho:\mathbb{R}\to\mathbb{R}$ applied to each node $[\rho(\mathbf{x})]_{i}=\rho([\mathbf{x}]_{i})$ and

The LSIGF $\mathsf{H}_{\ell}^{\text{lsi}}(\cdot;\mathbf{S},\mathbf{h}_{\ell})$ is characterized by the specific filter taps $\mathbf{h}_{\ell}\in\mathbb{R}^{K_{\ell}+1}$. Note that the output of the GCNN is collected as the output of the last layer $\mathbf{x}_{L}=\mathsf{\Phi}(\mathbf{x};\mathbf{S},\mathcal{H})$. This notation emphasizes that the input is $\mathbf{x}$, while the matrix $\mathbf{S}$ is given by the problem, and the filter taps $\mathcal{H}=\{\mathbf{h}_{\ell}\}_{\ell=1}^{L}$ are learned from data.

Nonlinear activation functions are used in GCNNs to enable them to learn nonlinear relationships between input and output. Additionally, theoretical results have found that they play a key role in creating frequency content that can be better processed by subsequent graph convolutional filters. As previously discussed, NVGFs are also capable of creating new frequency content, albeit in a linear manner. Therefore, by replacing the nonlinear activation functions by NVGFs, it is possible to decouple the contribution made by frequency creation from that made by the architecture’s nonlinearity.

The first architecture proposed here is to use a designed NVGF in lieu of the activation function. That is, instead of using the nonlinear activation function $\rho$, an optimal NVGF filter designed as in Proposition 5 is used. This requires estimating the first and second moments of the NVGF input data. The architecture, herein termed “Design NVGF”, is given by

Note that, in this case, the filter taps $\mathbf{H}_{\ell}^{\star}\in\mathbb{R}^{N\times(K_{\ell}+1)}$ of the NVGF are obtained by Proposition 5, while the filter taps $\mathcal{H}=\{\mathbf{h}_{\ell}\}_{\ell=1}^{L}$ of the LSIGF are learned from data.

Alternatively, the filter taps of the NVGF replacing the nonlinear activation function can also be learned from data, together with the filter taps of the LSIGF:

where the filter taps to be learned are $\mathcal{H}=\{(\mathbf{h}_{\ell},\mathbf{H}_{\ell})\}_{\ell=1}^{L}$. This approach avoids the need to estimate first and second moments. Additionally, it allows the NVGF to learn how to create frequency content tailored to the application at hand, instead of just approximating the chosen nonlinear activation function. We note that while the increased number of parameters may lead to overfitting, this can be tackled by dropout. This architecture is termed “Learn NVGF”.

The objectives of the experiments are twofold. First, they aim to show how the architecture obtained by replacing the nonlinear activation function with a NVGF performs when compared to other popular GNNs. The second objective is to isolate the impact that frequency creation has on the overall performance. Due to space constraints, we focus on the problem of authorship attribution [5]. Other applications can be found in the supplementary material, where the observations are summarized at the end of this section for reference.

Problem statement. In the problem of authorship attribution, the goal is to determine whether a given text has been written by a certain author, relying on other texts that the author is known to have written. To this end, word adjacency networks (WANs) are leveraged. WANs are graphs that are built by considering function words (i.e., words without semantic meaning) as nodes, and considering their co-occurrences as edges [5]. As it happens, the way each author uses function words gives rise to a stylistic signature that can be used to attribute authorship. In what follows, we address this problem by leveraging previously constructed WANs as graphs, and the frequency count (histogram) of function words as the corresponding graph signals.

Dataset. For illustrative purposes, in what follows, works by Jane Austen are considered. Attribution of other $\text{19}^{\text{th}}$ century authors can be found in the supplementary material. A WAN consisting of $189$ nodes (function words) and $9812$ edges is built from texts belonging to a given corpus considered to be the training set. These texts are partitioned into segments of approximately $1000$ words each, and the frequency count of those $189$ function words in each of the texts is obtained. These represent the graph signals $\mathbf{x}$ that are considered to be part of the training set. Each of these is assigned a label $1$ to indicate that they have been authored by Jane Austen. An equal number of segments from other contemporary authors are randomly selected, and then their frequency count is computed and added to the training set with the label $0$ to indicate that they have not been written by Jane Austen. The total number of labeled samples in the training set is $1464$, of which $118$ are left aside for validation. The test set is built analogously by considering other text segments that were not part of the training set (and thus, not used to build the WAN either), totaling $78$ graph signals (half corresponding to texts authored by Jane Austen, and half corresponding to texts by other contemporary authors).

Architectures and training. For the first experiment, we compare the Learn NVGF architecture (18) with arguably three of the most popular non-GSP GNNs, namely, GCN [24], SGC [25], and GAT [26]. Note that the Learn NVGF is an entirely linear architecture, but one capable of creating frequencies due to the nature of the NVGF. The filter taps of both the LSIGF and the NVGF are learned from data. The other three architectures are nonlinear since they include a ReLU activation function after the first filtering layer. All architectures include a learnable linear readout layer. Dropout with a probability $0.5$ is included after the first layer. All architectures are trained for $25$ epochs with a batch size of $20$, using the ADAM algorithm [32] with the forgetting factors $0.9$ and $0.999$, respectively, as well as a learning rate $\eta$. The value of the learning rate $\eta$, the number of hidden units $F$, and the number of filter taps $K$ are chosen by exhaustive search over triplets $(\eta,F,K)$ in the set $\{0.001,0.005,0.01\}\times\{16,32,64\}\times\{2,3,4\}$.

Experiment 1: Performance comparison. The objective of this first experiment is to illustrate that the performance of the Learn NVGF is comparable to the performance of popular (non-GSP) GNNs. The best results for each architecture are shown in Figure 1a, where the classification error was averaged over $10$ random splits of texts that are assigned to the training and test sets. One third of the standard deviation is also shown. It is observed that the Learn NVGF architecture has a comparable performance. It is emphasized that the objective of this paper is not to achieve state-of-the-art performance, but to offer insight on the role of nonlinear activation functions in frequency creation and how this translates to performance, as discussed next. Among the three popular architectures, GAT ($\eta=0.01$, $F=64$) exhibits the best performance, which is $102\%$ better than SGC ($\eta=0.005$, $F=64$, $K=2$) and $30\%$ better than GCN ($\eta=0.01$, $F=64$). This is expected since the graph filters involved in GAT are non-convolutional, and thus have more expressivity. Yet, when compared to the linear Learn NVGF ($\eta=0.001$, $F=32$, $K=3$), it is observed that the latter exhibits $20\%$ better performance than the GAT.

Experiment 2: The role of frequency creation. For the second experiment, we compare the Learn NVGF of the previous experiment with (i) a simple LSIGF, (ii) a Design NVGF as in (17), and (iii) a GCNN as in (16). The same values of $(\eta=0.001,F=32,K=3)$ are used for all architectures as a means of fixing all other variabiliity except for the nonlinearity/frequency creation. The results are shown in Figure 1b. Note that the GCNN in (16) can be interpreted as a stand-in for ChebNets [23], arguably the most popular GSP-based GNN architecture.

Discussion. First, note that the LSIGF, Design NVGF, and Learn NVGF architectures are linear, whereas the GCNN is not. The LSIGF, however, is not capable of creating frequencies, while the other three are, albeit through different mechanisms. Second, Figure 1b shows the percentage difference in performance with respect to the base architecture (the GCNN). Essentially, the difference in performance between the LSIGF and the Learn NVGF can be pinned down to the frequency creation, because both architectures are linear, while the difference between the Learn NVGF and the GCNN can be tied to the nonlinear nature of the GCNN. It is thus observed that the LSIGF performs considerably worse than the Learn NVGF and the GCNN, which perform the same. It is observed that the Design NVGF performs halfway between the GCNN and the LSIGF. The Design NVGF depends on the ability to accurately estimate the first and second moments from the data, and this has an impact on its performance. In any case, it is noted that this experiment suggests that the main driver of improved performance is the frequency creation and not necessarily the nonlinear nature of the GCNN.

From a qualitative standpoint, the average frequency response of the signals in the test set is shown in Figure 1c. Since the high-eigenvalue content is significant, it is expected that the ability to better process this content will impact the overall performance. This explains the relatively poor performance of the LSIGF. In Figure 2, we show the frequency response of the output for each of the three architectures (LSIGF, Learn NVGF, and GCNN) when the input has a single high frequency, i.e., $\mathbf{x}=\mathbf{v}_{N}$ so that $\mathbf{\tilde{x}}=\mathbf{e}_{N}$. Figure 2a shows that the output frequency response of the LSIGF exhibits a single frequency, the same as the input. Figure 2b shows that the output frequency response of the GCNN has content in all frequencies, but most notably a low-frequency peak appears. Figure 2c shows that the output frequency response of the Learn NVGF contains all frequencies, in a much more spread manner than the GCNN.

General observations. In the supplementary material, a similar analysis is carried out for $21$ other authors. Additionally, it is noted that Jane Austen is representative of the largest group (consisting of $11$ authors) where the Learn NVGF and the GCNN have similar performance and are better than the LSIGF. For $7$ other authors, the Learn NVGF actually performs better than the GCNN. Finally, for the remaining $3$ authors, there is no significant difference between the performance of the LSIGF, the GCNN, and the Learn NVGF, which implies that the high-frequency eigenvalue content is less significant for these authors. Additionally, the problem of movie recommendation is considered [33]. First, it is observed that the Learn NVGF exhibits better performance than the methods in [3, 4] and the nearest neighbor algorithm. Second, it is noted that the input signals do not have significant high-eigenvalue content and thus the LSIGF, the GCNN, and the Learn NVGF exhibit similar performance. For a detailed analysis, please see the supplementary material. ¹¹1It is noted that the popular benchmark of semi-supervised learning over citation networks does not fit the empirical risk minimization framework nor the graph signal processing framework, and thus, it does not admit a frequency analysis, precluding their use in this work.

Problems beyond processing graph signals. Two popular tasks that utilize graph-based data are semi-supervised learning and graph classification. The former involves a framework where each sample represents a node in the graph, i.e., $\mathbf{x}\in\mathbb{R}^{F}$ for $F$ features instead of $\mathbf{x}\in\mathbb{R}^{N}$ for $N$ nodes. This implies that the notion of graph frequency response used in (4) does not translate to the semi-supervised setting. Therefore, extending these results to this problem requires careful determination of the notion of frequency. For the graph classification problem, each sample in the dataset represents a signal together with a graph, and the graph associated to each sample is usually different. In this case, while the notion of frequency is properly defined for each sample, an overarching frequency response for all the samples is not. Possible extensions of this framework to the graph classification problem would entail redefining a common notion of frequency response for the dataset, for example, using graphons.

The objective of this work was to study the role of frequency creation in GSP problems. To do so, nonlinear activation functions (which theoretical findings suggest give rise to frequency creation) are replaced by NVGFs, which are also capable of creating frequencies, but in a linear manner. In this way, frequency creation was decoupled from the nonlinear nature of activation functions. Numerical experiments show that the main driver of improved performance is frequency creation and not necessarily the nonlinear nature of GCNNs. As future work, we are interested in extending this frequency analysis to non-GSP related problems such as semi-supervised node classification or graph classification problems, which require a careful definition of a notion of frequency. We discussed the caveats of extending this framework to include semi-supervised learning and graph classification problems, relating to the need of defining an appropriate notion of graph frequency. This opens up an interesting area of future research. We also note that it would be possible to use shift-variant filters to leverage this framework when using CNNs.