A Gromov–Wasserstein Geometric View of Spectrum-Preserving
Graph Coarsening

We denote an undirected graph as $G$, whose node set is $\mathcal{V}\vcentcolon=\left\{v_{i}\right\}_{i=1}^{N}$ with size $N=|\mathcal{V}|$ and whose symmetric weighted adjacency is $\bm{A}\vcentcolon=[a_{ij}]$. The $N$-by-$N$ (combinatorial) Laplacian matrix is defined as $\bm{L}\vcentcolon=\bm{D}-\bm{A}$, where $\bm{D}\vcentcolon=\text{diag}\left(\bm{A1}_{N}\right)$ is the degree matrix. The normalized Lapalcian is $\bm{\mathcal{L}}\vcentcolon=\bm{D}^{-\frac{1}{2}}\bm{L}\bm{D}^{-\frac{1}{2}}$. Without loss of generality, we assume that $G$ is connected, in which case the smallest eigenvalue of $\bm{L}$ and $\bm{\mathcal{L}}$ is zero and is simple.

Given a graph $G$, graph coarsening amounts to finding a smaller graph $G^{(c)}$ with $n\leq|\mathcal{V}|$ nodes to approximate $G$. One common coarsening approach obtains the coarsened graph from a parititioning $\mathcal{P}=\left\{\mathcal{P}_{1},\mathcal{P}_{2},\dots,\mathcal{P}_{n}\right\}$ of the node set $\mathcal{V}$ (Loukas & Vandergheynst, 2018). In this approach, each subset $\mathcal{P}_{j}$ of nodes are collapsed to a supernode in the coarsened graph and the edge weight between two supernodes is the sum of the edge weights crossing the two corresponding partitions. Additionally, the sum of the edge weights in a partition becomes the weight of the supernode. In the matrix notation, we use $\bm{C}_{p}\in\left\{0,1\right\}^{n\times N}$ to denote the membership matrix induced by the partitioning $\mathcal{P}$, with the $(k,i)$ entry being $\bm{C}_{p}(k,i)=1_{\{v_{i}\in\mathcal{P}_{k}\}}$, where $1_{\{\cdot\}}$ is the indicator function. For notational convenience, we define the adjacency matrix of the coarsened graph to be $\bm{A}^{(c)}=\bm{C}_{p}\bm{A}\bm{C}_{p}^{T}$. Note that $\bm{A}^{(c)}$ includes not only edge weights but also node weights.

If we similarly define $\bm{D}^{(c)}\vcentcolon=\text{diag}\left(\bm{A^{(c)}1}_{n}\right)$ to be the degree matrix of the coarsened graph $G^{(c)}$, then it can be verified that the matrix $\bm{L}^{(c)}=\bm{C}_{p}\bm{L}\bm{C}_{p}^{\intercal}=\bm{D}^{(c)}-\bm{A}^{(c)}$ is a (combinatorial) Laplacian (because its smallest eigenvalue is zero and is simple). Additionally, $\bm{\mathcal{L}}^{(c)}=\left(\bm{D}^{(c)}\right)^{-\frac{1}{2}}\bm{L}^{(c)}\left(\bm{D}^{(c)}\right)^{-\frac{1}{2}}$ is the normalized Laplacian.

In the literature, the objective of coarsening is often to minimize some difference between the original and the coarsened graph. For example, in spectral graph coarsening, Loukas (2019) proposes that the $\bm{L}$-norm of any $N$-dimensional vector $\bm{x}$ be close to the $\bm{L}^{(c)}$-norm of the $n$-dimensional vector $(\bm{C}_{p}^{\intercal})^{+}\bm{x}$; while Jin et al. (2020) propose to minimize the difference between the ordered eigenvalues of the Laplacian of $G$ and those of the lifted Laplacian of $G^{(c)}$ (such that the number of eigenvalues matches). Such objectives, while being natural and interesting in their own right, are not the only choice. Questions may be raised; for example, it is unclear why the objective is to preserve the Laplacian spectrum but not the degree distribution or the clustering coefficients. Moreover, it is unclear how preserving the spectrum benefits the downstream use. In this paper, we consider a different objective—preserving the graph distance—which may help, for example, maintain the decision boundary in graph classification problems. To this end, we review the Gromov–Wasserstein (GW) distance.

The GW distance was originally proposed by Mémoli (2011) to measure the distance between two metric measure spaces $M_{\mathcal{X}}$ and $M_{\mathcal{Y}}$.²²2A metric measure space $M_{\mathcal{X}}$ is the triple $(\mathcal{X},d_{X},\mu_{X})$, where $(\mathcal{X},d_{X})$ is a metric space with metric $d_{X}$ and $\mu_{X}$ is a Borel probability measure on $\mathcal{X}$. However, using metrics to characterize the difference between elements in sets $\mathcal{X}$ and $\mathcal{Y}$ can be too restrictive. Peyré et al. (2016) relaxed the metric notion by proposing the GW discrepancy, which uses dissimilarity, instead of metric, to characterize differences. Chowdhury & Mémoli (2019) then extended the concept of GW distance/discrepancy from metric measure spaces to measure networks, which can be considered a generalization of graphs.

In particular, a graph $G$ (augmented with additional information) can be taken as a discrete measure network. We let $\mathcal{X}$ be the set of graph nodes $\left\{v_{i}\right\}_{i=1}^{N}$ and associate with it a probability mass $\mu_{X}=\bm{m}=\left[m_{1},\dots,m_{N}\right]^{\intercal}\in\mathbb{R}_{+}^{N}$, $\sum_{i=1}^{N}m_{i}=1$. Additionally, we associate $G$ with the node similarity matrix $\bm{S}=[s_{ij}]\in\mathbb{R}^{N\times N}$, whose entries are induced from the measurable map $\omega_{X}$: $s_{ij}=\omega_{X}\left(v_{i},v_{j}\right)$. Note that the mass $\bm{m}$ and the similarity $\bm{S}$ do not necessarily need to be related to the node weights and edge weights, although later we will justify and advocate the use of some variant of the graph Lapalcian as $\bm{S}$.

For a source graph $G_{s}$ with $N_{s}$ nodes and mass $\bm{m}_{s}$ and a target graph $G_{t}$ with $N_{t}$ nodes and mass $\bm{m}_{t}$, we can define a transport matrix $\bm{T}=[t_{ij}]\in\mathbb{R}^{N_{s}\times N_{t}}$, where $t_{ij}$ specifies the probability mass transported from $v_{i}^{s}$ (the $i$-th node of $G_{s}$) to $v_{j}^{t}$ (the $j$-th node of $G_{t}$). We denote the collection of all feasible transport matrices as $\Pi_{s,t}$, which includes all $\bm{T}$ that satisfy $\bm{T1}=\bm{m}_{s}$ and $\bm{T}^{\intercal}\bm{1}=\bm{m}_{t}$. Using the $\ell_{p}$ transportation cost $L(a,b)=(a-b)^{p}$, Chowdhury & Mémoli (2019) define the $\text{GW}_{p}$ distance for graphs as

whre the cross-graph dissimilarity matrix $\bm{M}\in\mathbb{R}^{N_{s}\times N_{t}}$ has entries $\bm{M}_{jj^{\prime}}=\sum_{i,i^{\prime}}\left|s^{s}_{ij}-s^{t}_{i^{\prime}j^{\prime}}\right|^{p}\bm{T}_{ii^{\prime}}$ (which by themselves are dependent on $\bm{T}$).

The computation of the $\text{GW}_{p}$ distance (GW distance for short) can be thought of as finding a proper alignment of nodes in two graphs, such that the aligned nodes $v^{s}_{j}$ and $v^{t}_{j^{\prime}}$ have similar interactions with other nodes in their respective graphs. Intuitively, this is achieved by assigning large transport mass $\bm{T}_{jj^{\prime}}$ to a node pair $(v^{s}_{j},v^{t}_{j^{\prime}})$ with small dissimilarity $\bm{M}_{jj^{\prime}}$, making the GW distance a useful tool for graph matching (Xu et al., 2019b). We note that variants of the GW distance exist; for example, Titouan et al. (2019) proposed a fused GW distance by additioanlly taking into account the dissimilarity of node features. We also note that computing the GW distance is NP-hard, but several approximate methods were developed (Xu et al., 2019a; Zheng et al., 2022). In this work, we use the GW distance as a theoretical tool and may not need to compute it in action.

On closing this section, we remark that Chowdhury & Mémoli (2019, Theorem 18) show that the GW distance is indeed a metric for measure networks, modulo weak isomorphism.³³3The weak isomorphism allows a node to be split into several identical nodes with the mass preserved. See Definition 3 of Chowdhury & Mémoli (2019). Therefore, we can formally establish the metric space of interest.

The membership matrix $\bm{C}_{p}$ is a kind of coarsening matrices: it connects the adjacency matrix $\bm{A}$ with the coarsened version $\bm{A}^{(c)}$ through $\bm{A}^{(c)}=\bm{C}_{p}\bm{A}\bm{C}_{p}^{\intercal}$. There are, however, different variants of coarsening matrices. We consider three here, all having a size $n\times N$.

1. (i)

   Accumulation. This is the matrix $\bm{C}_{p}$. When multiplied to the left of $\bm{A}$, the $i$-th row of the product is the sum of all rows of $\bm{A}$ corresponding to the partition $\mathcal{P}_{i}$.

2. (ii)

   Averaging. A natural alternative to summation is (weighted) averaging. We define the diagonal mass matrix $\bm{W}=\text{diag}\left(m_{1},\cdots,m_{N}\right)$ and for each partition $\mathcal{P}_{i}$, the accumulated mass $c_{i}=\sum_{j\in\mathcal{P}_{i}}m_{j}$ for all $i\in[n]$. Then, the averaging coarsening matrix is

   $$\bar{\bm{C}}_{w}\vcentcolon=\mathrm{diag}(c_{1}^{-1},\cdots,c_{n}^{-1})\bm{C}_{p}\bm{W}.$$

   This matrix takes the effect of averaging because of the division over $c_{i}$. Moreover, when the probability mass $\bm{m}$ is uniform (i.e., all $m_{j}$’s are the same), we have the relation $\bar{\bm{C}}_{w}^{+}=\bm{C}_{p}^{\intercal}$.

3. (iii)

   Projection. Neither $\bm{C}_{p}$ nor $\bar{\bm{C}}_{w}$ is orthogonal. We define the projection coarsening matrix as

   $$\bm{C}_{w}\vcentcolon=\mathrm{diag}(c_{1}^{-1/2},\cdots,c_{n}^{-1/2})\bm{C}_{p}\bm{W}^{\frac{1}{2}},$$

   by noting that $\bm{C}_{w}\bm{C}_{w}^{\intercal}$ is the identity and hence $\bm{C}_{w}$ has orthonoral rows. Therefore, the $N$-by-$N$ matrix $\bm{\Pi}_{w}\vcentcolon=\bm{W}^{\frac{1}{2}}\bm{C}_{p}^{\intercal}\bar{\bm{C}}_{w}\bm{W}^{-\frac{1}{2}}=\bm{C}_{w}^{\intercal}\bm{C}_{w}$ is a projection operator.

In Section 3.2, we have seen that the combinatorial Laplacian $\bm{L}$ takes $\bm{C}_{p}$ as the coarsening matrix, because the relationship $\bm{L}^{(c)}=\bm{C}_{p}\bm{L}\bm{C}_{p}^{\intercal}$ inherits from $\bm{A}^{(c)}=\bm{C}_{p}\bm{A}\bm{C}_{p}^{\intercal}$. On the other hand, it can be proved that, if we take the diagonal mass matrix $\bm{W}$ to be the degree matrix $\bm{D}$, the normalized Laplacian defined in Section 3.2 can be written as $\bm{\mathcal{L}}^{(c)}=\bm{C}_{w}\bm{\mathcal{L}}\bm{C}_{w}^{\intercal}$ (see Section B.1). In other words, the normalized Laplacian uses $\bm{C}_{w}$ as the coarsening matrix. The matrix $\bm{\mathcal{L}}^{(c)}$ is called a doubly-weighted Laplacian (Chung & Langlands, 1996).

For a general similarity matrix $\bm{S}$ (not necessarily a Laplacian), we use the averaging coarsening matrix $\bar{\bm{C}}_{w}$ and define $\bm{S}^{(c)}\vcentcolon=\bar{\bm{C}}_{w}\bm{S}\bar{\bm{C}}_{w}^{\intercal}$. This definition appears to be more natural in the GW setting; see a toy example in Section B.2. It is interesting to note that Vincent-Cuaz et al. (2022) proposed the concept of semi-relaxed GW (srGW) divergence, wherein the first-order optimality condition of the constrained srGW barycenter problem is exactly the equality $\bm{S}^{(c)}=\bar{\bm{C}}_{w}\bm{S}\bar{\bm{C}}_{w}^{\intercal}$. See Section B.3.

In the next subsection, we will consider the matrix $\bm{U}\vcentcolon=\bm{W}^{\frac{1}{2}}\bm{S}\bm{W}^{\frac{1}{2}}$. We define the coarsened version by using the projection coarsening matrix $\bm{C}_{w}$ as in $\bm{U}^{(c)}\vcentcolon=\bm{C}_{w}\bm{U}\bm{C}_{w}^{\intercal}$. We will bound the distance of the original and the coarsened graph by using the eigenvalues of $\bm{U}$ and $\bm{U}^{(c)}$. The reason why $\bm{U}$ and $\bm{S}$ use different coarsening matrices lies in the technical subtlety of $\bm{W}^{\frac{1}{2}}$: $\bm{C}_{w}$ absorbs this factor from $\bar{\bm{C}}_{w}$.

Now we consider the GW distance between two graphs. For theoretical and computational convenience, we take the $\ell_{2}$ transportation cost (i.e., taking $p=2$ in (3.3)). When two graphs $G_{1}$ and $G_{2}$ are concerned, we inherit the subscripts ₁ and ₂ to all related quantities, such as the probability masses $\bm{m}_{1}$ and $\bm{m}_{2}$, avoiding verbatim redundancy when introducing notations. This pair of subscripts should not cause confusion with other subscripts, when interpreted in the proper context. Our analysis can be generalized from the $\text{GW}_{2}$ distance to other distances, as long as the transportation cost satisfies the following decomposable condition.

Clearly, for the squared cost $L(a,b)=(a-b)^{2}$, we may take $f_{1}(a)=a^{2}$, $f_{2}(b)=b^{2}$, $h_{1}(a)=a$, and $h_{2}(b)=2b$.

We start the analysis by first bounding the distance between $G$ and $G^{(c)}$.

We now bound the difference of distances. The following theorem suggests that the only terms dependent on coarsening are $\Delta_{1}$ and $\Delta_{2}$, counterparts of $\Delta$ in Theorem 4.2, for graphs $G_{1}$ and $G_{2}$ respectively.

The spectral difference $\Delta=\sum_{i=1}^{n}\left(\lambda_{i}-\lambda^{(c)}_{i}\right)$ in Theorem 4.2 can be used as the loss function for defining an optimal coarsening. Because $\Delta+\sum_{i=n+1}^{N}\lambda_{i}=\operatorname{Tr}(\bm{U})-\operatorname{Tr}(\bm{C}_{w}\bm{U}\bm{C}_{w}^{\intercal})$ and because $\sum_{i=n+1}^{N}\lambda_{i}$ is independent of coarsening, minimizing $\Delta$ is equivalent to the following problem

by recalling that $\bm{\Pi}_{w}=\bm{C}_{w}^{\intercal}\bm{C}_{w}$ is a projector. The problem (4) is a well-known trace optimization problem, which has rich connections with many spectral graph techniques (Kokiopoulou et al., 2011).

Connection with truncated SVD. At first sight, a small trace difference does not necessarily imply the two matrices are close. However, because $\bm{\Pi}_{w}$ is a projector, the Poincaré separation theorem (see Section D.1) suggests that their eigenvalues can be close. A well-known example of using the trace to find optimal approximations is the truncated SVD, which retains the top singular values (equivalently eigenvalues for PSD matrices). The truncated SVD is a technique to find the optimal rank-$n$ approximation of a general matrix $\bm{U}$ in terms of the spectral norm or the Frobenius norm, solving the problem

where $\mathcal{O}_{n}$ is the class of all rank-$n$ orthogonal matrices. The projection coarsening matrix $\bm{C}_{w}$ belongs to this class.

Connection with Laplacian eigenmaps. The Laplacian eigenmap (Belkin & Niyogi, 2003) is a manifold learning technique that learns an $n$-dimensional embedding for $N$ points connected by a graph. The embedding matrix $\bm{Y}$ solves the trace problem $\min_{\bm{Y}\bm{D}\bm{Y}^{\intercal}=\bm{I}_{n}}\operatorname{Tr}\left(\bm{Y}\bm{L}\bm{Y}^{\intercal}\right)$. If we let $\bar{\bm{Y}}=\bm{Y}\bm{D}^{-\frac{1}{2}}$, the problem is equivalent to

Different from truncated SVD, which uses the top singular vectors (eigenvectors) to form the solution, the Laplacian eigenmap uses the bottom eigenvectors of $\bm{\mathcal{L}}$ to form the solution.

The theory established in Section 4.2 is applicable to any PSD matrix $\bm{S}$, but for practical uses we still have to define it. Based on the foregoing exposition, it is tempting to let $\bm{S}$ be the Laplacian $\bm{L}$ or the normalized Laplacian $\bm{\mathcal{L}}$, because they are PSD and they reveal important information of the graph structure (Tsitsulin et al., 2018). In fact, as a real example, Chowdhury & Needham (2021) used the heat kernel as the similarity matrix to define the GW distance (and in this case the GW framework is related to spectral clustering). However, there are two problems that make such a choice troublesome. First, the (normalized) Laplacian is sparse but its off-diagonal, nonzero entries are negative. Thus, when $\bm{S}$ is interpreted as a similarity matrix, a pair of nodes not connected by an edge becomes more similar than a pair of connected nodes, causing a dilema. Second, under the trace optimization framework, one is lured to intuitively look for solutions toward the bottom eigenvectors of $\bm{S}$, like in Laplacian eigenmaps, an opposite direction to the true solution of (4), which is toward the top eigenvectors instead.

To resolve these problems, we propose to use the signless Laplacian (Cvetković et al., 2007), $\bm{D}+\bm{A}$, or its normalized version, $\bm{I}_{N}+\bm{D}^{-\frac{1}{2}}\bm{A}\bm{D}^{-\frac{1}{2}}$, as the similarity matrix $\bm{S}$. These matrices are PSD and their nonzero entries are all positive. With such a choice, the spectral difference $\Delta$ in (3) has a fundamentally different behavior from the spectral distance used by Jin et al. (2020) when defining the coarsening objective.

For a sanity check, we evaluate each coarsening method on the approximation of the GW distance, to support our motivation of minimizing the upper bound in Theorem 4.2.

We first compare the average squared GW distance, by using the normalized signless Laplacian $\bm{I}_{N}+\bm{D}^{-\frac{1}{2}}\bm{A}\bm{D}^{-\frac{1}{2}}$ as the similarity matrix $\bm{S}$ (see Section 4.4). We vary the coarsened graph size $n=\lceil c*N\rceil$ for $c=0.3,\cdots,0.9$. For each graph in PTC and IMDB, we compute $\operatorname{\mathrm{GW}}_{2}^{2}(G,G^{(c)})$ and report the average in Table 5, Appendix C.1. We obtain similar observations across the two datasets. In particular, KGC and KGC(A) outperform baselines. When $c$ is small, it is harder for $K$-means clustering to find the best clustering plan and hence KGC(A) works better. When $c$ gets larger, KGC can work better, probably because the baseline outputs are poor intiializations.

We then report the average gap between the left- and right-hand sides of the bound (2) in Table 6, Appendix C.1. For all methods, including the baselines, the gap is comparable to the actual squared distance shown in Table 5, showcasing the quality of the bound. Moreover, the gap decreases when $c$ increases, as expected.

We next compute the matrix of GW distances, $\bm{Z}$ (before coarsening) and $\bm{Z}^{(c)}$ (after coarsening), and compute the change $\|\bm{Z}-\bm{Z}^{(c)}\|_{F}$. Following previous works (Chan & Airoldi, 2014; Xu et al., 2020), we set $n=\lfloor N_{\mathrm{max}}/\log(N_{\mathrm{max}})\rfloor$. The similarity matrix for the coarsened graph uses the averaging coarsening matrix as advocated in Section 4.1; that is, $\bm{S}^{(c)}=\bar{\bm{C}}_{w}\left(\bm{I}_{N}+\bm{D}^{-\frac{1}{2}}\bm{A}\bm{D}^{-\frac{1}{2}}\right)\bar{\bm{C}}_{w}^{\intercal}$. Additionally, we use a variant of the similarity matrix, $\bm{S}^{(c)}=\bm{I}_{n}+\left(\bm{D}^{(c)}\right)^{-\frac{1}{2}}\bm{A}^{(c)}\left(\bm{D}^{(c)}\right)^{-\frac{1}{2}}$, resulting from the projection coarsening matrix, for comparison.

Table 2 summarizes the results for PTC. It confirms that using “Averaging” is better than using “Projection” for the coarsening matrix. Additionally, KGC(A) initialized with MGC (the best baseline) induces a significantly small change (though no better than the change caused by KGC), further verifying that minimizing the loss (4) will lead to a small change in the GW distance. The runtime reported in the table confirms the analysis in Section 5, showing that KGC is efficient. Moreover, the runtime of KGC(A) is nearly negligible compared with that of the baseline method used for initializing KGC(A).

We evaluate the capability of the methods in preserving the Laplacian spectrum, by noting that the spectral objectives of the baseline methods differ from that of ours (see remarks after Theorem 4.2). We coarsen each graph to its $10\%,20\%,30\%$, and $40\%$ size and evaluate the metric $\frac{1}{5}\sum_{i=1}^{5}\frac{\lambda_{i}-\lambda_{i}^{(c)}}{\lambda_{i}}$ (the relative error of the first five largest eigenvalues). The experiment is conducted on Tumblr and the calculation of the error metric is repeated ten times.

In Figure 2, we see that KGC has a comparable error to the variational methods VNGC and VEGC, while incurring a lower time cost, especially when the coarsening ratio is small. Additionally, KGC(A) consistently improves the initialization method SGC and attains the lowest error.

We test the performance of the various coarsening methods for graph classification. We follow the setting of Jin et al. (2020) and adopt the Network Laplacian Spectral Descriptor (Tsitsulin et al., 2018, NetLSD) along with a $1$-NN classifier as the classification method. We set $n=0.2N$. For evaluation, we select the best average accuracy of 10-fold cross validations among three different seeds. Additionally, we add two baselines, EIG and FULL, which respectively use the first $n$ eigenvalues and the full spectrum of $\bm{\mathcal{L}}$ in NetLSD. Similarly to before, we initialize KGC(A) with the best baseline.

Table 3 shows that preserving the spectrum does not necessarily leads to the best classification, since EIG and FULL, which use the original spectrum, are sometime outperformed by other methods. On the other hand, our proposed method, KGC or KGC(A), is almost always the best.

We follow the setting of Dwivedi et al. (2020) to perform graph regression on AQSOL and ZINC (see Appendix C.4 for details). For evaluation, we pre-coarsen the whole datasets, reduce the graph size by 70%, and run GCN on the coarsened graphs. Table 4 suggests that KGC(A) initialized with VEGC (the best baseline) always returns the best test MAE. On ZINC, KGC sometimes suffers from the initialization, but its performance is still comparable to a reported MLP baseline (Dwivedi et al., 2020, TestMAE $0.706\pm 0.006$).