Lost in the Shuffle: Testing Power in the Presence of Errorful Network Vertex Labels

Given an undirected graph $n$-vertex graph $G$ (all graphs considered will be undirected graphs with no self-loops), we let $[n]:=\{1,2,...,n\}$ denote the vertices of $G$. The adjacency matrix A $\in\{0,1\}^{n\times n}$ of $G$ is given by: $A_{ij}=\mathds{1}\{\{i,j\}\in E(G)\}$ for all $i,j\in[n]$. We note here that we will refer to a graph and its adjacency matrix interchangeably, as these objects (in the setting we consider herein) encode equivalent information. We denote the $i^{th}$ row of any matrix M with the notation $M_{i}$ (or via $\mathbf{M}_{n}[i,:]$ if the subscript is needed to denote explicit dependence on $n$).

We define the usual Frobenius norm $\|\cdot\|_{F}$ of a matrix A via

$$\|\textbf{A}\|_{F}=(\sum_{i,j=1}^{n}A_{ij}^{2})^{1/2}.$$

For positive integers $d$ and $n$, we will define the set of orthogonal matrices in $\mathbb{R}^{d\times d}$ via $\mathcal{O}_{d}$ and the set of $n\times n$ permutation matrices via $\Pi_{n}$. We indicate a matrix of all ones of size $d\times d$ by $\mathbf{J}_{d}$, and $\mathbf{J}_{n,d}$ is the matrix of all one of size $n\times d$. Similarly, we denote the corresponding matrices of all 0’s by $\textbf{0}_{d}=\{0\}^{d\times d}$ and $\textbf{0}_{n,d}=\{0\}^{n\times d}.$ Lastly, the direct sum of two matrices A and B is denoted by $\textbf{A}\oplus\textbf{B}$.

For functions $f,g:\mathbb{Z}_{\geq 0}\mapsto\mathbb{R}_{\geq 0}$, we use here the standard asymptotic notations:

	$\displaystyle f=O(g)$	$\displaystyle\text{ if }\exists\,C>0,\text{ and }n_{0}\in\mathbb{Z}_{\geq 0}\text{ s.t. }f(n)\leq Cg(n)\text{ for }n\geq n_{0};$
	$\displaystyle f=\Omega(g)$	$\displaystyle\text{ if }\exists\,C>0,\text{ and }n_{0}\in\mathbb{Z}_{\geq 0}\text{ s.t. }Cg(n)\leq f(n)\text{ for }n\geq n_{0};$
	$\displaystyle f=\Theta(g)$	$\displaystyle\text{ if }f=\Omega(g),\text{ and }f=O(g);\,\,f\sim g\text{ if }\lim_{n\rightarrow\infty}f(n)/g(n)=1;$
	$\displaystyle f=o(g)$	$\displaystyle\text{ if }\lim_{n\rightarrow\infty}f(n)/g(n)=0;\,f=\omega(g)\text{ if }\lim_{n\rightarrow\infty}g(n)/f(n)=0.$

Note that when $f=o(g)$ (resp., $f=\omega(g)$ and $f=\Theta(g)$) when $g$ is a complicated function of $n$, we will often write $f\ll g$ (resp., $f\gg g$ and $f\approx g$) to ease notation.

To formalize our partially aligned graph setting, suppose we have two networks $\textbf{A}_{1}$ and $\textbf{A}_{2}$ on a common vertex set $V(\textbf{A}_{1})=V(\textbf{A}_{2})=[n]$, and that the label correspondence across networks is known for $n-k$ of the vertices; denote the set of these vertices via $M_{n,k}$. We further assume that the user has knowledge of which vertices are in $M_{n,k}$. This, for example, could be the result of graph matching algorithms that provide a measure of certainty for the validity of each matched vertex (see, for example, the soft matching approach of [47] or the vertex nomination work of [48, 49]). The veracity of the correspondence across the remaining $k$ vertices not in $M_{n,k}$ (which we shall denote via $U_{n,k}:=[n]\setminus M_{n,k}$), is assumed unknown a priori. This may be due, for example, to algorithmic uncertainty in aligning nodes across networks or noise in the data. We then have that there exists an unknown (where $\Pi_{n}$ is the set of $n\times n$ permutation matrices)

$$\widetilde{\textbf{Q}}\in\Pi_{n,k}:=\{\textbf{Q}\in\Pi_{n}\text{ s.t. }\!\!\!\sum_{i\in U_{n,k}}\!\!\!Q_{ii}\leq k;\,\!\!\!\sum_{j\in M_{n,k}}\!\!\!Q_{jj}=n-k\}$$

such that the practitioner observes $\textbf{A}_{1}$ and $\textbf{B}_{2}=\widetilde{\textbf{Q}}\textbf{A}_{2}\widetilde{\textbf{Q}}^{T}$. Given the above framework, it is natural to consider the following semiparametric adaptation of the traditional parametric tests. From $\textbf{A}_{1}$ and $\textbf{B}_{2}$, the user seeks to test if the distribution of $\textbf{A}_{1}$ is different than from that of $\textbf{A}_{2}$, i.e., to test the following hypotheses (where $\mathcal{L}(\textbf{A}_{i})$ denotes the distribution (law) of $\textbf{A}_{i}$), $H_{0}:{\mathcal{L}}(\textbf{A}_{1})=\mathcal{L}(\textbf{A}_{2})$ versus $H_{1}:\mathcal{L}(\textbf{A}_{1})\neq\mathcal{L}(\textbf{A}_{2}).$ In this work, we will be considering testing within the family of (conditionally) edge-independent graph models, so that $\mathcal{L}(\textbf{A}_{i})$ is completely determined by $\textbf{P}_{i}=\mathbb{E}(\textbf{A}_{i})$. Focusing our attention first on a simple Frobenius-norm based hypothesis test (later considering the spectral embedding based tests of [45, 11, 12]), we reject $H_{0}$ if

$$T=T(\textbf{A}_{1},\textbf{B}_{2}):=\|\widehat{\textbf{P}}_{1}-\widehat{\textbf{P}}_{2,\widetilde{\textbf{Q}}}\|_{F}^{2}$$

is suitably large; here $\widehat{\textbf{P}}_{1}$ is an estimate of $\textbf{P}_{1}$ obtained from $\textbf{A}_{1}$, and $\widehat{\textbf{P}}_{2,\widetilde{\textbf{Q}}}$ an estimate of $\widetilde{\textbf{Q}}\textbf{P}_{2}\widetilde{\textbf{Q}}^{T}=\mathbb{E}(\textbf{B}_{2})$ (this will be formalized later in Section 1.5; for intuition and experimental validation on why the test statistic using $\widehat{\textbf{P}}$ is preferable to the adjacency-based test using $\|\textbf{A}_{1}-\textbf{B}_{2}\|_{F}^{2}$ as its statistic, see Section 3). To account for the uncertainty in the labeling of $\textbf{B}_{2}$, for each $\alpha>0$ and $\textbf{Q}\in\Pi_{n,k}$, define $c_{\alpha,\textbf{Q}}>0$ to be the smallest value such that

$$\mathbb{P}_{H_{0}}(\|\widehat{\textbf{P}}_{1}-\mathbf{Q}\widehat{\textbf{P}}_{2}\mathbf{Q}^{T}\|_{F}\geq c_{\alpha,\textbf{Q}})\leq\alpha.$$

As we do not know which element of $\Pi_{n,k}$ yields the shuffling from $\textbf{A}_{2}$ to $\textbf{B}_{2}$, a valid (conservative) level-$\alpha$ test using the Frobenius norm test statistic would reject $H_{0}$ if

$$T(\textbf{A}_{1},\textbf{B}_{2})>\max_{\textbf{Q}\in\Pi_{n,k}}c_{\alpha,\textbf{Q}}.$$

The price of this validity is a loss in testing power against any fixed alternative, especially in the scenario where $\widetilde{\textbf{Q}}$ (the true, but unknown, shuffling) shuffles fewer than $k$ vertices in $\textbf{A}_{2}$. In this case the conservative test is over-correcting for the uncertainty in $U_{n,k}$. The question we seek to answer is how much testing power is lost in this shuffle, and how robust the adaptations of different testing methods (i.e., different $T$ test statistics) are to this shuffling.

Note that our choice of Frobenius norm for the test statistic is natural here in light of the metric’s extensive use for network comparison; see, e.g., its use in the estimation, testing, and matching literatures [50, 51, 40].

To motivate the shuffled testing problem further, we first present the following real-data example. We consider the test/retest connectomic dataset from [52] processed via the algorithmic pipeline at [53] (note that this data is available for download at http://www.cis.jhu.edu/~parky/Microsoft/JHU-MSR/ZMx2/BNU1/DS01216-xyz.zip). This dataset represents human connectomes derived from DTMRI scans, where there are multiple (i.e., test/retest) scans per each of the 57 individuals in the study. We consider three such scans, yielding connectomes $\textbf{A}_{1}$ and $\textbf{A}_{2}$ and $\textbf{A}_{3}$. Here, $\textbf{A}_{1}$ and $\textbf{A}_{2}$ represent test/retest scans from one subject (subject 1) and $\textbf{A}_{3}$ a scan from a different subject (subject 2 scan 1). In each scan, vertices represent voxel regions of the brain with edges denoting whether a neuronal fiber bundle connects the two regions or not (so that the graphs are binary and undirected). Considering only vertices common to the three connectomes, we are left with three graphs each with $n=1085$ vertices.

For $k>0$, let $\textbf{Q}\in\Pi_{n,k}$ be an unknown permutation. Observing $\textbf{A}_{1}$ and $\textbf{A}_{2}$ and $\textbf{B}_{3}=\textbf{Q}\textbf{A}_{3}\textbf{Q}^{T}$ (rather than $\textbf{A}_{1},\textbf{A}_{2},$ and $\textbf{A}_{3}$) we seek to test whether $\textbf{B}_{3}$ is a connectome drawn from the same person as $\textbf{A}_{1}$ and $\textbf{A}_{2}$ or from a different person (i.e., under the reasonable assumption that $\mathcal{L}(\textbf{A}_{1})=\mathcal{L}(\textbf{A}_{2})$, we seek to test whether $\textbf{A}_{3}$ is from this same distribution). Ideally, we would then construct our test statistic as

$\displaystyle T(\textbf{A}_{i},\textbf{A}_{j})=\|\widehat{\textbf{P}}_{i}-\widehat{\textbf{P}}_{j}\|_{F}$

(1)

where $\widehat{\textbf{P}}_{i}$ is the estimate of $\textbf{P}_{1}=\mathbb{E}(\textbf{A}_{1})=\mathbb{E}(\textbf{A}_{2})=\textbf{P}_{2}$ or $\textbf{P}_{3}=\mathbb{E}(\textbf{A}_{3})$ derived from $\textbf{A}_{i}$ as in Section 1.5.

Incorporating the unknown shuffling of $U_{n,k}$ in $\textbf{B}_{3}$ is tricky here, as for moderate $k$ it is computationally infeasible to compute $c_{\alpha,\mathbf{Q}}$ for all $\mathbf{Q}\in\Pi_{n,k}$ (where we recall that $c_{\alpha,\mathbf{Q}}$ is the smallest value such that $\mathbb{P}(\|\widehat{\textbf{P}}_{1}-\mathbf{Q}\widehat{\textbf{P}}_{2}\mathbf{Q}^{T}\|_{F}^{2}\geq c_{\alpha,\textbf{Q}})\leq\alpha;$ indeed, the order of $\Pi_{n,k}$ is $k!$ given we know the vertices in $M_{n,k}$), and so it is difficult to compute the conservative critical value $\max_{\textbf{Q}\in\Pi_{n,k}}c_{\alpha,\mathbf{Q}}$. Here, the task of finding the worse-case shuffling in $\Pi_{n,k}$ can be cast as finding an optimal matching between one graph and the complement of the second, which we suspect is computationally intractable. To proceed forward, then, we consider the following modification of the overall testing regime: We consider a fixed (randomly chosen) sequence of nested permutations $\textbf{Q}_{k}\in\Pi_{n,k}$ for $k=0,\,50,\,150,\,200,\,250,\,350$ and consider shuffling $\textbf{A}_{2}$ by $\textbf{Q}_{k}$ and $\textbf{A}_{3}$ by $\textbf{Q}_{\ell}$ for all $\ell\leq k$. We then repeat this process $nMC=100$ times, each time obtaining an estimate (via bootstrapping as outlined below) of testing power against $H_{0}$. This is done here out of computational necessity, and although the test does not achieve level-$\alpha$ here, this will nonetheless be sufficient to demonstrate the dramatic loss in testing performance due to shuffling.

In order to estimate the testing power here, we rely on the bootstrapping heuristic inspired by [45, 51] for latent position networks. Informally, we will model $\textbf{A}_{1}$, $\textbf{A}_{2}$, and $\textbf{A}_{3}$ as instantiations of the Random Dot Product Graph (RDPG) modeling framework of [54].

The RDPG model posits a tractable parameterization of the networks provided by the latent position matrices, and there are a number of statistically consistent methods for estimating these parameters under a variety of model variants. Here, we will use the Adjacency Spectral Embedding (ASE) to estimate X and Y; see [55] for a survey of recent work in estimating and inference in RDPGs.

Estimated power for 200 bootstrapped samples of these test statistics at approximate level $\alpha=0.05$ are plotted in Figure 1 left panel (averaged over the $nMC=100$ Monte Carlo replicates). In the figure, the $x$-axis represents the number of vertices shuffled via $\textbf{Q}_{\ell}$ (from $0$ to $k$) while the curve colors represent the maximum number of vertices potentially shuffled via $\Pi_{k,n}$; here all shuffled by $\textbf{Q}_{k}$. As seen in figure, the power of this test increases as both $k$ and $\ell$ increase, implying that the test is able to correctly distinguish the difference between the two subjects when the effect of the shuffling is either minimal (small $k$) or when the shuffling is equally severe in both the null and alternative cases (i.e., $\textbf{Q}_{\ell}$ shuffles as much as $\textbf{Q}_{k}$). When $k$ is much bigger than $\ell$, the test is overly conservative (see Figure 1 right panel), as expected. In this case the shuffling in $H_{0}$ has the effect of inflating the critical value compared to the true (i.e., unshuffled) testing critical value, yielding an overly conservative test that cannot distinguish between the different test subjects.

As referenced above, to tackle the question of power loss statistically, we will anchor our analysis in commonly studied random graph models from the literature. In addition to the Random Dot Product Graph (RDPG) model [56, 54] mentioned above, we will also consider the Stochastic Blockmodel (SBM) [57] as a data generating mechanism. These models provide tractable settings for the analysis of graphs where the connectivity is driven by latent features—community membership in the SBM and the latent position vector in the RDPG.

The Stochastic Blockmodel—and its myriad variants including mixed membership [58], degree corrected [59], and hierarchical SBMs [25, 60]—provide a simple framework for networks with latent community structure.

In SBMs, the connectivity structure is driven by the latent community membership of the vertices. In the space of latent feature models, a natural extension of this idea is to have connectivity modeled as a function of more nuanced, vertex-level, features. In this direction, we will also consider framing our inference in the popular Random Dot Product Graph model introduced in Definition 1.1. Note that the RDPG model encompasses SBM models with positive semidefinite $\Lambda$. Indefinite and negative definite SBM’s, are encompassed via the generalized RDPG [61], though the ordinary RDPG will be sufficient for our present purposes. Note also that our theory will be presented for the fixed latent position RDPG above, though it translates immediately to the random latent position setting (i.e., where the rows of $\mathbf{X}$, namely the $X_{i}$, are i.i.d. from an appropriate distribution $F$).

As in [43], we will choose to control the sparsity of our graphs via $\nu$ and not through the latent position matrix $\mathbf{X}$ or block probability matrix $\Lambda$. As such, we will implicitly make the following assumption throughout the remainder for all RDPGs and positive semidefinite SBMs (when viewed as RDPGs):

In the RDPG (and positive semidefinite SBM) setting, our initial hypothesis test will be predicated upon having a suitable estimate of $\textbf{P}=\mathbb{E}(\textbf{A}|\textbf{X})$. In this setting, the Adjacency Spectral Embedding (ASE) (see Definition 1.2) of [21] has proven to be practically useful and theoretically tractable means for obtaining such an estimate. The adjacency spectral embedding has a rich, recent history in the literature (see [55]) as a tool for estimating tractable graph representations, achieving its greatest estimation strength in the class of latent position networks (the RDPG being one such example). In these settings, it is often assumed that the rank of the latent position matrix X is $d$, and that $d$ is considerably smaller than $n$, the number of vertices in the graph.

A great amount of inference in the RDPG setting is predicated upon $\widehat{\textbf{X}}$ being a suitably accurate estimate of X. To this end, the key statistical properties of consistency and asymptotic residual normality are established for the ASE in [21, 64, 61] and [65, 55] respectively. These results (and analogues for unscaled variants of the ASE) have laid the groundwork for myriad subsequent inference results, including clustering [21, 64, 22, 66], classification [24], time-series analysis [41, 67], and vertex nomination [49, 68], among others.

In this section, we will present a trio of theorems, namely Theorems 2.1–2.3, in which we characterize the impact on power of the two distinct sources of noise here: the shuffling error ($k$ and $\ell$) and the error in the alternative captured here by $\epsilon$. When the difference between $k$ and $\ell$ is comparably large (for example, when $\ell\leq r$ $\epsilon\ll\sqrt{(k-\ell)/r}$ and $\epsilon\ll\frac{k-\ell}{\ell}$), then the power of the resulting test will be low even in the presence of modest error $\epsilon$. In this case, the relative size of the error in the alternative is overwhelmed by the excess shuffling in the null which is needed to maintain testing level $\alpha$. The actual shuffling error (i.e., $\ell$) is much less than the conservative null shuffling (i.e., $k$), and the test is not able to distinguish the two graphs in light of the conservative test’s overcompensation. Even in the case where $k-\ell$ is relatively small, if the error $\epsilon$ is sufficiently small, we will have low testing power, as expected. However, when the difference between $k$ and $\ell$ is relatively small compared to $\epsilon$, or $k$ and $\ell$ are both relatively small compared to $\epsilon$ (see the conditions in Theorem 2.1 and Theorem 2.2), then the difference in the number of vertices being shuffled across the conservative null and truly shuffled in the alternative is overwhelmed by the error in the alternative. In this case, the noise created by the relatively small differences in shuffling between null and alternative can be overcome, and high power can still be achieved.