Concentration and regularization of random graphs

The cleanest concentration results are available for the classical Erdös-Rényi model $G(n,p)$ in the dense regime. In terms of the expected degree $d=pn$, we have with high probability that

see [16, 44, 28]. Since $\|\operatorname{\mathbb{E}}A\|=d$, we see that the typical deviation here behaves like the square root of the magnitude of expectation – just like in many other classical results of probability theory. In other words, dense random graphs concentrate well.

The lower bound on density in (1.1) can be essentially relaxed all the way down to $d=\Omega(\log n)$. Thus, with high probability we have

This result was proved in [15] based on the method developed by J. Kahn and E. Szemeredi [17]. More generally, (1.2) holds for any inhomogeneous Erdös-Rényi model $G(n,(p_{ij}))$ with maximal expected degree $d=\max_{i}\sum_{j}p_{ij}$. This generalization can be deduced from a recent result of S. Bandeira and R. van Handel [4, Corollary 3.6], while a weaker bound $O(\sqrt{d\log n})$ follows from concentration inequalities for sums of independent random matrices [35]. Alternatively, an argument in [15] can be used to prove (1.2) for a somewhat larger but still useful value

see [27, 12]. The same can be obtained by using Seginer’s bound on random matrices [20]. As we will see shortly, our paper provides an alternative and completely different approach to general concentration results like (1.2).

In the sparse regime, where the expected degree $d$ is bounded, concentration breaks down. According to [24], a random graph from $G(n,p)$ satisfies with high probability that

where $d(A)$ denotes the maximal degree of the graph (a random quantity). So in this regime we have $\|A\|\gg\|\operatorname{\mathbb{E}}A\|=d$, which shows that sparse random graphs do not concentrate.

What exactly makes the norm $A$ abnormally large in the sparse regime? The answer is, the vertices with too high degrees. In the dense case where $d\gg\log n$, all vertices typically have approximately the same degrees $(1+o(1))d$. This no longer happens in the sparser regime $d\ll\log n$; the degrees do not cluster tightly about the same value anymore. There are vertices with too high degrees; they are captured by the second inequality in (1.4). Even a single high-degree vertex can blow up the norm of the adjacency matrix. Indeed, since the norm of $A$ is bounded below by the Euclidean norm of each of its rows, we have $\|A\|\geq\sqrt{d(A)}$.

If high-degree vertices destroy concentration, can we “tame” these vertices? One proposal would be to remove these vertices from the graph altogether. U. Feige and E. Ofek [15] showed that this works for $G(n,p)$ – the removal of the high degree vertices enforces concentration. Indeed, if we drop all vertices with degrees, say, larger than $2d$, the the remaining part of the graph satisfies

with high probability, where $A^{\prime}$ denotes the adjacency matrix of the new graph. The argument in [15] is based on the method developed by J. Kahn and E. Szemeredi [17]. It extends to the inhomogeneous Erdös-Rényi model $G(n,(p_{ij}))$ with $d$ defined in (1.3), see [27, 12]. As we will see, our paper provides an alternative and completely different approach to such results.

Although the removal of high degree vertices solves the concentration problem, such solution is not ideal, since those vertices are in some sense the most important ones. In real-world networks, the vertices with highest degrees are “hubs” that hold the network together. Their removal would cause the network to break down into disconnected components, which leads to a considerable loss of structural information.

Would it be possible to regularize the graph in a more gentle way – instead of removing the high-degree vertices, reduce the weights of their edges just enough to keep the degrees bounded by $O(d)$? The main result of our paper states that this is true. Let us first state this result informally; Theorem 2.1 provides a more general and formal statement.

The regularization procedure in Theorem 1.1 is very flexible. Depending on how one chooses the weights, one can obtain as partial cases several results we summarized earlier, as well as some new ones.

1. 1.

   Do not do anything to the graph. In the dense regime where $d=\Omega(\log n)$, all degrees are already bounded by $2d$ with high probability. This means that the original graph satisfies $\|A-\operatorname{\mathbb{E}}A\|=O(\sqrt{d})$. Thus we recover the result of U. Feige and E. Ofek (1.2), which states that dense random graphs concentrate well.

2. 2.

   Remove all high degree vertices. If we remove all vertices with degrees larger than $2d$, we recover another result of U. Feige and E. Ofek (1.5), which states that the removal of the high degree vertices enforces concentration.

3. 3.

   Remove just enough edges from high-degree vertices. Instead of removing the high-degree vertices with all of their edges, we can remove just enough edges to make all degrees bounded by $2d$. This milder regularization still produces the concentration bound (1.5).

4. 4.

   Reduce the weight of edges proportionally to the excess of degrees. Instead of removing edges, we can reduce the weight of the existing edges, a procedure which better preserves the structure of the graph. For instance, we can assign weight $\sqrt{\lambda_{i}\lambda_{j}}$ to the edge between vertices $i$ and $j$, choosing $\lambda_{i}:=\min(2d/d_{i},1)$ where $d_{i}$ is the degree of vertex $i$. One can check that this makes the $\ell_{2}$ norms of all rows of the adjacency matrix bounded by $2d$. By Theorem 1.1, such regularization procedure leads to the same concentration bound (1.5).

So far, we have looked at random graphs through the lens of their adjacency matrices. A different matrix that captures the geometry of a graph is the (symmmetric, normalized) Laplacian matrix, defined as

Here $I$ is the identity matrix and $D=\operatorname{diag}(d_{i})$ is the diagonal matrix with degrees $d_{i}=\sum_{j=1}^{n}A_{ij}$ on the diagonal. The reader is referred to [13] for an introduction to graph Laplacians and their role in spectral graph theory. Here we mention just two basic facts: the spectrum of $\mathcal{L}(A)$ is a subset of $[0,2]$, and the smallest eigenvalue is always zero.

Concentration of Laplacians of random graphs has been studied in [35, 11, 39, 23, 18]. Just like the adjacency matrix, the Laplacian is known to concentrate in the dense regime where $d=\Omega(\log n)$, and it fails to concentrate in the sparse regime. However, the obstructions to concentration are opposite. For the adjacency matrices, as we mentioned, the trouble is caused by high-degree vertices. For the Laplacian, the problem lies with low-degree vertices. In particular, for $d=o(\log n)$ the graph is likely to have isolated vertices; they produce multiple zero eigenvalues of $\mathcal{L}(A)$, which are easily seen to destroy the concentration.

In analogy to our discussion of adjacency matrices, we can try to regularize the graph to “tame” the low-degree vertices in various ways, for example remove the low-degree vertices, connect them to some other vertices, artificially increase the degrees $d_{i}$ in the definition (1.6) of Laplacian, and so on. Here we will focus on the following simple way of regularization proposed in [3] and analyzed in [23, 18]. Choose $\tau>0$ and add the same number $\tau/n$ to all entries of the adjacency matrix $A$, thereby replacing it with

in the definition (1.6) of the Laplacian. This regularization raises all degrees $d_{i}$ to $d_{i}+\tau$. If we choose $\tau\sim d$, the regularized graph does not have low-degree vertices anymore.

The following consequence of Theorem 1.1 shows that such regularization indeed forces Laplacian to concentrate. Here we state this result informally; Theorem 4.1 provides a more formal statement.

We will deduce this result from Theorem 1.1 in Section 4. Theorem 1.2 is an improvement upon a bound in [18] that had an extra $\log d$ factor, and it was conjectured there that the logarithmic factor is not needed. Theorem 1.2 confirms this conjecture.

To conclude our discussion of various ways to regularize sparse graphs, let us illustrate the effect of regularization by a numerical experiment. Consider an inhomogeneous Erdös-Rényi graph with $n=1000$ vertices, $90\%$ of which have expected degrees $7$ and $10\%$ percent have expected degrees $35$. We then regularize the graph by reducing the weights of edges proportionally to the excess of degrees – just as we described in Section 1.4 item 4, except that we use the overall average degree (approximately $10$) instead of $d$ (which results in a more severe weight reduction suitable for our illustration purpose).

Figure 1 shows the histogram of the spectrum of $A$ (left) and $A^{\prime}$ (right). As we can see, the high degree vertices lead to the long tails in the histogram of the eigenvalues, and regularization shrinks these tails toward the bulk.

Concentration of random graphs has an important application to statistical analysis of networks, in particular to the problem of community detection. A common way of modeling communities in networks is the stochastic block model [22], which is a special case of the inhomogeneous Erdös-Rényi model considered in this paper. For the purpose of this example, we focus on the simplest version of the stochastic block model $G(n,\frac{a}{n},\frac{b}{n})$, also known as the balanced planted partition model, defined as follows. The set of vertices is divided into two subsets (communities) of size $n/2$ each. Edges between vertices are drawn independently with probability $a/n$ if they are in the same community and with probability $b/n$ otherwise.

The community detection problem is to recover the community labels of vertices from a single realization of the random graph model. A large literature exists on both the recovery algorithms and the theory establishing when a recovery is possible [14, 33, 34, 32, 1, 29, 8]. There are methods that perform better than a random guess (i.e. the fraction of misclassified vertices is bounded away from $0.5$ as $n\to\infty$ with high probability) under the condition

and no method can perform better than a random guess if this condition is violated.

Moreover, strong consistency, or exact recovery (labeling all vertices correctly with high probability) is possible when the expected degree $(a+b)/2$ is of order $\log n$ or larger and $a$ and $b$ are sufficiently separated, see [32, 30, 6, 20, 10]. Weak consistency (the fraction of mislabeled vertices going to 0 with high probability) is achievable if and only if

see [32]. Many of these results hold in the non-asymptotic regime, for graphs of fixed size $n$. Thus, for any $\varepsilon>0$ there exists $C_{\varepsilon}$ (which only depends on $\varepsilon$) such that one can recover communities up to $\varepsilon n$ mislabeled vertices as long as

In particular, recovery of communities is possible even for very sparse graphs – those with bounded expected degrees. Several types of algorithms are known to succeed in this regume, including non-backtracking walks [33, 29, 8], spectral methods [12] and methods based on semidefinite programming [19, 31].

As an application of the new concentration results, we show that the regularized spectral clustering [3, 23], one of the simplest most popular algorithms for community detection, can recover communities in the sparse regime. In general, spectral clustering works by computing the leading eigenvectors of either the adjacency matrix or the Laplacian or their regularized versions, and running the $k$-means clustering algorithm on these eigenvectors to recover the node labels. In the simple case of the model $G(n,\frac{a}{n},\frac{b}{n})$, one can simply assign nodes to communities based on the sign (positive or negative) of the corresponding entries of the eigenvector $v_{2}(\mathcal{L}(A_{\tau}))$ corresponding to the second smallest eigenvalue of regularized Laplacian matrix $\mathcal{L}(A_{\tau})$ (or the regularized adjacency matrix $A^{\prime}$).

Let us briefly explain how our concentration results validate regularized spectral clustering. If the concentration of random graphs holds and $\mathcal{L}(A_{\tau})$ is close to $\mathcal{L}(\operatorname{\mathbb{E}}A_{\tau})$, then the standard perturbation theory (Davis-Kahan theorem below) shows that $v_{2}(\mathcal{L}(A_{\tau}))$ is close to $v_{2}(\mathcal{L}(\operatorname{\mathbb{E}}A_{\tau}))$, and in particular, the signs of these two eigenvectors must agree on most vertices. An easy calculation shows that the signs of $v_{2}(\mathcal{L}(\operatorname{\mathbb{E}}A_{\tau}))$ recover the communities exactly: this vector is a positive constant on one community and a negative constant on the other. Therefore, the signs of $v_{2}(\mathcal{L}(A_{\tau}))$ must recover the communities up to a small fraction of misclassified vertices.

Before stating our result, let us quote a simple version of the Davis-Kahan theorem perturbation theorem (see e.g. [5, Theorem VII.3.2]).

In Section 2, we state a formal version of Theorem 1.1. We show there how to deduce this result from a new decomposition of random graphs, which we state as Theorem 2.6. We prove this decomposition theorem in Section 3. In Section 4, we state and prove a formal version of Theorem 1.2 about the concentration of the Laplacian. We conclude the paper with Section 5 where we propose some questions for further investigation.

The authors are grateful to Ramon van Handel for several insightful comments on the preliminary version of this paper.

In this section, we reduce Theorem 2.1 to the following decomposition of inhomogeneous Erdös-Rényi directed random graphs. This decomposition may have an independent interest. Throughout the paper, we denote by $B_{\mathcal{N}}$ the matrix which coincides with a matrix $B$ on a subset of edges $\mathcal{N}\subset[n]\times[n]$ and has zero entries elsewhere.

Figure 2 illustrates a possible decomposition Theorem 2.6 can provide. The edges in $\mathcal{N}$ form a big “core” where the graph concentrates well even without regularization. The edges in $\mathcal{R}$ and $\mathcal{C}$ can be thought of (at least heuristically) as those attached to high-degree vertices.

We will prove Theorem 2.6 in Section 3; let us pause to deduce Theorem 2.1 from it.

First, let us explain informally how the graph decomposition could lead to Theorem 2.1. The regularization of the graph does not destroy the properties of $\mathcal{N}$, $\mathcal{R}$ and $\mathcal{C}$ in Theorem 2.6. Moreover, regularization creates a new property for us, allowing for a good control of the columns of $\mathcal{R}$ and rows of $\mathcal{C}$. Let us focus on $A_{\mathcal{R}}$ to be specific. The $\ell_{1}$ norms of all columns of this matrix are at most $d^{\prime}$, and the $\ell_{1}$ norms of all rows are $O(r)$ by Theorem 2.6. By a simple calculation which we will do in Lemma 2.7, this implies that $\|A_{\mathcal{R}}\|=O(\sqrt{rd^{\prime}})$. A similar bound can be proved for $\mathcal{C}$. Combining $\mathcal{N}$, $\mathcal{R}$ and $\mathcal{C}$ together will lead to the error bound $O(r^{3/2}(\sqrt{d}+\sqrt{d^{\prime}}))$ in Theorem 2.1.

To make this argument rigorous, let us start with the simple calculation we just mentioned.

We are ready to formally deduce the main part of Theorem 2.1 from Theorem 2.6; we defer the “moreover” part to Section 3.6.

We will construct the decomposition in Theorem 2.6 by an iterative procedure. The first and crucial step is to find a big block¹¹1In this paper, by block we mean a product set $I\times J$ with arbitrary index subsets $I,J\subset[n]$. These subsets are not required to be intervals of successive integers. $\mathcal{N}^{\prime}\subset[n]\times[n]$ of size at least $(n-n/d)\times n/2$ on which $A$ concentrates, i.e.

To find such block, we first establish concentration in $\ell_{\infty}\to\ell_{2}$ norm; this can be done by standard probabilistic techniques. Next, we can automatically upgrade this to concentration in the spectral norm ($\ell_{2}\to\ell_{2}$) once we pass to an appropriate block $\mathcal{N}^{\prime}$. This can be done using a general result from functional analysis, which we call Grothendieck-Pietsch factorization.

Repeating this argument for the transpose, we obtain another block $\mathcal{N}^{\prime\prime}$ of size at least $n/2\times(n-n/d)$ where the graph concentrates as well. So the graph concentrates on $\mathcal{N}_{0}:=\mathcal{N}^{\prime}\cup\mathcal{N}^{\prime\prime}$. The “core” $\mathcal{N}_{0}$ will form the first part of the class $\mathcal{N}$ we are constructing.

It remains to control the graph on the complement of $\mathcal{N}_{0}$. That set of edges is quite small; it can be described as a union of a block $\mathcal{C}_{0}$ with $n/d$ rows, a block $\mathcal{R}_{0}$ with $n/d$ columns and an exceptional $n/2\times n/2$ block; see Figure 3(b) for illustration. We may consider $\mathcal{C}_{0}$ and $\mathcal{R}_{0}$ as the first parts of the future classes $\mathcal{C}$ and $\mathcal{R}$ we are constructing.

Indeed, since $\mathcal{C}_{0}$ has so few rows, the expected number of ones in each column of $\mathcal{C}_{0}$ is bounded by $1$. For simplicity, let us think that all columns of $\mathcal{C}_{0}$ have $O(1)$ ones as desired. (In the formal argument, we will add the bad columns to the exceptional block.) Of course, the block $\mathcal{R}_{0}$ can be handled similarly.

At this point, we decomposed $[n]\times[n]$ into $\mathcal{N}_{0}$, $\mathcal{R}_{0}$, $\mathcal{C}_{0}$ and an exceptional $n/2\times n/2$ block. Now we repeat the process for the exceptional block, constructing $\mathcal{N}_{1}$, $\mathcal{R}_{1}$, and $\mathcal{C}_{1}$ there, and so on. Figure 3(c) illustrates this process. At the end, we choose $\mathcal{N}$, $\mathcal{R}$ and $\mathcal{C}$ to be the unions of the blocks $\mathcal{N}_{i}$, $\mathcal{R}_{i}$ and $\mathcal{C}_{i}$ respectively.

Two precautions have to be taken in this argument. First, we need to make concentration on the core blocks $\mathcal{N}_{i}$ better at each step, so that the sum of those error bounds would not depend of the total number of steps. This can be done with little effort, with the help of the exponential decrease of the size of the blocks $\mathcal{N}_{i}$. Second, we have a control of the sizes but not locations of the exceptional blocks. Thus to be able to carry out the decomposition argument inside an exceptional block, we need to make the argument valid uniformly over all blocks of that size. This will require us to be delicate with probabilistic arguments, so we can take a union bound over such blocks.

As we mentioned in the previous section, our proof of Theorem 2.6 is based on Grothendieck-Pietsch factorization. This general and well known result in functional analysis [36, 37] has already been used in a similar probabilistic context, see [26, Proposition 15.11].

Grothendieck-Pietsch factorization compares two matrix norms, the $\ell_{2}\to\ell_{2}$ norm (which we call the spectral norm ) and the $\ell_{\infty}\to\ell_{2}$ norm. For a $k\times m$ matrix $B$, these norms are defined as

The $\ell_{\infty}\to\ell_{2}$ norm is usually easier to control, since the supremum is taken with respect to the discrete set $\{-1,1\}^{m}$, and any vector there has all coordinates of the same magnitude.

To compare the two norms, one can start with the obvious inequality

Both parts of this inequality are optimal, so there is an unavoidable slack between the upper and lower bounds. However, Grothendieck-Pietsch factorization allows us to tighten the inequality by changing $B$ sightly. The next two results offer two ways to change $B$ – introduce weights and pass to a sub-matrix.

This result is a known combination of the Little Grothendieck Theorem (see [41, Corollary 10.10] and [38]) and Pietsch Factorization (see [41, Theorem 9.2]). In an explicit form, a version of this result can be found e.g. in [26, Proposition 15.11]. The weights $\mu_{j}$ can be computed algorithmically, see [42].

The following related version of Grothendieck-Pietsch’s factorization can be especially useful in probabilistic contexts, see [26, Proposition 15.11]. Here and in the rest of the paper, we denote by $B_{I\times J}$ the sub-matrix of a matrix $B$ with rows indexed by a subset $I$ and columns indexed by a subset $J$.

We are starting to work toward constructing the core part $\mathcal{N}$ in Theorem 2.6. In this section we will show how to find a big block on which the adjacency matrix $A$ concentrates. First we will establish concentration in $\ell_{\infty}\to\ell_{2}$ norm, and then, using Grothendieck-Pietsch factorization, in the spectral norm.

The lemmas of this and next section should be best understood for $m=n$, $I=J=[n]$ and $\alpha=1$. In this case, we are working with the entire adjacency matrix, and trying to make the first step in the iterative procedure. The further steps will require us to handle smaller blocks $I\times J$; the parameter $\alpha$ will then become smaller in order to achieve better concentration for smaller blocks.

Applying Lemma 3.3 followed by Grothendieck-Piesch factorization (Theorem 3.2), we obtain the following.

The two simple lemmas of this section will help us to handle the part of the adjacency matrix outside the core block constructed in Lemma 3.4. First, we show that almost all rows have at most $O(\alpha d)$ ones, and thus are included in the core block.

Next, we handle the block of rows that do have too many ones. We show that most columns of this block have $O(1)$ ones.