Information bottleneck theory of high-dimensional regression: relevancy, efficiency and optimality

Linear map—We consider training data of $N$ iid samples, $S\!=\!\{(x_{1},y_{1}),\dots,(x_{N},y_{N})\}$, each of which is a pair of $P$ dimensional input vector $x_{i}\!\in\!\mathbb{R}^{P}$ and scalar response $y_{i}\!\in\!\mathbb{R}$ for $i\!\in\!\{1,\dots,N\}$. We assume a linear relation between the input and response,

where $W\!\in\!\mathbb{R}^{P}$ denotes the unknown linear map and $\epsilon_{i}$ a scalar Gaussian noise with mean zero and variance $\sigma^{2}$. In other words the responses and the inputs are related via a Gaussian channel

where we define $Y\!=\!(y_{1},\dots,y_{N})^{\mkern-1.5mu\mathsf{T}}\!\in\!\mathbb{R}^{N}$ and $X\!=\!(x_{1},\dots,x_{N})\!\in\!\mathbb{R}^{P\times N}$.

Fixed design—We adopt the fixed design setting in which the inputs (design matrix) $X$ are deterministic and only the responses $Y$ are random variables (see, e.g., Ref [33, Ch 3]). As a result, the mutual information between the training data $S$ and any random variable $A$ is given by $I(A;S)\!=\!I(A;X,Y)\!=\!I(A;Y)$. In the following analyses, we use $S$ and $Y$ interchangeably.

Random effects—In addition we work in the random effects setting (see, e.g., [34, 35] for recent applications of this setting) in which the true regression parameter $W$ is a Gaussian vector,

Here we define the covariance such that the signal strength, $\operatorname{\mathbb{E}}\|W\|^{2}\!=\!\omega^{2}$, is independent of $P$.

The data generating process above results in training data $Y$ and true parameters $W$ that are Gaussian correlated (under the fixed design setting). In this case the IB optimization—minimizing residual information $I(T;Y\,|\,W)$ while maximizing relevant information $I(T;W)$—admits an exact solution [36], characterized by the eigenmodes of the normalized regression matrix,

where $\lambda^{*}$ denotes the scaled noise-to-signal ratio. The relevant and residual informations of an optimal representation $\tilde{T}$ read [36]

where $\nu_{i}$ denote the eigenvalues of $\Sigma_{Y|W}\Sigma_{Y}^{-1}$ and $\gamma$ parametrizes the IB trade-off²²2The arguments of the logarithms in Eqs (6-7) are always nonnegative since the data processing inequality means that the IB problem is well-defined only for $\gamma\!>\!1$, and the eigenvalues of a normalized regression matrix always range from zero to one [36]. [see Eq (1)]. In our setting it is convenient to recast the summations above as integrals (see Appendix A for derivation),

where $\Psi\!\equiv\!XX^{\mkern-1.5mu\mathsf{T}}/N$ and $F^{\Psi}$ denote the empirical covariance and its cumulative spectral distribution, respectively.³³3Note that the eigenvalues of $XX^{\mkern-1.5mu\mathsf{T}}$ and $X^{\mkern-1.5mu\mathsf{T}}X$ are identical except for the number of zero modes. In addition we introduce the parameter $\psi_{c}\!=\!\lambda^{*}/(\gamma-1)$ which controls the number and the weights of eigenmodes used in constructing the optimal representation $\tilde{T}$. In the limit $\psi_{c}\!\to\!0^{+}$, the residual information diverges logarithmically (Fig 1d) and the relevant information converges to the available relevant information in the data (Fig 1c),

Increasing $\psi_{c}$ from zero decreases both residual and relevant informations, tracing out the optimal frontier until the lower spectral cutoff $\psi_{c}$ reaches the upper spectral edge at which both informations vanish (Fig 1b) and beyond which no informative solution exists [36, 37, 38].

The exact characterization of the IB frontier provides a useful benchmark for information-theoretic analyses of learning algorithms, not least because it allows a precise definition of information efficiency. That is, we can now ask how many more residual bits a given algorithm needs to encode, compared to the IB optimal algorithm, in order to achieve the same level of relevant information. Here we define the information efficiency $\eta_{\mu}$ as the ratio between the residual bits encoded in the outputs of the IB optimal algorithm and the algorithm of interest—$\tilde{T}$ and $T$, respectively—at some fixed relevance level $\mu$, i.e.,

Since the optimal representation minimizes residual bits at fixed $\mu$ (Fig 1a), the information efficiency ranges from zero to one, $0\!\leq\!\eta_{\mu}\!\leq\!1$. In addition we have $0\!<\!\mu\!\leq\!1$, resulting from the data processing inequality $I(T;W)\!\leq\!I(Y;W)$ for the Markov constraint $T\!\leftrightarrow\!Y\!\leftrightarrow\!W$ (see, e.g., Ref [39]).

At first sight it appears that our analyses are not applicable in the high-information limit since the residual information diverges for both the optimal algorithm and Gibbs regression (see Figs 1d & 2c). However the rates of divergence differ. Here we use this difference to characterize the efficiency of Gibbs regression in the zero temperature limit $\beta\!\to\!\infty$.

We first analyze the limiting behaviors of relevant information. From Eq (10), we see that the relevant information ratio of the IB solution approaches one at $\psi_{c}\!=\!0$. Perturbing $\psi_{c}$ in Eq (8) away from zero results in a linear correction to the relevant information,

Keeping only the leading correction and recalling that $\mu\!=\!I(\tilde{T};W)/I(Y;W)$ [Eq (11)], we obtain

Similarly expanding the relevant information of Gibbs regression [Eq (19)] around $\beta\!\to\!\infty$ yields

As a result, the correspondence between the low-temperature and high-information limits reads

We turn to the residual bits. Expanding Eq (9) around $\psi_{c}\!=\!0$ and Eq (20) around $\beta^{-1}\!=\!0$ leads to

From Eqs (23) & (25), we see that the residual informations above have the same logarithmic singularity, $\ln(1-\mu)$, at $\mu\!=\!1$. Therefore their difference remains finite even as $\mu\!\to\!1$. Combining Eqs (23) & (25-27) and recalling our definition of information efficiency $\eta_{\mu}\!=\!I(\tilde{T};Y\,|\,W)/I(T;Y\,|\,W)$ [Eq (11)] gives

Note that Jensen’s inequality guarantees that the terms in the parentheses sum to a non-negative value.

It is worth pointing out that, at $\lambda\!=\!\lambda^{*}$, the correction term in Eq (28) vanishes and the efficiency of deterministic Gibbs regression becomes minimally sensitive to algorithmic noise. Incidentally, this value of $\lambda$ also minimizes the $L_{2}$ prediction error of ridge regression in the asymptotic limit [40].

For $\Sigma\!=\!I_{P}$, the empirical spectral distribution converges to to the standard Marchenko-Pastur law [49]

where $\psi_{\pm}\!=\!(1\pm 1/\sqrt{n})^{2}$ and $F^{\Psi}(0)\!=\!\max(0,1-n)$. We use this spectral distribution in Figs 1-2.

Optimal algorithm—In Fig 1b, the IB optimal frontiers illustrate the fundamental trade-off; optimal algorithms cannot encode fewer residual bits without becoming less relevant. Figure 1c-d shows that encoded relevant and residual bits go down as $\psi_{c}$ increases and fewer eigenmodes contribute to the IB optimal representation [Eqs (8-9)]. However relevant and residual informations exhibit different behaviors at high information; as $\psi_{c}\!\to\!0$, relevant information plateaus whereas residual information diverges logarithmically [see also Eq (26)].

Gibbs regression—Figure 2a depicts the information content of Gibbs regression at different regularization strengths [Eqs (19-20)] and illustrates the fundamental trade-off, similarly to the IB frontier (dotted) but at a lower relevance level. Here the information curves are parametrized by the inverse temperature $\beta$ which controls the algorithmic stochasticity; Gibbs posteriors become deterministic as $\beta\!\to\!\infty$ and completely random at $\beta\!=\!0$ [Eq (14)]. In Fig 2b-c, we see that Gibbs regression encodes fewer relevant and residual bits as temperature goes up. Decreasing Gibbs temperature results in an increase in encoded information. In the zero-temperature limit, the relevant bits saturate while the residual bits grow logarithmically (cf. Fig 1c-d; see Sec 3.1 for a detailed analysis of this limit). The amount of encoded information depends also on the regularization strength $\lambda$. Figure 2b-c shows that, at a fixed temperature, an increase in $\lambda$ leads to less information extracted. However this does not necessarily mean that a larger $\lambda$ hurts information efficiency. Indeed a lower temperature can compensate for the decrease in information. In Fig 2a, we see that the information curves can be closer to the optimal frontier as $\lambda$ increases. In general the maximum efficiency occurs at an intermediate regularization strength that depends on data structure and measurement density (see also Sec 4.2).

Efficiency—Figure 2d displays the information efficiency of Gibbs regression at different relevant information levels (see Sec 1.3). We see that the efficiency approaches optimality ($\eta_{\mu}\!=\!1$) in the limits $n\!\to\!0$ and $n\!\to\!\infty$. Away from these limits, Gibbs regression requires more residual bits than the optimal algorithm to achieve the same level of relevance with an efficiency minimum around $n\!=\!1$. We also see that the efficiency of Gibbs regression decreases with relevance level (see also Supplementary Figure in Appendix D).

Extensivity—Learning is qualitatively different in the over- and underparametrized regimes. In Fig 2e we see that both optimal algorithms and Gibbs regression exhibit nonmonotonic dependence on sample size. In the overparametrized regime $n\!<\!1$, the residual information is extensive in sample size, i.e., it grows linearly with $N$. This scaling behavior mirrors that of the relevant bits in the data (Fig 1b inset). But unlike the available relevant bits which continue to grow in the data-abundant regime, albeit sublinearly—the encoded residual bits decrease with sample size in this limit (see also Supplementary Figure in Appendix D). The resulting maximum is an information-theoretic analog of double descent—the decrease in overfitting level (test error) as the number of parameters exceeds sample size (decreasing $n$) [48, 31].

Redundancy—Indeed we could have anticipated the extensive behavior of the residual bits in the overparametrized regime (Fig 2e). In this limit, the extensivity of available relevant bits implies that the data encode relevant information with no redundancy. In other words, the relevant bits in one observation do not overlap with that in another. As a result, the dominant learning strategy is to treat each sample separately and extract the same amount of information from each of them, thus resulting in extensive residual information. In the data-abundant regime, on the other hand, the coding of relevant bits in the data becomes increasingly redundant (Fig 1b inset). Learning algorithms exploit this redundancy to better distinguish signals from noise, thereby encoding fewer residual bits.

To explore the effects of anisotropy, we consider a two-scale model in which the population spectral distribution $F^{\Sigma}$ is an equal mixture of two point masses at $s_{+}$ and $s_{-}$. We normalize the trace of the population covariance such that the signal variance, and thus the signal-to-noise ratio, does not depend on $F^{\Sigma}$—i.e., we set $\operatorname{tr}\Sigma/P\!=\!(s_{+}+s_{-})/2\!=\!1$ such that $\operatorname{\mathbb{E}}[(W\cdot x_{i})^{2}]\!=\!\operatorname{\mathbb{E}}\|W\|^{2}\!=\!\omega^{2}$. As a result the anisotropy in our two-scale model is parametrized completely by the eigenvalue ratio $r\!\equiv\!s_{-}/s_{+}$.

Unlike the isotropic case, the limiting empirical spectral distribution does not admit a closed form expression. We obtain $F^{\Psi}$ by solving the Silverstein equation and inverting the resulting Stieltjes transform [51] (see Appendix C). Figure 3c depicts the spectral density at various anisotropy ratios and measurement densities. At high measurement densities $n\!\gtrsim\!1$, anisotropy splits the continuum part of the spectrum into two bands, corresponding to the two modes of the population covariance. These bands broaden as $n$ decreases and eventually merge into one in the overparametrized limit.

Available information—In Fig 3a, we see that anisotropy decreases the relevant information in the data, but does not affect its qualitative behaviors: the available relevant bits are extensive in the overparametrized regime and subextensive in the data-abundant regime. Although fewer relevant bits are available, learning needs not be less information efficient. Indeed the IB frontiers in Fig 3b illustrate that it takes fewer residual bits in the anisotropic case to reach the same level of relevance as in the isotropic case. This behavior is also apparent in Fig 3d (dotted) as we increase anisotropy levels (from right to left panels).

Anisotropy effects—Anisotropy affects optimal algorithms via the different scales in the population covariance. The signals along high-variance (easy) directions are stronger and, as a result, the coding of relevant bits in these directions becomes subextensive and redundant around $n\!\approx\!1/2$ (the proportion of easy directions) instead of at one (Fig 3a). This earlier onset of redundancy allows for more effective signal-noise discrimination in the anisotropic case (Fig 3b & d). In the data-abundant limit, however, low-variance (hard) directions become important as learning algorithms already encode most of the relevant bits along the easy directions. Indeed the hard directions are harder for more anisotropic inputs and thus the required residual bits increase with anisotropy in the limit $n\!\to\!\infty$ (Fig 3d).

Triple descent—Perhaps the most striking effect of anisotropy is the emergence of an information-theoretic analog of (sample-wise) multiple descent, which describes disjoint regions where more data makes overfitting worse (larger test error) [32, 52]. In Fig 3d, we see that an additional residual information maximum emerges at large $n$. This behavior is a consequence of the separation of scales. The first maximum at $n\!\sim\!1$ originates from easy directions and the other maximum at higher $n$ from hard directions. In fact the IB cutoff $\psi_{c}$ in Fig 3c demonstrates that the residual information maxima roughly coincide with the inclusion of all high-variance modes around $n\!\sim\!1$ and low-variance modes at higher $n$.⁵⁵5Note that Fig 3c does not show the zero modes which are present at $n\!<\!1$. The fact that the spectral continuum appears to be above the IB cutoff at small $n$ does not mean all eigenmodes are used in the IB solution. In addition we note that for optimal algorithms the first maximum shifts to a lower $n$ as the anisotropy level increases. This observation is consistent with the fact that the onset of redundancy of relevant bits in the data occurs at smaller $n$ in the anisotropic case (Fig 3a).

Gibbs regression—Anisotropy makes Gibbs regression depend more strongly on regularization strengths, see Fig 3. In particular the information efficiency decreases with $\lambda$ near the first residual information minimum around $n\!\sim\!1$ but this dependence reverses near the second maximum and at larger $n$. This behavior is expected. Inductive bias from strong regularization helps prevent noise from poisoning the models at small $n$. But when the data become abundant, regularization is unnecessary.