Improved dimension dependence in the Bernstein-von Mises Theorem via a new Laplace approximation bound

Before conveying our results, we first raise an additional important consideration in the study of high-dimensional BvMs. Clearly, the TV distance between $\pi_{n}$ and $\gamma^{*}_{n}$ depends not only on $d$ and $n$, but also on the statistical model for the data-generating process. And in the high-dimensional regime, the contribution to the TV error stemming from the model can itself depend on $d$. Thus the condition on the growth of $n$ relative to $d$ required for asymptotic normality is model-dependent.

To demonstrate this phenomenon, consider the aforementioned works [8] and [25]. In [8], one can read the proof of the main result Theorem 3.7 to find that the intermediate Proposition 3.10 implies the bound

$$\mathrm{TV}(\pi_{n},\gamma^{*}_{n})=\mathcal{O}(\sqrt{d^{3+\epsilon}/n{\theta_{\mathrm{min}}^{*}}})$$

(1.3)

for any $\epsilon>0$. We have assumed a flat prior for simplicity, i.e. we set $w_{n}\equiv 1$ in the statement of the proposition, and made some other simplifications. Here, ${\theta_{\mathrm{min}}^{*}}$ is the minimum probability in the unknown probability mass function over $d+1$ states, and therefore ${\theta_{\mathrm{min}}^{*}}\leq 1/(d+1)$. Thus $n\geq d^{3+\epsilon}/{\theta_{\mathrm{min}}^{*}}\geq d^{4+\epsilon}$ is required.

In the work [25] on the BvM for inverse problems, it is shown that

$$\mathrm{TV}(\pi_{n},\gamma^{*}_{n})=\mathcal{O}\left(\sigma(d)^{2}\log\left(d\vee\sigma(d)\right)\sqrt{d^{3}/n}\right),$$

(1.4)

where $\sigma(d)$ characterizes the degree of ill-posedness of the inverse problem. The author states that $\sigma(d)$ grows as a power of $d$ for mildly ill-posed inverse problems, and exponentially in $d$ for severely ill-posed inverse problems.

To meaningfully compare BvM results across different works studying different statistical models, we separate the contribution to the TV error into a “model-dependent” factor and a “universal” factor. We informally define the universal factor as the term in the error bound depending on $d$ and $n$ only. For example, $1/\sqrt{{\theta_{\mathrm{min}}^{*}}}$ and $\sigma(d)^{2}\log(d\vee\sigma(d))$ are the model-dependent terms in the above TV bounds of [8] and [25], respectively, while $\sqrt{d^{3+\epsilon}/n}$ and $\sqrt{d^{3}/n}$ are the universal factors, respectively. Thus to clarify our above overview of prior works, the stated condition $n\gg d^{3}$ stems from the universal factor appearing in the bounds.

We prove nonasymptotic statements of the BvM for several representative statistical settings, showing for the first time that, up to model-dependent factors, $n\gg d^{2}$ suffices for $\mathrm{TV}(\pi_{n},\gamma^{*}_{n})\to 0$. Specifically, we prove this for posteriors $\pi_{n}$ over an unknown parameter $\theta$ in the following settings: 1) arbitrary generalized linear models (GLMs) with data $Y_{i}\sim p(\cdot\mid X_{i}^{\intercal}\theta)$, $i=1,\dots,n$ for an exponential family $p$ in canonical form and 2) a multinomial observation $Y^{n}\sim\mathrm{Multinomial}(n,\theta)$, where $\theta$ is an unknown probability mass function (pmf) on $d+1$ states (as in [8]). In each case, we show that, up to negligible terms,

$$\mathrm{TV}(\pi_{n},\gamma^{*}_{n})\leq\delta^{*}_{3}\sqrt{d^{2}/n}$$

(1.5)

with high probability. Here, $\delta^{*}_{3}$ is an explicit model dependent coefficient related to the third derivative of the population log likelihood. This quantity may certainly depend on $d$ in some cases, and the second setting is one such example, as discussed below.

Our first, GLM setting is very general, allowing for the exponential family $p(\cdot\mid\omega)$ to have multivariate parameter $\omega\in\mathbb{R}^{k}$, $k\geq 1$. Thus, the $X_{i}$ are $d\times k$ matrices. The dimension $k$ is itself also allowed to grow with $n$. In particular, taking $k=d$ and $X_{i}=I_{d}$ for all $i=1,\dots,n$, we recover the setting of i.i.d. observations of a $d$-parameter exponential family $p$. Our BvM in the GLM setting can be compared to the works [16, 5] on the BvM for exponential families and [32] on the BvM for univariate ($k=1$) GLMs. To make this comparison, some work is required to first recover explicit TV bounds from the proofs of [16, 5]. With some effort, the recovered bound in these two works, as well as the stated TV bound of [32], can be brought into the structure discussed above (see Appendix C.1). In the end, we see that our BvM tightens the dimension dependence of the universal factor from $\sqrt{d^{3}/n}$ to $\sqrt{d^{2}/n}$, while our model-dependent factor $\delta^{*}_{3}$ is of the same order of magnitude as that of prior works.

To apply our general result to a particular GLM of interest, it remains to bound the quantity $\delta^{*}_{3}$. We do so in two particular cases: an i.i.d. log-concave exponential family, and a logistic regression model with Gaussian design. In both cases we show that $\delta^{*}_{3}$ is bounded by an absolute constant, where in the second case, this holds with high probability over the random design. Thus $d^{2}\ll n$ suffices for the BvM to hold in these settings, with no extra dimension dependence stemming from the model. Moreover, our treatment of the random design case seems to be new; we are not aware of any works proving the BvM for a random design GLM.

In our second setting of inference of a pmf, the model-dependent factor is $\delta^{*}_{3}=1/\sqrt{{\theta_{\mathrm{min}}^{*}}}$, where ${\theta_{\mathrm{min}}^{*}}$ is as above, i.e. the minimum probability in the ground truth pmf. Thus $\delta^{*}_{3}\geq\sqrt{d}$ in this setting. Our BvM directly improves on the result of [8]: from $\mathrm{TV}(\pi_{n},\gamma^{*}_{n})=\mathcal{O}(\sqrt{d^{3}/n{\theta_{\mathrm{min}}^{*}}})$ to $\mathrm{TV}(\pi_{n},\gamma^{*}_{n})=\mathcal{O}(\sqrt{d^{2}/n{\theta_{\mathrm{min}}^{*}}})$.

The GLM setting differs from the pmf setting in that the former enjoys a stochastic linearity property, to use the terminology of [33], while the latter does not. Stochastic linearity means that the sample log likelihood deviates from the population log likelihood by a linear function of $\theta$, and this property simplifies certain elements of the BvM proof. Despite this difference between the two settings, we use the same overall structure for our two BvM proofs.

The key element of this common proof structure is a bound on the TV accuracy of the Laplace approximation (LA) to $\pi_{n}$. The LA is another Gaussian approximation $\gamma_{n}$ to $\pi_{n}$; for more details, see Section 1.3 below. We derive a new elegant and simple bound on the LA accuracy $\mathrm{TV}(\pi_{n},\gamma_{n})$ which is tight in its dimension dependence (tightness is known thanks to lower bounds on $\mathrm{TV}(\pi_{n},\gamma_{n})$ derived in [20]). This bound is at the heart of our BvM proof, and it may also be of independent interest due to its simplicity.

The main tool in our proof is a new bound on the accuracy of the Laplace approximation (LA). The LA is another Gaussian distribution similar in spirit to $\gamma^{*}_{n}$, but which can be computed in practice from given data. Namely, the LA to $\pi_{n}$ is given by

$$\gamma_{n}:=\mathcal{N}(\tilde{\theta}_{n},\,-\nabla^{2}(\log\pi_{n})(\tilde{\theta}_{n})^{-1}),\qquad\tilde{\theta}_{n}:=\operatorname*{argmax}_{\theta\in\mathbb{R}^{d}}\pi_{n}(\theta).$$

Bayesian inference can be very costly in high dimensions, since typical quantities of interest such as the posterior mean, covariance, and credible sets involve taking integrals against or sampling from $\pi_{n}$. Making the approximation $\pi_{n}\approx\gamma_{n}$ can simplify these computations. Here, we prove a new bound on the TV accuracy of the LA, which essentially takes the form

$$\mathrm{TV}(\pi_{n},\gamma_{n})\leq\delta_{3}\sqrt{d^{2}/n}.$$

(1.6)

See Theorem 2.2 for the precise statement. Here, $\delta_{3}$ depends on the third derivative of the function $\log\pi_{n}$ in a neighborhood of $\tilde{\theta}_{n}$; it is intimately related to the “model-dependent” factor $\delta^{*}_{3}$ in our bound (1.5) in the statement of the BvM.

The bound (1.6) on $\mathrm{TV}(\pi_{n},\gamma_{n})$ is the key ingredient in our BvM proof. Namely, we apply the triangle inequality $\mathrm{TV}(\pi_{n},\gamma^{*}_{n})\leq\mathrm{TV}(\pi_{n},\gamma_{n})+\mathrm{TV}(\gamma_{n},\gamma^{*}_{n})$, and use the bound (1.6) on the first term on the righthand side. Since $\delta_{3}$ is random (depending on the log posterior, which in turn depends on the data $Y_{1:n}$), it remains to prove a high-probability, deterministic upper bound on $\delta_{3}$. The second term $\mathrm{TV}(\gamma_{n},\gamma^{*}_{n})$ is a TV distance between two Gaussians, and can be bounded in a straightforward manner. There is an additional first step in which we quantify the effect of removing the prior from $\pi_{n}$, but we neglect this technicality in the present overview to simplify the discussion.

The LA bound (1.6) has two important features which are noteworthy on their own and which have bearing on the BvM result. First, we see that the “universal” factor in the upper bound is $\sqrt{d^{2}/n}$. This directly translates to the tightened dimension dependence in the BvM. Second, the bound is remarkably simple and transparent, which allows for a very clear-cut proof of the BvM.

Let us put our bound (1.6) into context. Like prior work on the BvM, most prior work on the LA accuracy in high dimensions also required $d^{3}\ll n$. In particular, the three works [11, 34, 18] all show, under slightly varying conditions, a bound of the form $\mathrm{TV}(\pi_{n},\gamma_{n})\leq\delta_{3}^{\prime}\sqrt{d^{3}/n}$. Here, $\delta_{3}^{\prime}$ are various model-dependent coefficients analogous to $\delta_{3}$ from (1.6). Recently, however, the works [19, 20] obtained bounds on the TV distance with “universal” factor $d/\sqrt{n}$. The bound we derive here is significantly simpler than that of [19, 20]; see Section 2.2 for a more detailed comparison.

We note that in certain special cases, the limit of posterior distributions can be obtained in regimes in which $d^{2}/n$ is not small, including infinite-dimensional regimes. However, the type of limit, and the shape of the limiting posterior distribution, may be different.

The work [7] proves the BvM for Gaussian linear regression in increasing dimensions. This is a case in which the log likelihood is quadratic, so that the only non-Gaussian contribution to the posterior stems from the prior. The author shows that under certain conditions on the prior, $d\ll n$ suffices for the BvM to hold. This weaker requirement on $n$ (compared to our $n\gg d^{2}$) is consistent with our findings. Indeed, our analysis shows that the main source of error in the BvM vanishes when the log likelihood is quadratic; see Remark 3.6.

The work [10] proves a BvM for the coefficient vector in a sparse linear regression model, in which $d$ may be larger than $n$ but the true coefficient vector has few nonzero entries (exactly how many nonzero entries are allowed depends on various problem parameters). The authors show that the posterior is asymptotically a mixture of degenerate normal distributions (owing in part to a special prior construction incorporating sparsity). In particular, the posterior marginal of each coefficient is a mixture of a point mass at zero and a continuous distribution. This has implications for the construction of credible sets.

In infinite dimensions, [13] gives an example in which the posterior distribution of a functional of an infinite-dimensional parameter converges in distribution to a Gaussian, but which gives rise to credible intervals differing from frequentist confidence intervals. The work [9] proves a BvM for the Gaussian white noise model, in the sense that the authors show the posterior converges to the “correct” (from the frequentist perspective) normal distribution in a weaker sense than total variation norm. The notion of convergence is still strong enough to prove linear functionals of the parameter are asymptotically normal, centered at efficient estimators and with asymptotically efficient variance. Moreover, certain weighted $L^{2}$-credible ellipsoids coincide asymptotically with their frequentist counterparts.

Since the BvM is closely related to a high probability bound on the LA error, we mention a few works which prove the latter. This is in contrast to the Laplace bounds cited above, which are for a fixed realization of the data. In [3], the authors prove that for sparse generalized linear models with $n$ samples and $q$ active covariates, the relative LA error of the posterior normalizing constant is bounded by $(q^{3}\log^{3}n/n)^{1/2}$ with high probability. In [37], the authors study the pointwise ratio between the density of the posterior $\pi_{n}$ and that of the LA $\gamma_{n}$. The authors show that for generalized linear models with Gaussian design, the relative error is bounded by $d^{3}\log n/n$ with probability tending to 1, provided $d^{3+\epsilon}\ll n$ for some $\epsilon>0$. For logistic regression with Gaussian design, they obtain the improved rate $d^{2}\log n/n$ provided $d^{2.5}\ll n$. Finally, [11] shows $\mathrm{KL}\left(\,\pi_{n}\;||\;\gamma_{n}\right)$ goes to 0 with probability tending to 1 if $d^{3}/n\to 0$.

Finally, we note that the LA is just one approach to approximate a posterior measure by a simpler distribution. Another approach that is similar in spirit is to approximate the posterior $\pi_{n}$ with a log concave distribution [27, 6]. Roughly speaking, the log concave approximation coincides with $\pi_{n}$ in a neighborhood of the mode $\tilde{\theta}_{n}$ (where the negative log posterior is indeed convex), and it “convexifies” the tails of $-\log\pi_{n}$. The two works [27, 6] prove bounds on the accuracy of such log concave approximations in the context of high-dimensional Bayesian nonlinear inverse problems. Under suitable restrictions on the growth of $d$ relative to $n$, it is shown that the Wasserstein-2 distance between the two measures is exponentially small in $n$ with high probability under the ground truth data generating process.

The rest of the paper is organized as follows. In Section 2 we state, discuss, and outline the proof of our new Laplace approximation bound in a general setting, where $\pi_{v}\propto e^{-nv}$ need not have the structure of a posterior. In Section 3 we introduce the Bayesian setting and use our Laplace bound to prove some key preliminary results for the BvM proofs. In Sections 4 and 5 we prove the BvM for posteriors stemming from GLMs and observations of a discrete probability distribution, respectively. Omitted proofs can be found in the Appendix.

The letter $C$ always denotes an absolute constant. The inequality $a\lesssim b$ denotes $a\leq Cb$ for an absolute constant $C$. The statement that $a\lesssim b$ on a random event indicates that there is an absolute nonrandom constant $C$ such that $a\leq Cb$ on the event. We let $|\cdot|$ denote the standard Euclidean norm in $\mathbb{R}^{d}$. If $A\in\mathbb{R}^{d\times d}$ is a symmetric positive definite matrix and $u\in\mathbb{R}^{d}$ is a vector, we let

$$|u|_{A}=|A^{1/2}u|.$$

Let $k\geq 1$ and $S$ be a symmetric $k$-linear form, identified with its symmetric tensor representation. We define the $A$-weighted operator norm of $S$ to be

$$\|S\|_{A}:=\sup_{|u|_{A}=1}\langle S,u^{\otimes k}\rangle=\sup_{|u|_{A}=1}\sum_{i_{1},\dots,i_{k}=1}^{d}S_{i_{1}i_{2}\dots i_{k}}u_{i_{1}}u_{i_{2}}\dots u_{i_{k}}.$$

(1.7)

For example, if $k=2$ then $S$ is a symmetric matrix and $\|S\|_{A}=\|A^{-1/2}SA^{-1/2}\|$, where $\|\cdot\|$ is the standard matrix operator norm. Note that if $k=1$ and $S$ is a 1-linear form identified with its vector representation, then

$$\|S\|_{A}=|A^{-1/2}S|,\qquad|S|_{A}=|A^{1/2}S|.$$

Thus it is important to distinguish between the meaning of $\|\cdot\|_{A}$ and $|\cdot|_{A}$ when the quantity inside the norm is a vector. By Theorem 2.1 of [39], for symmetric tensors the definition (1.7) coincides with the standard definition of operator norm:

$$\sup_{|u_{1}|_{A}=\dots=|u_{k}|_{A}=1}\langle S,u_{1}\otimes\dots\otimes u_{k}\rangle=\|S\|_{A}=\sup_{|u|_{A}=1}\langle S,u^{\otimes k}\rangle.$$

The assumptions of Theorem 2.2 imply two key properties used in the proof: first, $\pi_{f}$ is strongly log concave in the neighborhood ${\mathcal{U}}(r)$ of $\hat{\theta}$. Second, $f$ grows linearly away from $\hat{\theta}$ in ${\mathcal{U}}(r)^{c}$. These two observations are stated in the following lemma:

See Appendix A for the proof of the lemma. The bound (2.5) does not require convexity. Similarly, (2.6) only uses convexity via the property of convex functions that the infimum of $(f(\theta)-f(\hat{\theta}))/|\theta-\hat{\theta}|_{H}$ over $\theta\in{\mathcal{U}}(r)^{c}$ is achieved on the boundary $\{|\theta-\hat{\theta}|_{H}=r\sqrt{d/n}\}$. Thus any other function with this property will satisfy the lemma. Note also that (2.6) proves $e^{-nf}\in L_{1}(\Theta)$.

Next, the following lemma presents the main tools in the proof of Theorem 2.2.

See Appendix A.1 for the proof of this lemma. The bound (2.8) stems from a log-Sobolev inequality (LSI) [2], which can be applied since $\mu$ is strongly log-concave in the neighborhood ${\mathcal{U}}$. The inspiration for this technique comes from [19], who also prove a bound on $\mathrm{TV}(\pi_{f},\gamma_{f})$ in the context of Bayesian inference. See Section 2.2 for a comparison between our bound and that of [19].

We apply (2.9) with ${\mathcal{U}}={\mathcal{U}}(r)$, $\mu=\pi_{f}$ (extended to be zero in $\mathbb{R}^{d}\setminus\Theta$) and $\hat{\mu}=\gamma_{f}\propto e^{-n\hat{f}}$, where $\hat{f}(\theta)=\frac{1}{2}|\theta-\hat{\theta}|^{2}_{H}$. Since $\nabla^{2}f\succeq\frac{1}{2}H$ on ${\mathcal{U}}(r)$ by (2.5), the function $f$ enjoys local strong convexity. In particular, the lower bound condition in the lemma is satisfied with $\lambda=1/2$. Therefore, in our setting, the bound (2.9) takes the form

$$\mathrm{TV}(\pi_{f},\gamma_{f})\leq\pi_{f}({\mathcal{U}}^{c})+\gamma_{f}({\mathcal{U}}^{c})+\sqrt{\frac{n}{2}}\mathbb{E}\,_{\theta\sim\gamma_{f}|{\mathcal{U}}}\left[\|\nabla f(\theta)-H(\theta-\hat{\theta})\|^{2}_{H}\right]^{\frac{1}{2}}.$$

(2.10)

We see that to bound the TV distance between $\pi_{f}$ and $\gamma_{f}$, it suffices to bound the two tail integrals and the local expectation in (2.10). The tail integral $\pi_{f}({\mathcal{U}}^{c})=\int_{{\mathcal{U}}^{c}}e^{-nf}/\int e^{-nf}$ can be bounded using the linear growth of $f$ in ${\mathcal{U}}^{c}$. The bound on $\gamma_{f}({\mathcal{U}}^{c})$ follows by standard Gaussian tail inequalities. Finally, to bound the local expectation we use a Taylor expansion of $\nabla f(\theta)$ about $\hat{\theta}$.to get

$$\nabla f(\theta)-H(\theta-\hat{\theta})=\nabla f(\theta)-\nabla f(\hat{\theta})-\nabla^{2}f(\hat{\theta})(\theta-\hat{\theta})=(\nabla^{2}f(\xi)-\nabla^{2}f(\hat{\theta}))(\theta-\hat{\theta})$$

for a point $\xi$ between $\theta$ and $\hat{\theta}$. From here we use the definition of $\delta_{3}$ and simple Gaussian calculations to show that the third term in (2.10) is bounded as $\delta_{3}(r)d/\sqrt{n}$.See Appendix A.2 for these calculations, which finish the proof of Proposition 2.2.

So far, only two other works have obtained bounds on $\mathrm{TV}(\pi_{f},\gamma_{f})$ scaling as $d/\sqrt{n}$ — that of [19] and [20]. We compare (2.2) to these two bounds from the perspective of the BvM.

The bound of [19] requires $f\in C^{3}(\Theta)$. Here, we require that $f\in C^{2}(\Theta)$ and that $\delta_{3}(r)$ defined in (2.1) is finite. The bound of [19] is also significantly more complex than ours; see Theorem 3.1 in [19] (which relies on the quantities defined in Section 2) compared to our (2.2). The bound of [20] involves fourth derivatives of $f$. Thus greater regularity of $f$ is required, and the presence of fourth derivatives makes the bound more complex than (2.2).

The weaker regularity condition required in our bound (2.2) translates into weaker regularity of the log likelihood in the statement of the BvM. As can be seen from the definition of event $E_{3}$ in (3.4) and the third condition in (3.5), only boundedness of the Lipschitz constant of the log likelihood’s second derivative is needed. Also, as already mentioned, the simplicity of (2.2) is useful to obtain a streamlined proof and statement of the BvM.

Finally, let us compare our proof technique to that of [19] and [20]. The key idea behind our proof is the same as in [19], that is, to use the log-Sobolev inequality in a local region. However, the implementation is different. Here, we (1) do not split up the function $f$ into a sum of two parts coming from the log likelihood and the log prior, (2) apply the log-Sobolev inequality in an affine invariant way, and (3) handle the tail integrals differently, relying on the linear growth of $f$ at infinity rather than the fact that the prior is a proper Lebesgue density integrating to 1.

The proof technique in [20] is quite different. In this work, the leading order term in the TV distance between $\pi_{f}$ and $\gamma_{f}$ is derived, and the main challenge is to bound the difference between $\mathrm{TV}(\pi_{f},\gamma_{f})$ and this leading term. This is done using high-dimensional Gaussian concentration. Overall, the calculation in [20] is more delicate, but leads to a more complicated bound involving fourth derivatives.

Let $\{P^{n}_{\theta}\;:\;\theta\in\Theta\}$ be a parameterized family of probability distributions, with $\Theta$ an open convex subset of $\mathbb{R}^{d}$. We assume there is a fixed measure with respect to which $P^{n}_{\theta}$ has density $p^{n}(\cdot\mid\theta)$, for each $\theta\in\Theta$. We observe a sample $Y^{n}\sim P^{n}_{\theta^{*}}$ for a ground truth parameter ${\theta^{*}}\in\Theta$. We define the likelihood $L:\Theta\to(0,\infty)$ by $L(\theta)=p^{n}(Y^{n}\mid\theta)$, and the negative normalized log likelihood $\ell:\Theta\mapsto\mathbb{R}$ by

$$\ell(\theta)=-\frac{1}{n}\log L(\theta)=-\frac{1}{n}\log p^{n}(Y^{n}\mid\theta).$$

The normalization by $1/n$ is natural in the standard case that $P^{n}_{\theta}$ is an $n$-fold product measure. In other words, $Y^{n}=(Y_{1},\dots,Y_{n})$ and $p^{n}(Y^{n}\mid\theta)=\prod_{i=1}^{n}p_{i}(Y_{i}\mid\theta)$. We then have that $\log L$ is a sum of $n$ terms. When it exists and is unique, we define

$$\hat{\theta}=\operatorname*{argmin}_{\theta\in\Theta}\ell(\theta),$$

the maximum likelihood estimator (MLE). We define the negative population log likelihood to be the function

$$\ell^{*}(\theta)=\mathbb{E}\,_{Y^{n}\sim P^{n}_{\theta^{*}}}[\ell(\theta)]=-\frac{1}{n}\mathbb{E}\,_{Y^{n}\sim P^{n}_{\theta^{*}}}\left[\log p^{n}(Y^{n}\mid\theta)\right].$$

Let the prior probability density over $\theta$ be denoted $\pi_{0}$. We assume without loss of generality that $\Theta$ lies in the support of $\pi_{0}$; otherwise, redefine $\Theta$ to be its intersection with the support of $\pi_{0}$. Thus there is a function $v_{0}:\Theta\to\mathbb{R}$ such that

$$\pi_{0}(\theta)\propto e^{-v_{0}(\theta)},\quad\theta\in\Theta.$$

Finally, the posterior distribution is the probability density $\pi_{v}$ such that

$$\pi_{v}(\theta)\propto e^{-n\ell(\theta)}\pi_{0}(\theta)\propto e^{-nv(\theta)},\qquad v=\ell+n^{-1}v_{0},\quad\theta\in\Theta.$$

Classically, the BvM states that $\mathrm{TV}(\pi_{v},\gamma^{*})=o(1)$ with probability tending to 1 (under $P^{n}_{\theta^{*}}$) as $n\to\infty$ [38]. Here,

$$\gamma^{*}=\mathcal{N}(\hat{\theta},(n{F^{*}})^{-1}).$$

(3.1)

From the definition of $\gamma^{*}$, we see that a necessary ingredient to prove the BvM is showing the MLE exists and is unique with probability tending to 1.

Here, we will be interested in a nonasymptotic version of the BvM. Namely, we will prove a result of the form: “under certain conditions on $d$, $n$, and the statistical model, $\mathrm{TV}(\pi_{v},\gamma^{*})\leq\epsilon$ with probability at least $1-p$”, for explicit quantities $\epsilon$ and $p$ depending on $d$, $n$, and the model. We are only aware of one other work which proves BvMs in this style [32]. Our nonasymptotic BvM results then easily lead to the more classical asymptotic statements.

The proof idea is as follows. First, we show that discarding the prior has negligible effect by bounding $\mathrm{TV}(\pi_{v},\pi_{\ell})$. Here, $\pi_{\ell}\propto e^{-n\ell}$. Second, we bound $\mathrm{TV}(\pi_{\ell},\gamma_{\ell})$ by directly applying Theorem 2.2 with $f=\ell$. Here,

$$\gamma_{\ell}=\mathcal{N}(\hat{\theta},(n\nabla^{2}\ell(\hat{\theta}))^{-1})$$

(3.2)

is the LA to $\pi_{\ell}$. Third, we bound $\mathrm{TV}(\gamma_{\ell},\gamma^{*})$ by comparing the inverse covariance matrices $n\nabla^{2}\ell(\hat{\theta})$ and $n\nabla^{2}\ell^{*}({\theta^{*}})$ of the two Gaussian distributions (which have the same mean). We then use the triangle inequality to bound $\mathrm{TV}(\pi_{v},\gamma^{*})$ via the sum of the three bounds. To be more precise, we show in this section that these three bounds hold on the event $E(s,\epsilon_{2})$ from Definition 3.1 below. To finish the BvM proof, it remains to prove this is a high-probability event. We will carry out this final probabilistic step using the particulars of each of our models in the following two sections.

We now introduce several quantities and specify the event $E(s,\epsilon_{2})$.

Here, “$\exists!\,\mathrm{MLE}\,\hat{\theta}$” is an abbreviation for “the MLE exists and is unique”. Also, recall from Section 1.5 that $\|\nabla\ell({\theta^{*}})\|_{F^{*}}=|{F^{*}}^{-1/2}\nabla\ell({\theta^{*}})|$.

We will show that $E(s,\epsilon_{2})\subset\bar{E}(s,\epsilon_{2})$, so that in particular, a unique MLE $\hat{\theta}$ is guaranteed to exist on $E(s,\epsilon_{2})$ and $\hat{\theta}$ is close to ${\theta^{*}}$. We then state our key bounds on the event $\bar{E}(s,\epsilon_{2})$.