PAC-Bayesian Generalization Bounds for
MultiLayer Perceptrons

PAC-Bayesian bounds on the generalization risk (Germain et al.[10]).

Given a distribution $\mathcal{D}$ over $\mathcal{X}\times\mathcal{Y}$, a hypothesis set $\mathcal{H}$, a loss function $\ell^{\prime}:\mathcal{F}\times\mathcal{X}\times\mathcal{Y}\rightarrow[0,1]$, a prior distribution $\pi$ over $\mathcal{H}$, a real number $\delta\in(0,1]$, and a real number $\beta>0$, with probability at least $1-\delta$ over the samples $\mathcal{S}\sim\mathcal{D}^{n}$, we have

$$\forall\rho\text{ on }\mathcal{H}:\underset{{h\sim\rho}}{\boldsymbol{E}}\mathcal{L}^{\ell^{\prime}}_{\mathcal{D}}(h)\leq\frac{1}{1-e^{-\beta}}[1-e^{-\beta{\boldsymbol{E}_{h\sim\rho}}\hat{\mathcal{L}}^{\ell^{\prime}}_{\mathcal{S}}(h)-\frac{1}{n}(\text{KL}[\rho||\pi]+ln\frac{1}{\delta})}].$$

(2)

We can extend it to arbitraray bounded loss, i.e., $\ell:\mathcal{F}\times\mathcal{X}\times\mathcal{Y}\rightarrow[a,b]$, where $[a,b]\in\mathbb{R}$ through defining $\beta:=b-a$ and $\ell^{\prime}(h,x,y):=(\ell(h,x,y)-a)/(b-a)\in[0,1]$.

$$\forall\rho\text{ on }\mathcal{H}:\underset{{h\sim\rho}}{\boldsymbol{E}}\mathcal{L}^{\ell}_{\mathcal{D}}(h)\leq\frac{b-a}{1-e^{a-b}}\{1-e^{[-{\boldsymbol{E}_{h\sim\rho}}\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)-\frac{1}{n}(\text{KL}[\rho||\pi]+ln\frac{1}{\delta})+a]}\}.$$

(3)

Formula 3 indicates that minimizing the generalization bound is equivalent to minimizing

$$n{\boldsymbol{E}_{h\sim\rho}}\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)+\text{KL}[\rho||\pi].$$

(4)

To facilitate subsequent discussions, we define $X$, $Y$, and $F_{Y}$ as the random variables of the sample $x_{i}\in\mathcal{X}$, the label $y_{i}\in\mathcal{Y}$, and the hypothesis $h(x_{i})\in\mathcal{Y}$, respectively. Therefore, $\pi:=P(F_{Y}|X)$ and $\rho:=P(F_{Y}|X,Y)$ denote the prior distribution and the posterior distribution of $h$, respectively. Based on Bayes’ rule, we can formulate the connection between $P(F_{Y}|X)$ and $P(F|X,Y)$ as

$$P(F_{Y}|X,Y)=\frac{P(F_{Y}|X)P(Y|F_{Y},X)}{P(Y|X)},$$

(5)

where $P(Y=y_{i}|F_{Y},X=x_{i})$ is the likelihood of $y_{i}\in\mathcal{Y}$ given $h$ and $x_{i}\in\mathcal{X}$, and the evidence can be formulated as $P(Y|X)=\sum_{l\in\mathcal{Y}}P(F_{Y}=l|X)P(Y|F_{Y}=l,X)$²²2We only consider the discrete case, because we prove $F_{Y}$ as a discrete random variable in the next section..

In the context of supervised classification based on MLPs, $\mathcal{Y}=\{1,\cdots,L\}$ consists of finite $L$ labels, and $P_{F_{Y}|X}(l|x_{i})$³³3For simplicity, $P_{F_{Y}|X}(l|x_{i}):=P(F_{Y}=l|X=x_{i})$. denotes the prior probability of $x_{i}$ belonging to class $l$ based on the $h$ expressed by MLPs. We define $P^{*}(F_{Y}|X)$ as the truly prior distribution of the true $h^{*}$. Given $P^{*}(F_{Y}|X)$ and $P(Y|F_{Y},X)$, $P^{*}(F_{Y}|X,Y)$ is the truly posterior distribution with the one-hot format, i.e., $\forall l\in\mathcal{Y}$, $P^{*}_{F_{Y}|X,Y}(l|x_{i},y_{i})=1$ if $l=y_{i}$; otherwise $P^{*}_{F_{Y}|X,Y}(l|x_{i},y_{i})=0$. We aim to learn an optimal MLP from $\mathcal{S}$ such that it approaches $P^{*}(F_{Y}|X,Y)$ as close as possible. In this paper, we use the $\text{MLP}=\{{x;f_{1};f_{2};f_{Y}}\}$ (Figure 1) for most theoretical derivations unless otherwise specified.

the probability space $(\Omega,\mathcal{T},Q_{F})$ for a fully connected layer.

Given a fully connected layer ${f}$ with $T$ neurons $\{f_{t}=\sigma[g_{t}({x_{i}})]=\sigma(\sum_{m=1}^{M}\omega_{mt}\cdot x_{im}+b_{t})\}_{t=1}^{T}$, where $\sigma(\cdot)$ is an activation function, e.g, ReLU function, $x_{i}=\{x_{im}\}_{m=1}^{M}\in\mathcal{X}$ is the input of ${f}$, and $g_{t}(\cdot)$ is the $t$th linear filter with $\omega_{mt}$ being the weight and $b_{t}$ being the bias, let inputting ${x_{i}}$ into ${f}$ be an experiment, we define the probability space $(\Omega,\mathcal{T},Q_{F})$ as follows.

First, the sample space $\Omega$ consists of $T$ possible outcomes $\{\boldsymbol{\omega}_{t}\}_{t=1}^{T}$, where $\boldsymbol{\omega}_{t}=\{\omega_{mt}\}_{m=1}^{M}$. In terms of machine learning, a possible outcome $\boldsymbol{\omega}_{t}$ defines a potential feature of ${x_{i}}$. Since scalar values cannot describe the dependence within ${x_{i}}$, we do not take into account $b_{t}$ for defining $\Omega$. Second, we define the event space $\mathcal{T}$ as the $\sigma$-algebra. For example, if ${f}$ has $T=2$ linear filters and $\Omega=\{\boldsymbol{\omega}_{1},\boldsymbol{\omega}_{2}\}$, thus ${\textstyle\mathcal{T}=\{\emptyset,\{\boldsymbol{\omega}_{1}\},\{\boldsymbol{\omega}_{2}\},\{\{\boldsymbol{\omega}_{1},\boldsymbol{\omega}_{2}\}\}}$ indicates that none of outcomes, one of outcomes, or both outcomes could happen in the experiment. Third, we prove the probability measure $Q_{F}$ as a Gibbs distribution to quantify the probability of possible outcomes $\{\boldsymbol{\omega}_{t}\}_{t=1}^{T}$ occurring in ${x_{i}}$.

$$Q_{F}(\boldsymbol{\omega}_{t})=\frac{1}{Z_{F}(x_{i})}\text{exp}(f_{t})=\frac{1}{Z_{F}(x_{i})}\text{exp}[\sigma(\langle\boldsymbol{\omega}_{t},{x_{i}}\rangle+b_{t})],$$

(6)

where $\langle\cdot,\cdot\rangle$ is the inner product, $Z_{F}(x_{i})=\sum_{t=1}^{T}\text{exp}(f_{t})$ is the partition function only depending on $x_{i}$, and the energy function $E_{\boldsymbol{\omega}_{t}}({x_{i}})$ equals the negative activation, i.e., $E_{\boldsymbol{\omega}_{t}}({x_{i}})=-f_{t}$. Notably, the higher activation $f_{t}$ indicates the lower energy $E_{\boldsymbol{\omega}_{t}}({x_{i}})$, thus $\boldsymbol{\omega}_{t}$ has the higher probability of occurring in ${x_{i}}$. The detailed proofs and supportive simulations are presented in Appendix B⁴⁴4We recommend reading the supplementary file connecting the paper and the appendix together.

Based on $(\Omega,\mathcal{T},Q_{F})$ for the fully connected layer ${f}$, we can explicitly define the corresponding random variable $F:\Omega\rightarrow E$. Specifically, since $\Omega=\{\boldsymbol{\omega}_{t}\}_{t=1}^{T}$ includes finite $T$ possible outcomes, $E$ should be a discrete measurable space and $F$ is a discrete random variable. For example, $Q(F=t)$ indicates the probability of the $t$th outcome $\boldsymbol{\omega}_{t}$ occurring in ${x_{i}}$. Furthermore, the probability space of a fully connected layer enables us to derive probabilistic explanations for MLPs.

the $\text{MLP}=\{{x;f_{1};f_{2};f_{Y}}\}$ corresponds to a family of Gibbs distributions

$$Q_{F_{Y}|X}(l|x_{i})=\frac{1}{Z_{\text{MLP}}(x_{i})}\text{exp}[-E_{yl}(x_{i})]=\frac{1}{Z_{\text{MLP}}(x_{i})}\text{exp}[{f_{yl}(f_{2}(f_{1}({x_{i}})))}],$$

(7)

where $E_{yl}(x_{i})=-{f_{yl}(f_{2}(f_{1}(x_{i})))}$ is the energy function of $l\in\mathcal{Y}$ given $x_{i}$, and the partition function $Z_{\text{MLP}}(x_{i})=\sum_{l=1}^{L}\sum_{k=1}^{K}\sum_{t=1}^{T}Q(F_{Y},F_{2},F_{1}|X=x_{i})$.

Proof: Definition 1 implies $\omega_{mt}$ defining $\Omega$, thus $\Omega$ would be fixed if not considering training. Therefore, ${F_{i+1}}$ is entirely determined by ${F_{i}}$ in the MLP, where $F_{i}$ denotes the random variable of $f_{i}$, and the MLP forms a Markov chain ${X}\boldsymbol{\leftrightarrow}{F_{1}}\boldsymbol{\leftrightarrow}{F_{2}}{\leftrightarrow}{F_{Y}}$. As a result, given a sample $x=x_{i}$, the conditional distribution $Q(F_{Y}|X)$ corresponding to the MLP can be derived as

$$Q_{F_{Y}|X}(l|x_{i})=\sum_{k=1}^{K}\sum_{t=1}^{T}Q_{F_{Y},F_{2},F_{1}|X}(l,k,t|x_{i})=\frac{1}{Z_{\text{MLP}}(x_{i})}\text{exp}[{f_{yl}(f_{2}(f_{1}({x_{i}})))}].$$

(8)

The derivation is presented in Appendix C. The energy function $E_{yl}(x_{i})=-{f_{yl}(f_{2}(f_{1}(x_{i})))}$ implies that $Q_{F_{Y}|X}(l|x_{i})$ is entirely determined by the architecture of the MLP. In other words, the MLP formulates a family of Gibbs distributions $Q(F_{Y}|{X})$.

Given the samples $\mathcal{S}$ and the $\text{MLP}=\{{x;f_{1};f_{2};f_{Y}}\}$, if $\ell$ is the cross entropy, then minimizing $\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)$ is equivalent to maximizing $Q_{\mathcal{S}}(F_{Y}|X)$.

$$\underset{h\in\mathcal{H}}{\text{min}}\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)\Rightarrow\underset{Q(F_{Y}|X)}{\text{max}}\frac{1}{n}\text{log}Q_{\mathcal{S}}(F_{Y}|X),$$

(9)

where $Q_{\mathcal{S}}(F_{Y}|X)=\prod_{i=1}^{n}Q_{F_{Y}|X}(y_{i}|x_{i})$.

Proof: since the cross entropy $H[P^{*}(F_{Y}|X,Y),Q(F_{Y}|X)]=\underset{P^{*}(F_{Y}|X,Y)}{-\boldsymbol{E}}[\text{log}Q(F_{Y}|X)]$, we have

$$\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)=-\frac{1}{n}\sum_{i=1}^{n}H[P^{*}(F_{Y}|X,Y),Q(F_{Y}|X)]=-\frac{1}{n}\sum_{i=1}^{n}\sum_{l=1}^{L}P^{*}_{F_{Y}|X,Y}(l|x_{i},y_{i})\text{log}Q_{F_{Y}|X}(l|x_{i}).$$

(10)

Since $P^{*}(F_{Y}|X,Y)$ is one-hot, namely that if $l=y_{i}$, then $P^{*}_{F_{Y}|X,Y}(l|x_{i},y_{i})=1$, otherwise $P^{*}_{F_{Y}|X,Y}(l|x_{i},y_{i})=0$, we can simplify Equation (10) as

$$\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)=-\frac{1}{n}\sum_{i=1}^{n}\text{log}Q_{F_{Y}|X}(y_{i}|x_{i})=-\frac{1}{n}\text{log}Q_{\mathcal{S}}(F_{Y}|X).$$

(11)

Therefore, minimizing $\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)$ is equivalent to maximizing the likelihood $Q_{\mathcal{S}}(F_{Y}|X)$. In addition, based on Theorem 2 (Equation 7), we can extend $\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)$ as

$$n\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)=-\sum_{i=1}^{n}\text{log}Q_{F_{Y}|X}(y_{i}|x_{i})=\sum_{i=1}^{n}[E_{yy_{i}}(x_{i})+\text{log}Z_{\text{MLP}}(x_{i})].$$

(12)

Recall that $Z_{\text{MLP}}(x_{i})=\sum_{l=1}^{L}\sum_{k=1}^{K}\sum_{t=1}^{T}Q(F_{Y},F_{2},F_{1}|X=x_{i})$ sums up all the cases of $F_{Y}$, thus $\text{log}Z_{\text{MLP}}(x_{i})$ can be viewed as a constant with respect to $Q(F_{Y}|X)$.

given the samples $\mathcal{S}$ and the $\text{MLP}=\{{x;f_{1};f_{2};f_{Y}}\}$, if $\ell$ is the cross entropy, then minimizing $\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)$ can be explained as Bayesian variational inference.

$$\underset{h\in\mathcal{H}}{\text{min}}\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)\Rightarrow\underset{Q(F_{Y}|X)}{\text{min}}\text{KL}[Q_{\mathcal{S}_{x}}(F_{Y}|X)||P^{*}_{\mathcal{S}}(F_{Y}|X,Y)],$$

(13)

where $Q_{\mathcal{S}_{x}}(F_{Y}|X)=\prod_{i=1}^{n}Q(F_{Y}|X=x_{i})$ and $P^{*}_{\mathcal{S}}(F_{Y}|X,Y)=\prod_{i=1}^{n}P^{*}(F_{Y}|X=x_{i},Y=y_{i})$.

Proof: based on the property $H(P,Q)=H(P)+\text{KL}[P||Q]$, we can derive $\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)$ as

$$\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)=\frac{1}{n}\sum_{i=1}^{n}H[{P^{*}(F_{Y}|X=x_{i},Y=y_{i})}]+\text{KL}[{P^{*}(F_{Y}|X=x_{i},Y=y_{i})||Q(F_{Y}|X=x_{i})}].$$

(14)

Since $P^{*}_{F_{Y}|X,Y}(l|x_{i},y_{i})$ is one-hot, namely that if $l=y_{i}$, $P^{*}_{F_{Y}|X,Y}(l|x_{i},y_{i})=1$, otherwise $P^{*}_{F_{Y}|X,Y}(l|x_{i},y_{i})=0$, $H[P^{*}(F_{Y}|X=x_{i},Y=y_{i})]=0$, thus we can simplify $\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)$ as

$$\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)=\frac{1}{n}\sum_{i=1}^{n}\text{KL}[P^{*}(F_{Y}|X=x_{i},Y=y_{i})||Q(F_{Y}|X=x_{i})].$$

(15)

To keep consistent with the definition of variational inference, we reformulate $\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)$ as

$$\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)=\frac{1}{n}\sum_{i=1}^{n}\{\text{KL}[Q(F_{Y}|X=x_{i})||P^{*}(F_{Y}|X=x_{i},Y=y_{i})]+d\}.$$

(16)

where $d=\text{KL}[P^{*}(F_{Y}|X,Y)||Q(F_{Y}|X)]-\text{KL}[Q(F_{Y}|X)||P^{*}(F_{Y}|X,Y)]$.

Though KL-divergence is asymmetric, i.e., $\text{KL}[P||Q]\neq\text{KL}[Q||P]$, we show that $d$ converges to zero during minimizing $\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)$, because $P^{*}(F_{Y}|X,Y)$ is fixed and $Q(F_{Y}|X)$ is the only variable. The proof for $d\rightarrow 0$ is presented in Appendix D. Moreover, since $\mathcal{S}$ consists of i.i.d. samples and KL-divergence is additive for independent distributions, we can finally derive

$$\underset{h\in\mathcal{H}}{\text{min}}n\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)=\underset{Q(F_{Y}|X)}{\text{min}}\text{KL}[Q_{\mathcal{S}_{x}}(F_{Y}|X)||P^{*}_{\mathcal{S}}(F_{Y}|X,Y)].$$

(17)

As a result, minimizing $n\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)$ with the cross entropy loss is equivalent to Bayesian variational inference. In summary, Theorem 3 provides a novel probabilistic explanation for MLPs from the perspective of Bayesian variational inference. Specifically, the MLP formulates a family of Gibbs distribution $Q(F_{Y}|X)$, and minimizing $\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)$ aims to search an optimal distribution $Q(F_{Y}|X)$, which is closest to the truly posterior distribution $P^{*}(F_{Y}|X,Y)$ given $\mathcal{S}$.

given the samples $\mathcal{S}$ and the $\text{MLP}=\{{x;f_{1};f_{2};f_{Y}}\}$, if $\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)$ is the cross entropy, then minimizing $\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)$ inherently guarantees PAC-Bayesian generalization bounds for the MLP.

$$\underset{h\in\mathcal{H}}{\text{min}}\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)\Rightarrow\underbrace{\underset{Q(F|X)}{\text{min}}\text{KL}[Q_{\mathcal{S}_{x}}(F_{Y}|X)||P^{*}_{\mathcal{S}}(F_{Y}|X,Y)]}_{\text{Bayesian variational inference}}\Rightarrow\underbrace{\underset{\rho}{\text{min }}n\underset{h\sim\rho}{\boldsymbol{E}}\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)+\text{KL}[\rho||\pi]}_{\text{PAC-Bayesian optimization}}.$$

(18)

Proof: we only need prove Bayesian variational inference inducing PAC-Bayesian generalization bounds based on Theorem 3. Given the Bayesian variational inference (Equation 17), we have

$$\begin{split}\text{KL}[{\scriptstyle Q_{\mathcal{S}_{x}}(F_{Y}|X)}||{\scriptstyle P^{*}_{\mathcal{S}}(F_{Y}|X,Y)}]=&\underset{Q(F_{Y}|X)}{\boldsymbol{E}}\text{log}{\scriptstyle Q_{\mathcal{S}_{x}}(F_{Y}|X)}-\underset{Q(F_{Y}|X)}{\boldsymbol{E}}\text{log}{\scriptstyle P^{*}_{\mathcal{S}}(F_{Y}|X,Y)}\\ =&\underset{Q(F_{Y}|X)}{\boldsymbol{E}}\text{log}{\scriptstyle Q_{\mathcal{S}_{x}}(F_{Y}|X)}-\underset{Q(F_{Y}|X)}{\boldsymbol{E}}[\text{log}{\scriptstyle P^{*}_{\mathcal{S}}(F_{Y},Y|X)}-\text{log}{\scriptstyle P_{\mathcal{S}}(Y|X)}].\end{split}$$

(19)

Since $\text{log}P_{\mathcal{S}}(Y|X)$ is constant with respect to $Q(F_{Y}|X)$, we can derive the ELBO as

$$\text{log}P_{\mathcal{S}}(Y|X)=\underbrace{\underset{Q(F_{Y}|X)}{\boldsymbol{E}}\text{log}{\scriptstyle P_{\mathcal{S}}(Y|F_{Y},X)}-\text{KL}[{\scriptstyle Q_{\mathcal{S}_{x}}(F_{Y}|X)}||{\scriptstyle P^{*}_{\mathcal{S}_{x}}(F_{Y}|X)}]}_{\text{ELBO}}+\text{KL}[{\scriptstyle Q_{\mathcal{S}_{x}}(F_{Y}|X)}||{\scriptstyle P^{*}_{\mathcal{S}}(F_{Y}|X,Y)}],$$

(20)

where $P_{\mathcal{S}}(Y|X)=\prod_{i=1}^{n}P(Y=y_{i}|X=x_{i})$, $P_{\mathcal{S}}(Y|F_{Y},X)=\prod_{i=1}^{n}P(Y=y_{i}|F_{Y},X=x_{i})$, $Q_{\mathcal{S}_{x}}(F_{Y}|X)=\prod_{i=1}^{n}Q(F_{Y}|X=x_{i})$, and $P^{*}_{\mathcal{S}_{x}}(F_{Y}|X)=\prod_{i=1}^{n}P^{*}(F_{Y}|X=x_{i})$. The detailed derivation of the ELBO is presented in Appendix E.

Since $P^{*}(F_{Y}|X)$ is the truly prior distribution of the true hypothesis $h^{*}$, we have $\pi:=P^{*}_{\mathcal{S}_{x}}(F_{Y}|X)$. Since $Q(F_{Y}|X)$ infers the truly posterior distribution $P^{*}(F_{Y}|X,Y)$ of $h$, we have $\rho:=Q_{\mathcal{S}_{x}}(F_{Y}|X)$. Corollary 1 induces $n\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)=-\text{log}P_{\mathcal{S}}(Y|F_{Y},X)$ because we can express $P(Y=y_{i}|F_{Y},X=x_{i})$ as $Q(F_{Y}=y_{i}|X=x_{i})$. As a result, we derive

$$\text{ELBO}=-(n\underset{Q(F_{Y}|X)}{\boldsymbol{E}}{\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)}+\text{KL}[{Q_{\mathcal{S}_{x}}(F_{Y}|X)}||{P^{*}_{\mathcal{S}_{x}}(F_{Y}|X)}]).$$

(21)

It implies that maximizing ELBO is equivalent to minimizing PAC-Bayesian generalization bounds, thus minimizing $\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)$ inherently guarantees PAC-Bayesian generalization bounds for the MLP.

Theorem 4 theoretically explains why MLPs can achieve great generalization performance without explicit regularization. More specifically, the ELBO indicates that minimizing $\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)$ makes MLPs not only to learn the likelihood $P_{\mathcal{S}}(Y|F_{Y},X)$ but also to learn the prior knowledge via minimizing $\text{KL}[{\scriptstyle Q_{\mathcal{S}_{x}}(F_{Y}|X)}||{\scriptstyle P^{*}_{\mathcal{S}_{x}}(F_{Y}|X)}]$. In other words, minimizing $\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)$ inherently guarantees PAC-Bayesian generalization bounds for MLPs, even though they do not incorporate explicit regularization.

PAC-Bayesian generalization bounds for the MLP.

Given a distribution $\mathcal{D}$ over $\mathcal{X}\times\mathcal{Y}$, the samples $\mathcal{S}$, a probabilistic hypothesis set $\mathcal{H}$ derived from the $\text{MLP}=\{{x;f_{1};f_{2};f_{Y}}\}$, the cross entropy loss $\ell:\mathcal{F}\times\mathcal{X}\times\mathcal{Y}\rightarrow\mathbb{R}^{+}$ ⁵⁵5The previous works prove PAC-Bayesian bounds being valid for unbounded loss functions [10, 2]., a real number $b>0$, and a real number $\delta\in(0,1]$, with probability at least $1-\delta$ over the samples $\mathcal{S}\sim\mathcal{D}^{n}$, we have

$$\forall Q(F_{Y}|X)\text{ on }\mathcal{H}:\underset{{h\sim Q(F_{Y}|X)}}{\boldsymbol{E}}\mathcal{L}^{\ell}_{\mathcal{D}}(h)\leq\frac{b}{1-e^{-b}}\{1-\frac{\sqrt[n]{\delta\cdot P_{\mathcal{S}}(Y|X)}}{\text{exp}(\frac{1}{n}\sum_{i=1}^{n}[E_{yy_{i}}(x_{i})+\text{log}Z_{\text{MLP}}(x_{i})])}\}.$$

(22)

where $P_{\mathcal{S}}(Y|X)=\prod_{i=1}^{n}P_{Y|X}(y_{i}|x_{i})$, $E_{yy_{i}}(x_{i})=-f_{yy_{i}}(f_{2}(f_{1}(x_{i})))$ is the energy of $y_{i}$ given $x_{i}$, and $Z_{\text{MLP}}(x_{i})=\sum_{l=1}^{L}\sum_{k=1}^{K}\sum_{t=1}^{T}P(F_{Y},F_{2},F_{1}|X=x_{i})$ is the partition function.

Proof: the result is derived by two steps: (i) substituting $n\underset{h\sim\rho}{\boldsymbol{E}}\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)+\text{KL}[\rho||\pi]$ in Equation 3 for the $\text{ELBO}=\text{log}P_{\mathcal{S}}(Y|X)-\text{KL}[{\scriptstyle Q_{\mathcal{S}_{x}}(F_{Y}|X)}||{\scriptstyle P^{*}_{\mathcal{S}}(F_{Y}|X,Y)}]$ in Equation 20; and (ii) substituting $\text{KL}[{\scriptstyle Q_{\mathcal{S}_{x}}(F_{Y}|X)}||{\scriptstyle P^{*}_{\mathcal{S}}(F_{Y}|X,Y)}]$ for $\sum_{i=1}^{n}[E_{yy_{i}}(x_{i})+\text{log}Z_{\text{MLP}}(x_{i})]$ based on Equation 17 and 12.

Corollary 1 implies that when minimizing $\hat{\mathcal{L}}^{\ell}_{\mathcal{S}}(h)$ to zero, $\frac{1}{n}\sum_{i=1}^{n}[E_{yy_{i}}(x_{i})+\text{log}Z_{\text{MLP}}(x_{i})]\rightarrow 0$ and the proposed PAC-Bayesian generalization bound would be entirely determined by the marginal likelihood $P_{\mathcal{S}}(Y|X)$, which is consistent with the theoretical connection between PAC-Bayesian theory and Bayesian inference proposed by Germain et al. [10].

Based on Corollary 2, we derive an applicable bound for examining the generalization of MLPs. Though $P(Y|X)$ is unknown, the i.i.d. assumption for $\mathcal{S}$ induces $\sqrt[n]{P_{\mathcal{S}}(Y|X)}=P(Y|X)$ being a constant as long as the number of different labels are equivalent. As a result, we have

$$\text{the PAC-Bayesian generalization bound}\propto\text{exp}(\frac{1}{n}\sum_{i=1}^{n}[E_{yy_{i}}(x_{i})+\text{log}Z_{\text{MLP}}(x_{i})]).$$

(23)

Recall that $Z_{\text{MLP}}(x_{i})=\sum_{l=1}^{L}\sum_{k=1}^{K}\sum_{t=1}^{T}Q(F_{Y},F_{2},F_{1}|X=x_{i})$ sums up all the cases of $F_{Y}$, i.e., $Z_{\text{MLP}}(x_{i})$ is a constant with respect to $Q(F_{Y}|X)$. In addition, we show that the functionality of $Z_{\text{MLP}}(x_{i})$ is only to guarantee the validity of $Q(F_{Y}|X)$ (Appendix F). Therefore, $\sum_{i=1}^{n}\text{log}Z_{\text{MLP}}(x_{i})$ can be viewed as a constant for deriving PAC-Bayesian generalization bounds for MLPs, thus the bound can be further relaxed as

$$\text{the PAC-Bayesian generalization bound}\propto\frac{1}{n}\sum_{i=1}^{n}E_{yy_{i}}(x_{i}).$$

(24)

We use the above formula to approximate the generalization bound for subsequent experiments.