Non-Asymptotic Error Bounds for
Bidirectional GANs

Generative adversarial networks (GAN) (Goodfellow et al., 2014) is an important approach to implicitly learning and sampling from high-dimensional complex distributions. GANs have been shown to achieve impressive performance in many machine learning tasks (Radford et al., 2016; Reed et al., 2016; Zhu et al., 2017; Karras et al., 2018, 2019; Brock et al., 2019). Several recent studies have generalized GANs to bidirectional generative learning, which learns an encoder mapping the data distribution to the reference distribution simultaneously together with the generator doing reversely. These studies include the adversarial autoencoder (AAE) (Makhzani et al., 2015), bidirectional GAN (BiGAN) (Donahue et al., 2016), adversarially learned inference (ALI) (Dumoulin et al., 2016), and bidirectional generative modeling using adversarial gradient estimation (AGES) (Shen et al., 2020). A common feature of these methods is that they generalize the basic adversarial training framework of the original GAN from unidirectional to bidirectional. Dumoulin et al. (2016) showed that BiGANs make use of the joint distribution of data and latent representations, which can better capture the information of data than the vanilla GANs. Comparing with the unidirectional GANs, the joint distribution matching in the training of bidirectional GANs alleviates mode dropping and encourages cycle consistency (Shen et al., 2020).

Several elegant and stimulating papers have analyzed the theoretical properties of unidirectional GANs. Arora et al. (2017) considered the generalization error of GANs under the neural net distance. Zhang et al. (2018) improved the generalization error bound in Arora et al. (2017). Liang (2020) studied the minimax optimal rates for learning distributions with empirical samples under Sobolev evaluation class and density class. The minimax rate is $O(n^{-{1}/{2}}\vee n^{-{\alpha+\beta}/{(2\alpha+\beta)}})$, where $\alpha$ and $\beta$ are the regularity parameters for Sobolev density and evaluation class, respectively. Bai et al. (2019) analyzed the estimation error of GANs under the Wasserstein distance for a special class of distributions implemented by a generator, while the discriminator is designed to guarantee zero bias. Chen et al. (2020) studies the convergence properties of GANs when both the evaluation class and the target density class are Hölder classes and derived $O(n^{-{\beta}/{(2\beta+d)}}\log^{2}n)$ bound, where $d$ is the dimension of the data distribution and $\alpha$ and $\beta$ are the regularity parameters for Hölder density and evaluation class, respectively. While impressive progresses have been made on the theoretical understanding of GANs, there are still some drawbacks in the existing results. For example,

1. (a)

   The reference distribution and the target data distribution are assumed to have the same dimension, which is not the actual setting for GAN training.

2. (b)

   The reference and the target data distributions are assumed to be supported on bounded sets.

3. (c)

   The prefactors in the convergence rates may depend on the dimension $d$ of the data distribution exponentially.

In practice, GANs are usually trained using a reference distributions with a lower dimension than that of the target data distribution. Indeed, an important strength of GANs is that they can model low-dimensional latent structures via using a low-dimensional reference distribution. The bounded support assumption excludes some commonly used Gaussian distributions as the reference. Therefore, strictly speaking, the existing convergence analysis results do not apply to what have been done in practice. In addition, there has been no theoretical analysis of bidirectional GANs in the literature.

We describe the setup of the bidirectional GAN estimation problem and present the assumptions we need in our analysis.

We characterize the bidirectional GAN solutions based on minimizing the integral probability metric (IPM, Müller (1997)) between two distributions $\mu$ and $\nu$ with respect to a symmetric evaluation function class $\mathcal{F}$, defined by

$\displaystyle d_{\mathcal{F}}(\mu,\nu)=\sup_{f\in\mathcal{F}}[\mathbb{E}_{\mu}f-\mathbb{E}_{\nu}f].$

(3.1)

By specifying the evaluation function class $\mathcal{F}$ differently, we can obtain many commonly-used metrics (Liu et al., 2017). Here we focus on the following two

• •

  $\mathcal{F}=$ bounded Lipschitz function class $:d_{\mathcal{F}}=d_{BL}$, (bounded Lipschitz (or Dudley) metric: metrizing weak convergence, Dudley (2018)),

• •

  $\mathcal{F}=$ Lipschitz function class $:d_{\mathcal{F}}=W_{1}$ (Wasserstein GAN, Arjovsky et al. (2017)).

We consider the estimation error under the Dudley metric $d_{BL}$. Note that in the case when $\mu$ and $\nu$ have bounded supports, the Dudley metric $d_{BL}$ is equivalent to the 1-Wasserstein metric $W_{1}$. Therefore, under the bounded support condition for $\mu$ and $\nu$, all our convergence results also hold under the Wasserstein distance $W_{1}$. Even if the support of $\mu$ and $\nu$ are unbounded, we can still apply the result of Lu and Lu (2020) to avoid empirical process theory and obtain an stochastic error bound under the Wasserstein distance $W_{1}$. However, the result of Lu and Lu (2020) requires sub-gaussianity to obtain the $\sqrt{d}$ prefactor. In order to make it more general, we use the empirical processes theory to get the explicit prefactor. Also, the discriminator approximation error will be unbounded if we consider the Wasserstein distance $W_{1}$. Hence, we can only consider $d_{BL}$ for the unbounded support case.

The bidirectional GAN solution $(\hat{g}_{\theta},\hat{e}_{\varphi})$ in (2.1) also minimizes the distance between $(\tilde{g}_{\theta})_{\#}\hat{\nu}_{n}$ and $(\tilde{e}_{\varphi})_{\#}\hat{\mu}_{n}$ under $d_{\mathcal{F}_{NN}}$

$\displaystyle\underset{g_{\theta}\in\mathcal{G}_{NN},e_{\varphi}\in\mathcal{E}_{NN}}{\min}d_{\mathcal{F}_{NN}}((\tilde{g}_{\theta})_{\#}\hat{\nu}_{n},(\tilde{e}_{\varphi})_{\#}\hat{\mu}_{n}).$

However, even if two distributions are close with respect to $d_{\mathcal{F}_{NN}}$, there is no automatic guarantee that they will still be close under other metrics, for example, the Dudley or the Wasserstein distance   (Arora et al., 2017). Therefore, it is natural to ask the question:

• •

  How close are the two bidirectional GAN estimators $\hat{\boldsymbol{\nu}}:=(\hat{g}_{\theta},I)_{\#}\nu$ and $\hat{\boldsymbol{\mu}}:=(I,\hat{e}_{\varphi})_{\#}\mu$ under some other stronger metrics?

We consider the IPM with the uniformly bounded 1-Lipschitz function class on $\mathbb{R}^{d+1}$, as the evaluation class, which is defined as, for some finite $B>0$,

$\displaystyle\mathcal{F}^{1}:=\big{\{}f:\$

$\displaystyle\mathbb{R}^{d+1}\mapsto\mathbb{R}\big{|}\ |f(x)-f(y)|\leq\|x-y\|,x,y\in\mathbb{R}^{d+1}\text{ and }\|f\|_{\infty}\leq B\big{\}}$

(3.2)

In Theorem 3.1, we consider the bounded support case where $d_{\mathcal{F}}=W_{1}$; In Theorem 3.2, we extend the result to the unbounded support case; In Theorem 3.3, we extend the result to the case where the dimension of the reference distribution is arbitrary.

We first present a result when $\mu$ is supported on a compact subset $[-M,M]^{d}\subset\mathbb{R}^{d}$ and $\nu$ is supported on $[-M,M]\subset\mathbb{R}$ for a finite $M>0$.

The prefactor $C_{0}\sqrt{d}$ in the error bound depends on $d^{1/2}$ linearly. This is different from the existing works where the dependence of the prefactor on $d$ is either not clearly described or is exponential. In high-dimensional settings with large $d$, this makes a substantial difference in the quality of the error bounds. These remarks apply to all the results stated below.

The next theorem deals with the case of unbounded support.

Note that two methods are used in bounding stochastic errors (see appendix), which leads to two different bounds: one with an explicit $\sqrt{d}$ prefactor with the cost that we have an additional $\log n$ factor. Another one with an implicit prefactor but with a better $\log n$ factor. Hence, it is a tradeoff between the explicitness of prefactor and the order of $\log n$.

Our next result generalizes the results to the case when the reference distribution $\nu$ is supported on $\mathbb{R}^{k}$ for $k\in\mathbb{N}_{+}.$

With the above assumption, we have the following theorem providing theoretical guarantees for the validity of any dimensional reference $\nu$.

Note that the errors bounds established in Theorems 3.1-3.3 are tight up to a logarithmic factor, since the minimax rate measured in Wasserstein distance for learning distributions when the Lipschitz evaluation class is defined on $\mathbb{R}^{d}$ is $\tilde{O}(n^{-\frac{1}{d}})$ (Liang, 2020).

In this section we present a novel inequality for decomposing the total error into approximation and stochastic errors and establish bounds on these errors.

Recently, several impressive works have studied the challenging problem of the convergence properties of unidirectional GANs. Arora et al. (2017) noted that training of GANs may not have good generalization properties in the sense that even if training may appear successful but the trained distribution may be far from target distribution in standard metrics. On the other hand, Bai et al. (2019) showed that GANs can learn distributions in Wasserstein distance with polynomial sample complexity. Liang (2020) studied the rates of convergence of a class of GANs, including Wasserstein, Sobolev and MMD GANs. This work also established the nonparametric minimax optimal rate under the Sobolev IPM. The results of Bai et al. (2019) and Liang (2020) require invertible generator networks, meaning all the weight matrices need to be full-rank, and the activation function needs to be the invertible leaky ReLU activation. Chen et al. (2020) established an upper bound for the estimation error rate under Hölder evaluation and target density classes, where $\mathcal{H}^{\beta}$ is Hölder class with regularity $\beta$ and the density of the target $\mu$ is assumed to belong to $\mathcal{H}^{\alpha}$. They assumed that the reference distribution has the same dimension as the target distribution and applied the optimal transport theory to control the generator approximation error. However, how the prefactor depends in the error bounds on the dimension $d$ in the existing results (Liang, 2020; Chen et al., 2020) is either not clearly described or is exponential. In high-dimensional settings with large $d$, this makes a substantial difference in the quality of the error bounds.

Singh et al. (2019) studied minimax convergence rates of nonparametric density estimation under a class of adversarial losses and investigated how the choice of loss and the assumed smoothness of the underlying density together determine the minimax rate; they also discussed connections to learning generative models in a minimax statistical sense. Uppal et al. (2019) generates the idea of Sobolev IPM to Besov IPM, where both target density and the evaluation classes are Besov classes. They also showed how their results imply bounds on the statistical error of a GAN.

These results provide important insights in the understanding of GANs. However, as we mentioned earlier, some of the assumptions made in these results, including equal dimension between the reference and target distributions and bounded support of the distributions, are not satisfied in the training of GANs in practice. Our results avoid these assumptions. Moreover, the prefactors in our error bounds are clearly described as being dependent on the square root of the dimension $d$. Finally, the aforementioned results only dealt with unidirectional GANs. Our work is the first to address the convergence properties of bidirectional GANs.

This paper derives the error bounds for the bidirectional GANs under the Dudley distance between the latent joint distribution and the data joint distribution. The results are established without the two crucial conditions that are commonly assumed in the existing literature: equal dimensionality between the reference and the target distributions and bounded support for these distributions. Additionally, this work contributes to the neural network approximation theory by constructing neural network functions such that the pushforward distribution of an empirical distribution can perfectly approximate another arbitrary empirical distribution with a different dimension as long as their number of point masses are equal. A novel decomposition of integral probability metric is also developed for error analysis of bidirectional GANs, which can be useful in other generative learning problems.

A limitation of our results, as well as all the existing results on the convergence properties of GANs, is that they suffer from the curse of dimensionality, which cannot be circumvented by assuming sufficient smoothness assumptions. In many applications, high-dimensional complex data such as images, texts and natural languages, tend to be supported on approximate lower-dimensional manifolds. It is desirable to take into such structure in the theoretical analysis. An important extension of the present results is to show that bidirectional GANs can circumvent the curse of dimensionality if the target distribution is assumed to be supported on an approximate lower-dimensional manifold. This appears to be a technically challenging problem and will be pursued in our future work.

The authors wish to thank the three anonymous reviewers for their insightful comments and constructive suggestions that helped improve the paper significantly.

The work of J. Huang is partially supported by the U.S. NSF grant DMS-1916199. The work of Y. Jiao is supported in part by the National Science Foundation of China under Grant 11871474 and by the research fund of KLATASDSMOE. The work of Y. Wang is supported in part by the Hong Kong Research Grant Council grants 16308518 and 16317416 and HK Innovation Technology Fund ITS/044/18FX, as well as Guangdong-Hong Kong-Macao Joint Laboratory for Data-Driven Fluid Mechanics and Engineering Applications.