Rates of convergence for nonparametric estimation of singular distributions using generative adversarial networks

Given $D$-dimensional observations ${\bf X}_{1},\ldots,{\bf X}_{n}$ following $P_{0}$, suppose that we are interested in inferring the underlying distribution $P_{0}$ or related quantities such as its density function or the manifold on which $P_{0}$ is supported. The inference of $P_{0}$ is fundamental in unsupervised learning problems, for which numerous inferential methods are available in the literature [25, 43, 9]. In this paper, we model ${\bf X}_{i}$ as ${\bf X}_{i}={\bf g}({\bf Z}_{i})+\bm{\epsilon}_{i}$ for some function ${\bf g}:\mathcal{Z}\to{\mathbb{R}}^{D}$. Here, ${\bf Z}_{i}$ is a latent variable following the known distribution $P_{Z}$ supported on $\mathcal{Z}\subset{\mathbb{R}}^{d}$, and $\bm{\epsilon}_{i}$ is an error vector following the normal distribution $\mathcal{N}({\bf 0}_{D},\sigma^{2}{\mathbb{I}}_{D})$, where ${\bf 0}_{D}$ and ${\mathbb{I}}_{D}$ denote the $D$-dimensional zero vector and identity matrix, respectively. The dimension $d$ of latent variables is typically much smaller than $D$. The model is often called a (non-linear) factor model in statistical communities [68, 34] and a generative model in machine learning societies [22, 31]. Throughout the paper, we use the latter terminology. Accordingly, ${\bf g}$ will be referred to as a generator.

A fundamental issue in a generative model is to construct an estimator of ${\bf g}$ because inferences are mostly based on the estimation of the generator. Once we have an estimator $\hat{\bf g}$, for example, the distribution of $\hat{\bf g}({\bf Z}_{i})$ can serve as an estimator of $P_{0}$. While there are various nonparametric approaches for estimating $P_{0}$ [59, 24], the generative model approach does not provide a direct estimator due to an intractable integral. However, generative models are often more practical than the direct estimation methods because it is easy to generate samples from the estimated distribution.

Recent advances in deep learning have brought great successes to the generative model approach by modeling ${\bf g}$ through deep neural networks (DNN), for which we call a deep generative model. Two learning approaches are popularly used in practice. The former is likelihood approaches; variational autoencoder [31, 50] is perhaps the most well-known algorithm for estimating ${\bf g}$. The latter approach is known as the generative adversarial networks (GAN), which was originally developed by Goodfellow et al. [22] and generalized by several researchers. One of the extensions considers general integral probability metrics (IPM) as loss functions. Sobolev GAN [41], maximum mean discrepancy GAN [37] and Wasserstein GAN [3] are important examples. Another important direction of generalization is the development of novel architectures for generators and discriminators; deep convolutional GAN [49], progressive GAN [29] and style GAN [30] are successful architectures. In many real applications, GAN approaches tend to perform better than likelihood approaches, but training a GAN architecture is notorious for its difficulty. In particular, the estimator is very sensitive to the choice of the hyperparameters in the training algorithm.

In spite of the rapid development of GAN, theoretical understanding of it remains largely unexplored. This paper studies the statistical properties of GAN from a nonparametric distribution estimation viewpoint. Specifically, we investigate convergence rates of a GAN type estimator with a structural assumption on the generator. Although GAN does not yield an explicit estimator for $P_{0}$, it is crucial to study the convergence rate of the estimator, implicitly defined through $\hat{\bf g}$. A primary goal is to provide theoretical insights into why GAN performs well in many real-world applications. With regard to this goal, fundamental questions would be, “Which distributions can be efficiently estimated via GAN?” and “What is the main benefit of GAN compared to other methods for estimating these distributions?” The first question has recently been addressed by Chae et al. [10] to understand the benefit of deep generative models, although their results are limited to the likelihood approaches. They considered a certain class of structured distributions and tried to explain how deep generative models can avoid the curse of dimensionality in nonparametric distribution estimation problems.

To set the scene and the notation, let $Q_{\bf g}$ be the distribution of ${\bf g}({\bf Z})$, where ${\bf Z}\sim P_{Z}$. In other words, $Q_{\bf g}$ is the pushforward measure of $P_{Z}$ by the map ${\bf g}$. Let $P_{{\bf g},\sigma}=Q_{\bf g}*\mathcal{N}({\bf 0}_{D},\sigma^{2}{\mathbb{I}}_{D})$, the convolution of $Q_{\bf g}$ and $\mathcal{N}({\bf 0}_{D},\sigma^{2}{\mathbb{I}}_{D})$. A fundamental assumption of the present paper is that there exists a true generator ${\bf g}_{0}$ and $\sigma_{0}\geq 0$ such that ${\bf X}_{i}$’s are equal in distribution to ${\bf g}_{0}({\bf Z}_{i})+\bm{\epsilon}_{i}$, where $\bm{\epsilon}_{i}\sim\mathcal{N}({\bf 0}_{D},\sigma_{0}^{2}{\mathbb{I}}_{D})$ and $\bm{\epsilon}_{i}\!\perp\!\!\!\perp{\bf Z}_{i}$. With the above notation, this can be expressed as

$$P_{0}=P_{{\bf g}_{0},\sigma_{0}}=Q_{0}*\mathcal{N}({\bf 0}_{D},\sigma^{2}{\mathbb{I}}_{D}),$$

(1.1)

where $Q_{0}=Q_{{\bf g}_{0}}$. Under this assumption, it would be more reasonable to set $Q_{0}$, rather than $P_{0}$, as the target distribution to be estimated. We further assume that $Q_{0}$ possesses a certain low-dimensional structure and $\sigma_{0}\to 0$ as the sample size increases. That is, the data-generating distribution consists of the structured distribution $Q_{0}$ and small additional noise.

The above assumption has been investigated by Chae et al. [10], motivated by recent articles on structured distribution estimation [18, 17, 48, 1, 14]. Once the true generator ${\bf g}_{0}$ belongs to the class $\mathcal{G}_{0}$ possessing a low-dimensional structure that DNN can efficiently capture, deep generative models are highly appropriate for estimating $Q_{0}$. Chae et al. [10] considered a class of composite functions [26, 28] which have recently been studied in deep supervised learning [52, 6]. Some other structures have also been studied in literature [27, 44, 12]. The corresponding class of distributions $\mathcal{Q}_{0}=\{Q_{\bf g}:{\bf g}\in\mathcal{G}_{0}\}$ inherits low-dimensional structures of $\mathcal{G}_{0}$. In particular, when $\mathcal{G}_{0}$ consists of composite functions, the corresponding class $\mathcal{Q}_{0}$ is sufficiently large to include various structured distributions such as the product distributions, classical smooth distributions and distributions supported on submanifolds; see Section 4 of Chae et al. [10] for details.

The assumption on $\sigma_{0}$ is crucial for the efficient estimation of $Q_{0}$. Unless $\sigma_{0}$ is small enough, the minimax optimal rate is very slow, e.g. $1/\log n$. In statistical society, the problem is known as the deconvolution [15, 40, 17, 45]. Mathematically, the assumption for small $\sigma_{0}$ can be expressed as $\sigma_{0}\to 0$ with a suitable rate.

Once we have an estimator $\hat{\bf g}$ for ${\bf g}_{0}$, $\hat{Q}=Q_{\hat{\bf g}}$ can serve as an estimator for $Q_{0}$. Under the assumption described above, we study convergence rates of a GAN type estimator $\hat{Q}$. Note that $Q_{0}$ is singular with respect to the Lebesgue measure on ${\mathbb{R}}^{D}$ because $d$ is smaller than $D$. Therefore, standard metrics between densities, such as the total variation and Hellinger, are not appropriate for evaluating the estimation performance. We instead consider the $L^{1}$-Wasserstein metric, which is originally inspired by the problem of optimal mass transportation and frequently used in distribution estimation problems [62, 45, 66, 11].

When ${\bf g}_{0}$ possesses a composite structure with intrinsic dimension $t$ and smoothness $\beta$, see Section 3 for the definition, Chae et al. [10] proved that a likelihood approach to deep generative models can achieve the rate $n^{-\beta/2(\beta+t)}+\sigma_{0}$ up to a logarithmic factor. Due to the singularity of the underlying distribution, it plays a key role to perturb the data by an artificial noise. That is, the rate is obtained by a sieve maximum likelihood estimator based on the perturbed data $\widetilde{\bf X}_{i}={\bf X}_{i}+\widetilde{\bm{\epsilon}}_{i}$, where $\widetilde{\bm{\epsilon}}_{i}\sim\mathcal{N}({\bf 0}_{D},\widetilde{\sigma}^{2}{\mathbb{I}}_{D})$ and $\widetilde{\sigma}$ is the degree of perturbation. Without suitable data perturbation, likelihood approaches can fail to estimate $Q_{0}$ consistently. Note that the rate depends on $\beta$ and $t$, but not on $D$ and $d$.

Interestingly, a GAN type estimator considered in this paper can achieve a strictly faster rate than that of the likelihood approach. Our main result (Theorem 3.2) guarantees that a GAN type estimator achieves the rate $n^{-\beta/(2\beta+t)}+\sigma_{0}$ under the above assumption. Although Chae et al. [10] obtained only an upper bound for the convergence rate of likelihood approaches, it is hard to expect that the rate can be improved by likelihood approaches, based on the classical nonparametric theory [8, 35, 36, 67]. In this sense, our results provide some insights into why GAN approaches often perform better than likelihood approaches in real data analysis.

In addition to the convergence rate of a GAN type estimator, we obtain a lower bound $n^{-\beta/(2\beta+t-2)}$ for the minimax convergence rate; see Theorem 4.1. When $\sigma_{0}$ is small enough, this lower bound is only slightly smaller than the convergence rate of a GAN type estimator.

It would be worthwhile to mention the technical novelty of the present paper compared to existing theories about GAN reviewed in Section 1.1. Firstly, most existing theories analyze GAN from a classical nonparametric density estimation viewpoint, rather than a distribution estimation as in our paper. Classical methods such as the kernel density estimator and wavelets can also attain the minimax optimal convergence rate in their framework. Consequently, their results cannot explain why GAN outperforms classical approaches to density estimation problems.

Another notable difference lies in the discriminator architectures. While the discriminator architecture in literature depends solely on the evaluation metric ($L^{1}$-Wasserstein in our case), in our approach, it depends on the generator architecture as well. Although state-of-the-art GAN architectures such as progressive and style GANs are too complicated to render them theoretically tractable, it is crucial for the success of these procedures that discriminator architectures have pretty similar structures to the generator architectures. In the proof of Theorem 3.2, we carefully construct the discriminator class using the generator class. In particular, the discriminator class is constructed so that its complexity, expressed through the metric entropy, is of the same order as that of the generator class. Consequently, the discriminator can become a much smaller class than the set of every function with the Lipschitz constant bounded by one. This reduction can significantly improve the rate; see the discussion after Theorem 3.1.

The construction of the discriminator class in the proof of Theorem 3.2 is artificial and only for a theoretical purpose. In particular, the discriminator class is not a neural network, and the computation of the considered estimator is intractable. In spite of this limitation, our theoretical results provide important insights for the success of GAN. Focusing on the Wasserstein GAN, note that many algorithms for Wasserstein GAN [3, 23] pursue to find a minimizer, say $\hat{Q}^{W}$, of the Wasserstein distance from the empirical measure. However, even computing the Wasserstein distance between two simple distributions is very difficult; see Theorem 3 of Kuhn et al. [33]. In practice, a class of neural network functions is used as a discriminator class, and this has been understood as a technique for approximating $\hat{Q}^{W}$. However, our theory implies that $\hat{Q}^{W}$ might not be a decent estimator even when exact computation of it is possible. Stanczuk et al. [57] empirically demonstrated this by showing that Wasserstein GAN does not approximate the Wasserstein distance.

Besides the Wasserstein distance, we also consider a general integral probability metric as an evaluation metric (Theorem 3.3). For example, $\alpha$-Hölder classes can be used to define the evaluation metric. Considering state-of-the-art architectures, neural network distances [4, 71, 5, 39] would also be natural choices. Then, the corresponding GAN type estimator is much more natural than the one considered in the proof of Theorem 3.2.

The remainder of the paper is organized as follows. First, we review literature for the theory of GAN and introduce some notations in the following subsections. Then, Section 2 provides a mathematical set-up, including a brief introduction to DNN and GAN. An upper bound for the convergence rate of a GAN type estimator and a lower bound of the minimax convergence rates are investigated in Sections 3 and 4, respectively. Concluding remarks follow in Section 5. All proofs are deferred to Appendix.

For a given class $\mathcal{F}$ of functions from ${\mathbb{R}}^{D}$ to ${\mathbb{R}}$, the $\mathcal{F}$-IPM [42] between two probability measures $P_{1}$ and $P_{2}$ is defined as

$\displaystyle d_{\mathcal{F}}(P_{1},P_{2})=\sup_{f\in\mathcal{F}}|P_{1}f-P_{2}f|.$

For example, if $\mathcal{F}=\mathcal{F}_{\rm Lip}$, the class of every function $f:{\mathbb{R}}^{D}\to{\mathbb{R}}$ satisfying $|f({\bf x})-f({\bf y})|\leq|{\bf x}-{\bf y}|_{2}$ for all ${\bf x},{\bf y}\in{\mathbb{R}}^{D}$, then the corresponding IPM is the $L^{1}$-Wasserstein distance by the Kantorovich–Rubinstein duality theorem; see Theorem 1.14 of Villani [62]. Note that the $L^{p}$-Wasserstein distance (with respect to the Euclidean distance on ${\mathbb{R}}^{D}$) is defined as

$\displaystyle W_{p}(P_{1},P_{2})=\left(\inf_{\pi}\int|{\bf x}-{\bf y}|_{2}^{p}d\pi({\bf x},{\bf y})\right)^{1/p},$

where the infimum is taken over every coupling $\pi$ of $P_{1}$ and $P_{2}$.

Let $\mathcal{G}$ be a class of functions from $\mathcal{Z}\subset{\mathbb{R}}^{d}$ to ${\mathbb{R}}^{D}$, and $\mathcal{F}$ be a class of functions from ${\mathbb{R}}^{D}$ to ${\mathbb{R}}$. Two classes $\mathcal{G}$ and $\mathcal{F}$ are referred to as the generator and discriminator classes, respectively. Once $\mathcal{G}$ and $\mathcal{F}$ are given, a GAN type estimator is defined through a minimizer of $d_{\mathcal{F}}(Q_{\bf g},{\mathbb{P}}_{n})$ over $\mathcal{G}$, where ${\mathbb{P}}_{n}$ is the empirical measure based on the $D$-dimensional observations ${\bf X}_{1},\ldots,{\bf X}_{n}$. More specifically, let $\hat{\bf g}\in\mathcal{G}$ be an estimator satisfying

$$d_{\mathcal{F}}(Q_{\hat{\bf g}},{\mathbb{P}}_{n})\leq\inf_{{\bf g}\in\mathcal{G}}d_{\mathcal{F}}(Q_{\bf g},{\mathbb{P}}_{n})+\epsilon_{\rm opt},$$

(2.1)

and $\hat{Q}=Q_{\hat{\bf g}}$. Here, $\epsilon_{\rm opt}\geq 0$ represents the optimization error. Although the vanilla GAN [22] is not the case, the formulation (2.1) is quite general to include various GANs popularly used in practice [3, 37, 41]. At a population level, one may view (2.1) as a method to estimate the minimizer of $\mathcal{F}$-IPM from the data-generating distribution.

In practice, both $\mathcal{G}$ and $\mathcal{F}$ are modelled as DNNs. To be specific, let $\rho(x)=x\vee 0$ be the ReLU activation function [21]. We focus on the ReLU in this paper, but other activation functions can also be used once a suitable approximation property holds [46]. For a vector ${\bf v}=(v_{1},\ldots,v_{r})$ and ${\bf z}=(z_{1},\ldots,z_{r})$, define $\rho_{\bf v}({\bf z})=(\rho(z_{1}-v_{1}),\ldots,\rho(z_{r}-v_{r}))$. For a nonnegative integer $L$ and ${\bf p}=(p_{0},\ldots,p_{L+1})\in{\mathbb{N}}^{L+2}$, a neural network function with the network architecture $(L,{\bf p})$ is any function ${\bf f}:{\mathbb{R}}^{p_{0}}\to{\mathbb{R}}^{p_{L+1}}$ of the form

$${\bf z}\mapsto{\bf f}({\bf z})=W_{L}\rho_{{\bf v}_{L}}W_{L-1}\rho_{{\bf v}_{L-1}}\cdots W_{1}\rho_{{\bf v}_{1}}W_{0}{\bf z},$$

(2.2)

where $W_{i}\in{\mathbb{R}}^{p_{i+1}\times p_{i}}$ and ${\bf v}_{i}\in{\mathbb{R}}^{p_{i}}$. Let $\mathcal{D}(L,{\bf p},s,F)$ be the collection ${\bf f}$ of the form (2.2) satisfying

	$\displaystyle\max_{j=0,\ldots,L}|W_{j}|_{\infty}\vee|{\bf v}_{j}|_{\infty}\leq 1,$
	$\displaystyle\sum_{j=1}^{L}|W_{j}|_{0}+|{\bf v}_{j}|_{0}\leq s\quad\text{and}~{}~{}\|{\bf f}\|_{\infty}\leq F,$

where $|W_{j}|_{\infty}$ and $|W_{j}|_{0}$ denote the maximum-entry norm and the number of nonzero elements of the matrix $W_{j}$, respectively, and $\|{\bf f}\|_{\infty}=\||{\bf f}({\bf z})|_{\infty}\|_{\infty}=\sup_{\bf z}|{\bf f}({\bf z})|_{\infty}$.

When the generator class $\mathcal{G}$ consists of neural network functions, we call the corresponding class $\mathcal{Q}=\{Q_{\bf g}:{\bf g}\in\mathcal{G}\}$ of distributions as a deep generative model. In this sense, GAN can be viewed as a method for estimating the parameters in deep generative models. Likelihood approaches such as the variational autoencoder are another methods inferring the model parameters. When a likelihood approach is taken into account, $\mathcal{P}=\{P_{{\bf g},\sigma}:{\bf g}\in\mathcal{G},\sigma\in[\sigma_{\min},\sigma_{\max}]\}$ is often called a deep generative model as well. Note that $P_{{\bf g},\sigma}$ always possesses a density regardless whether $Q_{\bf g}$ is singular or not.

Although strict minimization of the map ${\bf g}\mapsto d_{\mathcal{F}}(Q_{\bf g},{\mathbb{P}}_{n})$ is computationally intractable, several heuristic approaches are available to approximate the solution to (2.1). In this section, we investigate the convergence rate of $\hat{Q}=Q_{\hat{\bf g}}$ under the assumption that the computation of it is possible. To this end, we suppose that the data-generating distribution $P_{0}$ is of the form $P_{0}=Q_{0}*\mathcal{N}({\bf 0}_{D},\sigma_{0}^{2}{\mathbb{I}}_{D})$ for some $Q_{0}$ and $\sigma_{0}\geq 0$. $Q_{0}$ will be further assumed to possess a low-dimensional structure. A goal is to find a sharp upper bound for $\mathbb{E}d_{\rm eval}(\hat{Q},Q_{0})$, where $d_{\rm eval}$ is the evaluation metric. In particular, we hope the rate to adapt to the structure of $Q_{0}$ and to be independent of $D$ and $d$. We consider an arbitrary evaluation metric $d_{\rm eval}$ for generality. The $L^{1}$-Wasserstein distance is of primary interest.

In literature, the evaluation metric is often identified with $d_{\mathcal{F}}$. In this sense, when $d_{\rm eval}=W_{1}$, $\mathcal{F}_{\rm Lip}$ might be a natural candidate for the discriminator class. Indeed, it is the original motivation of the Wasserstein GAN to find $\hat{Q}^{W}$, a minimizer of the map ${\bf g}\mapsto d_{\mathcal{F}_{\rm Lip}}(Q_{\bf g},{\mathbb{P}}_{n})$. Due to the computational intractability, $\mathcal{F}_{\rm Lip}$ is replaced by a class $\mathcal{F}$ of neural network functions in practice. Although minimizing ${\bf g}\mapsto d_{\mathcal{F}}(Q_{\bf g},{\mathbb{P}}_{n})$ is still challenging, several numerical algorithms can be used to approximate the solution. In initial papers concerning Wasserstein GAN [3, 2], this replacement was regarded only as a technique approximating $\hat{Q}^{W}$.

Theoretically, it is unclear whether $\hat{Q}^{W}$ is a decent estimator. If the generator class $\mathcal{G}$ is large enough, for example, $\hat{Q}^{W}$ would be arbitrarily close to the empirical measure. Consequently, the convergence rate of $\hat{Q}^{W}$ and ${\mathbb{P}}_{n}$ would be the same. Note that the convergence rate of the empirical measure with respect to the Wasserstein distance is well-known. Specifically, it holds that [16]

$$\mathbb{E}W_{1}({\mathbb{P}}_{n},P_{0})\lesssim\left\{\begin{array}[]{ll}n^{-1/2}&~{}~{}\text{if $D=1$}\\ n^{-1/2}\log n&~{}~{}\text{if $D=2$}\\ n^{-1/D}&~{}~{}\text{if $D>2$}.\end{array}\right.$$

(3.1)

The rate becomes slow as $D$ increases, suffering from the curse of dimensionality. Although ${\mathbb{P}}_{n}$ adapts to a certain intrinsic dimension and achieves the minimax rate in some sense [65, 55], this does not guarantee that ${\mathbb{P}}_{n}$ is a decent estimator, particularly when the underlying distribution possesses some smooth structure. The convergence rate of a GAN type estimator obtained by Schreuder et al. [54] is nothing but the rate (3.1).

If the size of $\mathcal{G}$ is not too large, then $\hat{Q}^{W}$ may achieve a faster rate than ${\mathbb{P}}_{n}$ due to the regularization effect. However, studying the behavior of $\hat{Q}^{W}$, possibly depending on the complexity of $\mathcal{G}$, is quite tricky. Furthermore, Stanczuk et al. [57] empirically showed that $\hat{Q}^{W}$ performs poorly in a simple simulation. In particular, their experiments show that estimators constructed from practical algorithms can be fundamentally different from $\hat{Q}^{W}$.

In another viewpoint, it would not be desirable to study the convergence rate of $\hat{Q}^{W}$ because it does not take crucial features of state-of-the-art architectures into account. As mentioned in the introduction, the structures of the generator and discriminator architectures are quite similar for most successful GAN approaches. In particular, the complexities of the two architectures are closely related. On the other hand, for $\hat{Q}^{W}$, the corresponding discriminator class $\mathcal{F}_{\rm Lip}$ have no connection with the generator class. In this sense, $\hat{Q}^{W}$ cannot be viewed as a fundamental estimator possessing essential properties of widely-used GAN type estimators.

Nonetheless, $d_{\mathcal{F}}$ must be close to $d_{\rm eval}$ in some sense to guarantee a reasonable convergence rate because $\mathcal{F}$ is the only way to take $d_{\rm eval}$ into account with the GAN approach (2.1). This is specified as condition (iv) of Theorem 3.1; $d_{\mathcal{F}}$ needs to be close to $d_{\rm eval}$ only on a relatively small class of distributions.

Two quantities $\epsilon_{1}$ and $\epsilon_{3}$ are closely related to the complexity of $\mathcal{G}$ and $\mathcal{F}$, respectively. In particular, $\epsilon_{1}$ represents an error for approximating $Q_{0}$ by distributions of the form $Q_{\bf g}$ over ${\bf g}\in\mathcal{G}$; the larger the generator class $\mathcal{G}$ is, the smaller the approximation error is. [69, 58, 46] Similarly, $\epsilon_{3}$ increases according as the complexity of $\mathcal{F}$ increases. Techniques for bounding $\mathbb{E}d_{\mathcal{F}}({\mathbb{P}}_{n},P_{0})$ are well-known in empirical process theory [61, 20]. The second error term $\epsilon_{2}$ is nothing but the optimization error. The fourth term $\epsilon_{4}$ is the deviance between the evaluation metric $d_{\rm eval}$ and $\mathcal{F}$-IPM over $\mathcal{Q}\cup\{Q_{0}\}$, connecting $d_{\mathcal{F}}$ and $d_{\rm eval}$. Finally, the term $d_{\mathcal{F}}(P_{0},Q_{0})$ in the rate depends primarily on $\sigma_{0}$. One can easily prove that

$$d_{\mathcal{F}}(P_{0},Q_{0})\leq W_{1}(P_{0},Q_{0})\leq W_{2}(P_{0},Q_{0})\lesssim\sigma_{0}.$$

(3.3)

provided that $\mathcal{F}\subset\mathcal{F}_{\rm Lip}$.

Ignoring the optimization error, suppose for a moment that $\mathcal{G}$ is given and we need to choose a suitable discriminator class to minimize $\epsilon_{3}+\epsilon_{4}$ in Theorem 3.1. We focus on the case of $d_{\rm eval}=W_{1}$. One can easily make $\epsilon_{4}=0$ by taking $\mathcal{F}=\mathcal{F}_{\rm Lip}$. In this case, however, $\epsilon_{3}$ would be too large because $\mathbb{E}W_{1}({\mathbb{P}}_{n},P_{0})\asymp n^{-1/D}$. That is, $\mathcal{F}_{\rm Lip}$ is too large to be used as a discriminator class; $\mathcal{F}$ should be a much smaller class than $\mathcal{F}_{\rm Lip}$ to obtain a fast convergence rate. To achieve this goal, we construct $\mathcal{F}$ so that $\epsilon_{3}$ is small enough while $\epsilon_{4}$ retains small. For example, we may consider

$$\mathcal{F}=\Big{\{}f_{Q_{1},Q_{2}}:Q_{1},Q_{2}\in\mathcal{Q}\cup\{Q_{0}\}\Big{\}},$$

(3.4)

where $f_{Q_{1},Q_{2}}$ is a (approximate) maximizer of $|Q_{1}f-Q_{2}f|$ over $f\in\mathcal{F}_{\rm Lip}$. In this case, $\epsilon_{4}$ vanishes, hence the convergence rate of $\hat{Q}$ will be determined solely by $\epsilon_{1}$, $\epsilon_{3}$ and $\sigma_{0}$. Furthermore, the complexity of $\mathcal{F}$ would roughly be the same to that of $\mathcal{G}\times\mathcal{G}$. If the complexity of a function class is expressed through a metric entropy, the complexities of $\mathcal{G}$ and $\mathcal{F}$ are of the same order. Three quantities $\epsilon_{1}$, $\epsilon_{3}$ and $\sigma_{0}$ can roughly be interpreted as the approximation error, estimation error and noise level. While we cannot control $\sigma_{0}$, both the approximation and estimation errors depend on the complexity of $\mathcal{G}$, hence a suitable choice of it would be important to achieve a fast convergence rate.

To give a specific convergence rate, we consider a class of structured distributions considered by Chae et al. [10] for which deep generative models have benefits. For positive numbers $\beta$ and $K$, let $\mathcal{H}^{\beta}_{K}(A)$ be the class of all functions from $A$ to ${\mathbb{R}}$ with $\beta$-Hölder norm bounded by $K$ [61, 20]. We consider the composite structure with low-dimensional smooth component functions as described in Section 3 of Schmidt-Hieber [52]. Specifically, we consider a function ${\bf g}:{\mathbb{R}}^{d}\to{\mathbb{R}}^{D}$ of the form

$${\bf g}={\bf h}_{q}\circ{\bf h}_{q-1}\circ\cdots\circ{\bf h}_{1}\circ{\bf h}_{0}$$

(3.5)

with ${\bf h}_{i}:(a_{i},b_{i})^{d_{i}}\to(a_{i+1},b_{i+1})^{d_{i+1}}$. Here, $d_{0}=d$ and $d_{q+1}=D$. Denote by ${\bf h}_{i}=(h_{i1},\ldots,h_{id_{i+1}})$ the components of ${\bf h}_{i}$ and let $t_{i}$ be the maximal number of variables on which each of the $h_{ij}$ depends. Let $\mathcal{G}_{0}(q,{\bf d},{\bf t},\bm{\beta},K)$ be the collection of functions of the form (3.5) satisfying $h_{ij}\in\mathcal{H}^{\beta_{i}}_{K}\big{(}(a_{i},b_{i})^{t_{i}}\big{)}$ and $|a_{i}|\vee|b_{i}|\leq K$, where ${\bf d}=(d_{0},\ldots,d_{q+1})$, ${\bf t}=(t_{0},\ldots,t_{q})$ and $\bm{\beta}=(\beta_{0},\ldots,\beta_{q})$. Let

	$\displaystyle\widetilde{\beta}_{j}$	$\displaystyle=$	$\displaystyle\beta_{j}\prod_{l=j+1}^{q}(\beta_{l}\wedge 1),\quad j_{*}=\operatorname*{argmax}_{j\in\{0,\ldots,q\}}\frac{t_{j}}{\widetilde{\beta}_{j}},$
	$\displaystyle\beta_{*}$	$\displaystyle=$	$\displaystyle\widetilde{\beta}_{j_{*}}\quad\text{and}~{}~{}t_{*}=t_{j_{*}}.$

We call $t_{*}$ and $\beta_{*}$ as the intrinsic dimension and smoothness of ${\bf g}$ (or of the class $\mathcal{G}_{0}(q,{\bf d},{\bf t},\bm{\beta},K))$, respectively. The class $\mathcal{G}_{0}(q,{\bf d},{\bf t},\bm{\beta},K)$ has been extensively studied in recent articles on deep supervised learning to demonstrate the benefit of DNN in nonparametric function estimation [52, 6].

Let

$\displaystyle\mathcal{Q}_{0}(q,{\bf d},{\bf t},\bm{\beta},K)=\Big{\{}Q_{\bf g}:{\bf g}\in\mathcal{G}_{0}(q,{\bf d},{\bf t},\bm{\beta},K)\Big{\}}.$

Quantities $(q,{\bf d},{\bf t},\bm{\beta},K)$ are constants independent of $n$. In the forthcoming Theorem 3.2, we obtain a Wasserstein convergence rate of a GAN type estimator $\hat{Q}$ under the assumption that $Q_{0}\in\mathcal{Q}_{0}(q,{\bf d},{\bf t},\bm{\beta},K)$.

In Theorem 3.2, network parameters $(L,{\bf p},s)$ of $\mathcal{G}$ depend on the sample size $n$. More specifically, it can be deduced from the proof that one can choose $L\lesssim\log n$, $|{\bf p}|_{\infty}\lesssim n^{-t_{*}/(2\beta_{*}+t_{*})}\log n$ and $s\lesssim n^{-t_{*}/(2\beta_{*}+t_{*})}\log n$. As illustrated above, the discriminator class $\mathcal{F}$ is carefully constructed using $\mathcal{G}$.

Ignoring the optimization error $\epsilon_{\rm opt}$, the rate (LABEL:eq:rate-composition) consists of the two terms, $\sigma_{0}$ and $n^{-\beta_{*}/(2\beta_{*}+t_{*})}$ up to a logarithmic factor. If $\sigma_{0}\lesssim n^{-\beta_{*}/(2\beta_{*}+t_{*})}$, it can be absorbed into the polynomial term; hence when $\sigma_{0}$ is small enough, $\hat{Q}$ achieves the rate $n^{-\beta_{*}/(2\beta_{*}+t_{*})}$. Note that this rate appears in many nonparametric smooth function estimation problems.

The dependence on $\sigma_{0}$ comes from the term $d_{\mathcal{F}}(P_{0},Q_{0})$ in Theorem 3.1 and the inequality (3.3). Note that (3.3) holds because $\mathcal{F}$ is a subset of $\mathcal{F}_{\rm Lip}$. We are not aware whether the term $\sigma_{0}$ in (LABEL:eq:rate-composition) can be improved in general. If we consider another evaluation metric, however, it is possible to improve this term. For example, if $\mathcal{F}$ consists of twice continuously differentiable functions, it would be possible to prove $d_{\mathcal{F}}(P_{0},Q_{0})\lesssim\sigma_{0}^{2}$. This is because for a twice continuously differentiable $f$,

$$\begin{split}&|P_{0}f-Q_{0}f|=\big{|}\mathbb{E}[f({\bf Y}+\bm{\epsilon})-f({\bf Y})]\big{|}\\ &\approx\Big{|}\mathbb{E}\Big{[}\bm{\epsilon}^{T}\nabla f({\bf Y})+\frac{1}{2}\bm{\epsilon}^{T}\nabla^{2}f({\bf Y})\bm{\epsilon}\Big{]}\Big{|}\asymp\sigma_{0}^{2},\end{split}$$

(3.7)

where ${\bf Y}\sim Q_{0},\bm{\epsilon}\sim\mathcal{N}({\bf 0}_{D},\sigma_{0}^{2}{\mathbb{I}})$ and ${\bf Y}\!\perp\!\!\!\perp\bm{\epsilon}$.

Note that a sieve MLE considered by Chae et al. [10] achieves the rate $n^{-\beta_{*}/2(\beta_{*}+t_{*})}+\sigma_{0}$ under a slightly stronger assumption than that of Theorem 3.2. Hence, for a moderately small $\sigma_{0}$, the convergence rate of a GAN type estimator is strictly better than that of a sieve MLE. This result provides some insight into why GAN approaches perform better than likelihood approaches in many real applications. In particular, if GAN performs significantly better than likelihood approaches, it might be a reasonable inference that the noise level of the data is not too large. On the other hand, if the noise level is larger than a certain threshold, $P_{0}$ is not nearly singular anymore. In this case, likelihood approaches would be preferable to computationally much more difficult GAN.

In practice, the noise level $\sigma_{0}$ is unknown, so one may firstly try likelihood approaches with different levels of perturbation. As empirically demonstrated in Chae et al. [10], the data perturbation significantly improves the quality of generated samples provided that $\sigma_{0}$ is small enough. Therefore, if the data perturbation improves the performance of likelihood approaches, one may next try GAN to obtain a better estimator.

So far, we have focused on the case $d_{\rm eval}=W_{1}$. Note that the discriminator class considered in the proof of Theorem 3.2 is of the form (3.4), which is far away from a practical one. In particular, it is unclear whether it is possible to achieve the rate (LABEL:eq:rate-composition) with neural network discriminators. If we consider a different evaluation metric, however, one can easily obtain a convergence rate using a neural network discriminator. When $\mathcal{F}_{0}$ consists of neural networks, $\mathcal{F}_{0}$-IPM is often called a neural network distance. It is well-known under mild assumptions that the convergence of probability measures in a neural network distance guarantees the weak convergence [71]. Therefore, neural network distances are also good candidates for evaluation metrics. In Theorem 3.3, more general integral probability metrics are taken into account.

The proof of Theorem 3.3 can be divided into two cases. Firstly, if the complexity of $\mathcal{F}_{0}$ is small enough in the sense of (3.9), one can ignore the approximation error ($\epsilon_{1}$ in Theorem 3.1) by taking an arbitrarily large $\mathcal{G}$. Also, $\mathcal{F}=\mathcal{F}_{0}$ leads to $\epsilon_{4}=0$. Hence, the rate is determined by $\sigma_{0}$, $\epsilon_{\rm opt}$ and $\mathbb{E}d_{\mathcal{F}_{0}}({\mathbb{P}}_{n},P_{0})$. On the other hand, if (3.9) does not hold, we construct the discriminator as in (3.4). This leads to the same convergence rate with Theorem 3.2.

If $\mathcal{F}_{0}$ is a subset of $\mathcal{D}(L_{0},{\bf p}_{0},s_{0},\infty)$, then it is not difficult to see that $\mathbb{E}d_{\mathcal{F}_{0}}({\mathbb{P}}_{n},P_{0})\lesssim\sqrt{s_{0}/n}$ up to a logarithmic factor. This can be proved using well-known empirical process theory and metric entropy of deep neural networks; see Lemma 5 of Schmidt-Hieber [52]. Note that functions in $\mathcal{F}_{0}$ are required to be Lipschitz continuous and there are several regularization techniques bounding Lipschitz constants of DNN [2, 51].

Another important class of metrics is a Hölder IPM. When $\mathcal{F}_{0}=\mathcal{H}^{\alpha}_{1}([-K,K]^{D})$ for some $\alpha>0$, Schreuder [53] has shown that

$\displaystyle\mathbb{E}d_{\mathcal{F}_{0}}({\mathbb{P}}_{n},P_{0})\lesssim\left\{\begin{array}[]{ll}n^{-\alpha/D}&~{}~{}\text{if $\alpha<D/2$}\\ n^{-1/2}\log n&~{}~{}\text{if $\alpha=D/2$}\\ n^{-1/2}&~{}~{}\text{if $\alpha>D/2$}.\end{array}\right.$

Furthermore, for $\alpha>2$, $\sigma_{0}$ in (LABEL:eq:rate-ipm) can be replaced by $\sigma_{0}^{2}$ using (LABEL:eq:sigma2).

In this section, we study a lower bound of the minimax convergence rate, particularly focusing on the case $d_{\rm eval}=W_{1}$. As in the previous section, suppose that ${\bf X}_{1},\ldots,{\bf X}_{n}$ are i.i.d. random vectors following $P_{0}=P_{{\bf g}_{0},\sigma_{0}}$. We investigate the minimax rate over the class $\mathcal{Q}_{0}=\{Q_{\bf g}:{\bf g}\in\mathcal{G}_{0}\}$, where $\mathcal{G}_{0}=\mathcal{G}_{0}(q,{\bf d},{\bf t},\bm{\beta},K)$. For simplicity, we consider the case $q=0$, $d_{0}=t_{0}=d$ and $\beta_{0}=\beta$. In this case, we have $\mathcal{G}_{0}=\mathcal{H}_{K}^{\beta}([0,1]^{d})\times\cdots\times\mathcal{H}_{K}^{\beta}([0,1]^{d})$. An extension to general $q$ would be slightly more complicated but not difficult. We also assume that $P_{Z}$ is the uniform distribution on $[0,1]^{d}$.

For given $\mathcal{G}_{0}$ and $\sigma_{0}\geq 0$, the minimax risk is defined as

$\displaystyle\mathfrak{M}(\mathcal{G}_{0},\sigma_{0})=\inf_{\hat{Q}}\sup_{{\bf g}_{0}\in\mathcal{G}_{0}}\mathbb{E}W_{1}(\hat{Q},Q_{0}),$

where the infimum ranges over all possible estimators. Several techniques are available to obtain a lower bound for $\mathfrak{M}(\mathcal{G}_{0},\sigma_{0})$ [59, 64]. We utilize the Fano’s method to prove the following theorem.

Note that the lower bound (4.1) does not depend on $\sigma_{0}$. With a direct application of the Le Cam’s method, one can easily show that $\mathfrak{M}(\mathcal{G}_{0},\sigma_{0})\gtrsim\sigma_{0}/\sqrt{n}$. As discussed in the previous section, it would not be easy to obtain a sharp rate with respect to $\sigma_{0}$. Since we are more interested in small $\sigma_{0}$ cases (i.e. nearly singular cases), our discussion focuses on the cases where $\sigma_{0}$ is ignorable.

Note that the lower bound (4.1) is only slightly smaller than the rate $n^{-\beta/(2\beta+d)}$, the first term in the right hand side of (LABEL:eq:rate-composition). Hence, the convergence rate of a GAN type estimator is at least very close to the minimax optimal rate.

For the gap between the upper and lower bounds, we conjecture that the lower bound is sharp and cannot be improved. This conjecture is based on the result in Uppal et al. [60] and Liang [38]. They considered GAN for nonparametric density estimation, hence $D=d$ in their framework. For example, Theorem 4 in Liang [38] guarantees that, for $D=d\geq 2$ and $\sigma_{0}=0$,

$$\inf_{\hat{Q}}\sup_{Q_{0}\in\widetilde{\mathcal{Q}}_{0}}\mathbb{E}W_{1}(\hat{Q},Q_{0})\asymp n^{-\frac{\beta^{\prime}+1}{2\beta^{\prime}+d}},$$

(4.2)

where $\widetilde{\mathcal{Q}}_{0}=\{Q:q\in\mathcal{H}^{\beta^{\prime}}_{1}([0,1]^{D})\}$. (More precisely, he considered Sobolev classes instead of Hölder classes.) Interestingly, there is a close connection between the density model $\widetilde{\mathcal{Q}}_{0}$ in literature and the generative model $\mathcal{Q}_{0}$ considered in our paper. This connection is based on the profound regularity theory of the optimal transport, often called the Brenier map. Roughly speaking, for a $\beta^{\prime}$-Hölder density $q$, there exits a $(\beta^{\prime}+1)$-Hölder function ${\bf g}$ such that $Q=Q_{\bf g}$; see Theorem 12.50 of Villani [63] for a precise statement. We also refer to Lemma 4.1 of Chae et al. [10] for a concise statement. In this sense, the density model $\widetilde{Q}_{0}$ matches with the generative model $Q_{0}$ when $\beta=\beta^{\prime}+1$. In this case, two rates (4.1) and (4.2) are the same, and this is why we conjecture that the lower bound (4.1) cannot be improved. Unfortunately, proof techniques in Uppal et al. [60] and Liang [38] for both upper and lower bounds are not generalizable to our case because $Q_{0}$ does not possess a Lebesgue density.

Under a structural assumption on the generator, we have investigated the convergence rate of a GAN type estimator and a lower bound of the minimax optimal rate. In particular, the rate is faster than that obtained by likelihood approaches, providing some insights into why GAN outperforms likelihood approaches. This result would be an important step-stone toward a more advanced theory that can take into account fundamental properties of state-of-the-art GANs. We conclude the paper with some possible directions for future work.

Firstly, reducing the gap between the upper and lower bounds of the convergence rate obtained in this paper would be necessary. As discussed in Section 4, it will be crucial to construct an estimator that achieves the lower bound in Theorem 4.1. More specifically, we wonder whether a GAN type estimator can do this. Next, when $d_{\rm eval}=W_{1}$, an important question is whether it is possible to choose $\mathcal{F}$ as a class of neural network functions. Perhaps, we may not obtain the rate in Theorem 3.2 because a large network would be necessary to approximate an arbitrary Lipschitz function. Finally, based on the approximation property of the convolutional neural networks (CNN) architectures [32, 70], studying the benefit of CNN-based GAN would be an intriguing problem.