GANs May Have No Nash Equilibria

Since their introduction in [1], generative adversarial networks (GANs) have gained great success in many tasks of learning the distribution of observed samples. Unlike the traditional approaches to distribution learning, GANs view the learning problem as a zero-sum game between the following two players: 1) generator $G$ aiming to generate real-like samples from a random noise input, 2) discriminator $D$ trying to distinguish $G$’s generated samples from real training data. This game is commonly formulated through a minimax optimization problem as follows:

Here, $\mathcal{G}$ and $\mathcal{D}$ are respectively the generator and discriminator function sets, commonly chosen as two deep neural nets, and $V(G,D)$ denotes the minimax objective for generator $G$ and discriminator $D$ capturing how dissimilar the generated samples and training data are.

GAN optimization problems are commonly solved by alternating gradient methods, which under proper regularization have resulted in state-of-the-art generative models for various benchmark datasets. However, GAN minimax optimization has led to several theoretical and empirical challenges in the machine learning literature. Training GANs is widely known as a challenging optimization task requiring an exhaustive hyper-parameter and architecture search and demonstrating an unstable behavior. While a few regularization schemes have achieved empirical success in training GANs [2, 3, 4, 5], still little is known about the conditions under which GAN minimax optimization can be successfully solved by first-order optimization methods.

To understand the minimax optimization in GANs, one needs to first answer the following question: What is the proper notion of equilibrium in the GAN zero-sum game? In other words, what are the optimality criteria in the GAN’s minimax optimization problem? A classical notion of equilibrium in the game theory literature is the Nash equilibrium, a state in which no player can raise its individual gain by choosing a different strategy. According to this definition, a Nash equilibrium $(G^{*},D^{*})$ for the GAN minimax problem (1.1) must satisfy the following for every $G\in\mathcal{G}$ and $D\in\mathcal{D}$:

As a well-known result, for a generator $G$ expressive enough to reproduce the distribution of observed samples, Nash equilibrium exists for the generator producing the data distribution [6]. However, such a Nash equilibrium would be of little interest from a learning perspective, since the trained generator merely overfits the empirical distribution of training samples [7]. More importantly, state-of-the-art GAN architectures [4, 5, 8, 9] commonly restrict the generator function through various means of regularization such as batch or spectral normalization. Such regularization mechanisms do not allow the generator to produce the empirical distribution of observed data-points. Since the realizability assumption does not apply to such regularized GANs, the existence of Nash equilibria will not be guaranteed in their minimax problems.

The above discussion motivates studying the equilibrium of GAN zero-sum games in the non-realizable settings where the generator cannot express the empirical distribution of training data. Here, a natural question is whether a Nash equilibrium still exists for the GAN minimax problem. In this work, we focus on this question and demonstrate through several theoretical and numerical results that:

• Nash equilibrium may not exist in GAN zero-sum games.

We provide theoretical examples of well-known GAN formulations including the vanilla GAN [1], Wasserstein GAN (WGAN) [3], $f$-GAN [10], and the second-order Wasserstein GAN (W2GAN) [11] where no local Nash equilibria exist in their minimax optimization problems. We further perform numerical experiments on widely-used GAN architectures which suggest that an empirically successful GAN training may converge to non-Nash equilibrium solutions.

Next, we focus on characterizing a new notion of equilibrium for GAN problems. To achieve this goal, we consider the Nash equilibrium of a new zero-sum game where the objective function is given by the following proximal operator applied to the minimax objective $V(G,D)$ with respect to a norm on discriminator functions:

We refer to the Nash equilibrium of the new zero-sum game as the proximal equilibrium. Given the inherent sequential nature of GAN problems where the generator moves first followed by the discriminator, we consider a Stackelberg game for its representation and focus on the subgame perfect equilibrium (SPE) of the game as the right notion of equilibrium for such problems [12]. We prove that the proximal equilibrium of Wasserstein GANs provides an SPE for the GAN problem. This result applies to both the first-order and second-order Wasserstein GANs. In these cases, we show a proximal equilibrium exists for the optimal generator minimizing the distance to the data distribution.

Inspired by these theoretical results, we propose a proximal approach for training GANs, which we call proximal training, by changing the original minimax objective to the proximal objective in (1.3). In addition to preserving the optimal solution to the GAN minimax problem, proximal training can further enjoy the existence of Nash equilibrium solutions in the new minimax objective. We discuss numerical results supporting the proximal training approach and the role of proximal equilibrium solutions in various GAN problems.

Understanding the minimax optimization in modern machine learning applications including GANs has been a subject of great interest in the machine learning literature. A large body of recent works [13, 14, 15, 16, 17, 18, 19, 20, 21] have analyzed the convergence properties of first-order optimization methods in solving different classes of minimax games.

In a related work, [12] proposes a new notion of local optimality, called local minimax, designed for general sequential machine learning games. Compared to the notion of local minimax, the proximal equilibrium proposed in our work gives a notion of global optimality, which as we show directly applies to Wasserstein GANs. [12] also provides examples of minimax problems where Nash equilibria do not exist; however, the examples do not represent GAN minimax problems. Some recent works [21, 22, 23] have analyzed the convergence of different optimization methods to local minimax solutions.

In another related work, [24] analyzes the stable points of the gradient descent ascent (GDA) and optimistic GDA [13] algorithms, proving that they will give strict supersets of the local saddle points. Regarding the stability of GAN algorithms, [25] proves that the GDA algorithm will be locally stable for the vanilla and regularized Wasserstein GAN problems. [11] shows the GDA algorithm is globally stable for W2GANs with linear generator and quadratic discriminator functions.

Regarding the equilibrium in GANs, [7] studies the Nash equilibrium of GAN minimax games in realizable settings. Also, [7, 26] develop methods for finding mixed strategy Nash equilibria. On the other hand, our results focus on the pure strategies in non-realizable settings. [27] empirically studies the equilibrium of GAN problems regularized via the gradient penalty, reporting positive results on the stability of regularized GANs. However, our focus is on the existence of pure Nash equilibrium solutions. [28] suggests a moment matching GAN formulation using the Sobolev norm. As a different direction, we use the Sobolev norm to analyze equilibrium in GANs. Finally, developing GAN architectures with improved equilibrium and stability properties has been studied in several recent works [2, 29, 30, 31, 32, 33, 34, 35].

To examine whether the Nash equilibrium exists in GAN problems empirically, we performed a simple numerical experiment. In this experiment, we applied three standard GAN implementations including the Wasserstein GAN with weight-clipping (WGAN-WC) [3], the improved Wasserstein GAN with gradient penalty (WGAN-GP) [4], and the spectrally-normalized vanilla GAN (SN-GAN) [5], to the two benchmark MNIST [36] and CelebA [37] databases. We used the convolutional architecture of the DC-GAN [38] optimized with the Adam [39] or RMSprop [40] (only for WGAN-WC) optimizers.

We performed each of the GAN experiments for 200,000 generator iterations to reach $(G_{{\bm{\theta}}_{\operatorname{final}}},$ $D_{{\mathbf{w}}_{\operatorname{final}}})$ with ${\bm{\theta}}_{\operatorname{final}}$ and ${\mathbf{w}}_{\operatorname{final}}$ denoting the trained generator and discriminator parameters at the end of the 200,000 iterations. Our goal is to examine whether the solution pair $(G_{{\bm{\theta}}_{\operatorname{final}}},D_{{\mathbf{w}}_{\operatorname{final}}})$ represents a Nash equilibrium or not. To do this, we fixed the trained discriminator and kept optimizing the generator, i.e. continuing optimizing the generator $G_{\bm{\theta}}$ without changing the discriminator $D_{{\mathbf{w}}_{\operatorname{final}}}$. Here we solved the following optimization problem initialized at $\bm{\theta}^{(0)}={\bm{\theta}}_{\operatorname{final}}$ using the default first-order optimizer for the generator function for 10,000 iterations:

If the pair $(G_{{\bm{\theta}}_{\operatorname{final}}},D_{{\mathbf{w}}_{\operatorname{final}}})$ was in fact a Nash equilibrium, it would give a local saddle point to the minimax optimization and the above optimization could not make the objective any smaller than its initial value. Also, the image samples generated by the generator $G_{\bm{\theta}}$ should have improved or at least preserved their initial quality during this optimization, since the discriminator $D_{{\mathbf{w}}_{\operatorname{final}}}$ would be the optimal discriminator against all generator functions.

Despite the above predictions, we observed that none of the mentioned statements hold in reality for any of the six experiments with the three standard GAN implementations and the two datasets. The optimization objective decreased rapidly from the beginning of the optimization, and the pictures sampled from the generator completely lost their quality over this optimization. Figures 1(a), 1(b) show the objective for the SN-GAN experiments over the 10,000 steps of the above optimization. These figures also demonstrate the SN-GAN generated samples before and during the optimization, which shows the significant drop in the quality of generated pictures. We defer the results for the WGAN-WC and WGAN-GP problems to the Appendix.

The results of the above experiments show that practical GAN experiments may not converge to local Nash equilibrium solutions. After fixing the trained discriminator, the trained generator can be further optimized using a first-order optimization method to reach smaller values of the generator objective. More importantly, this optimization not only does not improve the quality of the generator’s output samples, but also totally disturbs the trained generator. As demonstrated in these experiments, simultaneous optimization of the two players is in fact necessary for the proper convergence and stability behavior in GAN minimax optimization. The above experiments suggest that practical GAN solutions are not local Nash equilibrium. In the upcoming sections, we review some standard GAN formulations and then show that there are examples of GAN minimax problems for which no Nash equilibrium exists. Those theoretical results will further support our observations in the above experiments.

Consider a general GAN minimax problem (1.1) with a minimax objective $V(G,D)$. As discussed in the previous section, the optimal generator $G^{*}\in\mathcal{G}$ is defined to minimize the GAN’s target divergence to the data distribution. The following proposition is a well-known result regarding the Nash equilibrium of the GAN game in realizable settings where there exists a generator $G\in\mathcal{G}$ producing the data distribution.

The above proposition shows that in a realizable setting with a generator function generating the distribution of observed samples, a Nash equilibrium exists for that optimal generator. However, the realizability assumption in this proposition does not always hold in real GAN experiments. For example, in the GAN experiments discussed in Section 3, we observed that the divergence estimate never reached the zero value because of regularizing the generator function. Therefore, the Nash equilibrium described in Proposition 1 does not apply to the trained generator and discriminator in such GAN experiments.

Here, we address the question of the existence of Nash equilibrium solutions for non-realizable settings, where no generator $G\in\mathcal{G}$ can produce the data distribution. Do Nash equilibria always exist in non-realizable GAN zero-sum games? The following theorem shows that the answer is in general no. Note that $\sigma_{\max}(\cdot)$ in this theorem denotes the maximum singular value, i.e., the spectral norm.

The above theorem shows that under the stated assumptions the GAN zero-sum game does not have Nash equilibrium solutions. Consequently, the optimal divergence-minimizing generative model does not result in a Nash equilibrium. In contrast to Theorem 1, the following remark shows that the GAN zero-sum game in a non-realizable case may have Nash equilibrium solutions, of course if Theorem 1’s assumptions do not hold.

The above remark explains that the phenomenon shown in Theorem 1 does not always hold in non-realizable GAN settings. As a result, we need other notions of equilibrium which consistently explain optimality in GAN games.

To define a proper notion of equilibrium for GANs, note that due to the sequential nature of GAN games the equilibrium notion should be flexible to allow to some extent the optimization of the discriminator around the equilibrium solution. This property is in fact consistent with the stability feature observed for the first-order GAN training methods where the alternating first-order method stabilizes around a certain solution. To this end, we consider the following objective for a GAN problem with minimax objective $V(G,D)$:

The above definition represents the application of a proximal operator to $V(G,D)$, which further optimizes the original objective in the proximity of discriminator $D$. To keep the $\widetilde{D}$ function variable close to $D$, we penalize the distance among the two functions in the proximal optimization. Here the distance is measured using a norm $\|\cdot\|$ on the discriminator function space.

To extend the notion of Nash equilibrium to general minimax problems, we propose considering the Nash equilibria of the defined $V_{\lambda}^{\operatorname{prox}}(G,D)$.

The next proposition provides necessary and sufficient conditions in terms of the original objective $V(G,D)$ for the proximal equilibrium solutions.

The following result shows the proximal equilibria provide a hierarchy of equilibrium solutions for different $\lambda$ values.

Note that as $\lambda$ approaches infinity, $V_{\lambda}^{\operatorname{prox}}(G,D)$ tends to the original $V(G,D)$, implying that $\operatorname{PE}_{\lambda=+\infty}(V)$ is the set of $V(G,D)$’s Nash equilibria. In contrast, for $\lambda=0$ the proximal objective becomes the worst-case objective $\max_{D\in\mathcal{D}}V(G,D)$. As a result, $\operatorname{PE}_{\lambda=0}(V)$ is the set of global minimax solutions described in Proposition 2.

Concerning the proximal optimization problem in (6.1), the following proposition shows that if the original minimax objective is a smooth function of the discriminator parameters, the proximal optimization can be solved efficiently and therefore one can efficiently compute the gradient of the proximal objective.

The above proposition suggests that under the mentioned assumptions, one can efficiently compute the optimal solution to the proximal maximization through a first-order optimization method. The assumptions require the smoothness of the GAN minimax objective with respect to the discriminator parameters, which can be imposed by applying norm-based regularization tools to neural network discriminators.

As shown earlier, GAN minimax games may not have any Nash equilibria in non-realizable settings. As a result, we seek for a different notion of equilibrium which remains applicable to GAN problems. Here, we show the proposed proximal equilibrium provides such an equilibrium notion for Wasserstein GAN problems.

To define a proper proximal operator for defining proximal equilibria in Wasserstein GAN problems, we use the second-order Sobolev semi-norm averaged over the underlying distribution of data. Given the underlying distribution $P_{\mathbf{X}}$, we define the Sobolev semi-norm as

The above semi-norm is induced by the following semi-inner product and therefore leads to a semi-Hilbert space of functions:

Throughout our discussion, we consider a parameterized set of generators $\mathcal{G}=\{G_{\bm{\theta}}:\,\bm{\theta}\in\Theta\}$. For a GAN minimax objective $V(G,D)$, we define $D^{\bm{\theta}}$ to be the optimal discriminator function for the parameterized generator $G_{\bm{\theta}}$:

The following theorem shows that the Wasserstein distance-minimizing generator function in the second-order Wasserstein GAN problem satisfies the conditions of a proximal equilibrium based on the Sobolev semi-norm defined in (7.1).

The above theorem shows that while, as demonstrated in Theorem 1, the W2GAN problem may have no local Nash equilibrium solutions, the proximal equilibrium exists for the W2GAN problem and holds at the Wasserstein-distance minimizing generator $G_{\bm{\theta}^{*}}$. The next theorem extends this result to the first-order Wasserstein GAN (WGAN) problem.

The above theorem shows that if the magnitude of optimal transport map is everywhere lower-bounded by $\frac{\lambda}{2}$, then the Wasserstein distance-minimizing generator in the WGAN problem yields a $\lambda$-proximal equilibrium.

As shown for Wasserstein GAN problems, given the defined Sobolev norm and a small enough $\lambda$ the proximal objective $V_{\lambda}^{\operatorname{prox}}(G,D)$ will possess a Nash equilibrium solution. This result motivates performing the minimax optimization for the proximal objective $V_{\lambda}^{\operatorname{prox}}(G,D)$ instead of the original objective $V(G,D)$. Therefore, we propose proximal training in which we solve the following minimax optimization problem:

with the proximal operator defined according to the Sobolev norm in (7.1).

In order to take the gradient of $V_{\lambda}^{\operatorname{prox}}(G_{\bm{\theta}},D_{\mathbf{w}})$ with respect to $\bm{\theta}$, Proposition 4 suggests solving the proximal optimization followed by computing the gradient of the original objective $V(G_{\bm{\theta}},D_{\mathbf{w}^{*}})$ where the discriminator is parameterized with the optimal solution $\mathbf{w}^{*}$ to the proximal optimization.

Algorithm 1 summarizes the main two steps of proximal training. At every iteration, the discriminator is optimized with an additive Sobolev norm penalty forcing the discriminator to remain in the proximity of the current discriminator. Next, the generator is optimized using a gradient descent method with the gradient evaluated at the optimal discriminator solving the proximal optimization. The stepsize parameter $\gamma_{k}$ can be adaptively selected at every iteration $k$. In practice, we can solve the proximal maximization problem via a first-order optimization method for a certain number of iterations. Assuming the conditions of Proposition 4 hold, the proximal optimization leads to the maximization of a strongly-concave objective which can be solved linearly fast through first-order optimization methods.

To experiment the theoretical results of this work, we performed several experiments using the [4]’s implementation of Wasserstein GANs with the code available at the paper’s Github repository. In addition, we used the implementations of [5, 47] for applying spectral regularization to the discriminator network. In the experiments, we used the DC-GAN 4-layer CNN architecture for both the discriminator and generator functions [38] and ran each experiment for 200,000 generator iterations with 5 discriminator updates per generator update. We used the RMSprop optimzier [40] for WGAN experiments with weight clipping or spectral normalization and the Adam optimizer [39] for the other experiments.