Exploring The Effect of High-frequency Components in GANs Training

Generative Adversarial Networks (GANs) act a two-player adversarial game, where the generator $G(z)$ is a distribution mapping function that transforms low-dimensional latent distribution $p_{z}$ to target distribution $p_{g}$. And the discriminator $D(x)$ evaluates the distance between generated distribution $p_{g}$ and real distribution $p_{r}$. The generator and discriminator minimize and maximize the zero-sum loss function, respectively. This minimax game can be expressed as:

$$\begin{split}\mathop{\min}\limits_{\phi}\mathop{\max}\limits_{\theta}f(\phi,\theta)=&\mathbb{E}_{x\sim p_{r}}[g_{1}(D_{\theta}(x))]\\ +&\mathbb{E}_{z\sim p_{z}}[g_{2}(D_{\theta}(G_{\phi}(z)))],\end{split}$$

(1)

where $\phi$ and $\theta$ are parameters of the generator $G$ and discriminator $D$, respectively. Specifically, vanilla GAN [1] can be described as $g_{1}(t)=g_{2}(-t)=-\log(1+e^{-t})$; $f$-GAN [16] can be written as $g_{1}(t)=-e^{-t},g_{2}(t)=1-t$; Morever, Geometric GAN [17] and WGAN [18] are described as $g_{1}(t)=g_{2}(-t)=-\mathop{\max}(0,1-t)$ and $g_{1}(t)=g_{2}(-t)=t$, respectively.

Recently, the quality of the generated images has been improved dramatically. Karras et al. [19] describe a new training methodology in GANs, which grows both the generator and discriminator progressively. The progressive training method makes the quality of high-resolution image synthesis improved unprecedentedly. Also, Karras et al. propose StyleGAN [4] and StyleGAN++ [5], which use style-based method to the generation of specific content. Furthermore, some regularization and normalization methods are used to stabilize GANs training [20], WGAN-GP [21] and SNGAN [14] apply 1-Lipschitz continuity using gradient penalty and spectral normalization, respectively; GAN-DAT [22] minimizes Lipschitz constant to avoid the discriminator being attacked by the adversarial examples; SS-GAN [23] uses self-supervised techniques to avoid discriminator forgetting; Similarly, InfoMAXGAN [24] improves the image generation via information maximization and contrastive learning, which is an unsupervised method for mitigating catastrophic forgetting.

In this paper, two frequency operations are used in three loss functions: vanilla GAN loss, LSGAN loss, Geometric loss (Hinge loss) and three frameworks: SNGAN, SSGAN, and InfoMAXGAN. Experiments demonstrate the consistent improvement of our methods.

Recently, frequency analysis on Convolution Neural Networks (CNNs) has attracted more and more attention. According to the different analysis objects, related studies can be divided into two categories: Input (Image) Frequency and Response Frequency. Many works have shown that there is a principle in both categories of frequency: DNNs often fit target functions from low to high frequencies during the training, which is called Frequency Principle (F-Principle) in [25, 26] or Spectral Bias in [27]. Input (Image) Frequency corresponds to the rate of change of intensity across neighboring pixels of input images, where sharp textures contribute to the high frequency of the image. Wang et al. [15] demonstrate the F-Principle of Input Frequency and point that the high-frequency components of the image improve the performance of the model, but reduce its generalization.

Furthermore, Response Frequency is the frequency of general Input-Output mapping $f$, which measures the rate of change of outputs with respect to inputs. If $f$ possesses significant high frequencies, then a small change of inputs in the image might induce a large change of the output (e.g. adversarial example in [28]). F-Principle of Response Frequency is first proposed by Xu et al. [25] through regression problems on synthetic data and real data with principal component analysis. Similarly, Rahaman et al. [27] find empirical evidence of a spectral bias: lower frequencies are learned first. They also show that lower frequencies are more robust to random perturbations of the network parameters. Subsequently, Xu et al. [29] propose a theoretical analysis framework for one hidden layer neural network with 1-d input, which illustrates activation functions (including Tanh and Relu) in the Fourier domain decays as frequency increases. Furthermore, Xu et al. [26] also propose a Gaussian filtering method that can directly verify the F-Principle in high-dimensional datasets for both regression and classification problems. Certainly, there are still a lot of works [30, 31, 32, 33, 34] to be done in the frequency domain to interpret the success and failure of CNNs.

Frequency analysis of GANs is also popular, from which we are mainly concerned with the Input (Image) Frequency. The frequency bias of GANs mainly refers to the bias between the generated images and the real images in the frequency domain, which can be observed in Figure 1(a). The same phenomenon has also been found in other works [13, 12, 9, 11, 10, 8]. All of them demonstrate that this phenomenon is pervasive and hard to avoid, even if some generated images are flawless from human perception. In addition to two main hypotheses (i.e. the employment of upsampling operations and linear dependencies in the Conv filter’s spectrum) introduced in the Introduction, Chen et al. [11] reveal that downsampling layers cause the high frequencies missing in the discriminator. This issue may make the generator lacking the gradient information to model high-frequency content, resulting in a significant spectrum discrepancy between generated images and real images. To solve it, they propose the SSD-GAN that boosts a frequency-aware classifier into the discriminator to measure the realness of images in both spatial and spectral domains.

Although the proposed SSD-GAN [11] improves the performance of GANs by increasing the discrimination of discriminator in spectral domain, our proposed HFF and HFC are orthogonal to it and the performance of SSD-GAN can also be further improved. The study that most closely resembles our paper is proposed by Yamaguchi et al. [35]. Both methods have filtered out unnecessary high-frequency components of the images during the discriminator training. In addition, our method¹¹1The Preprint of our work was published to arXiv (https://arxiv.org/abs/2103.11093) on March 20, 2021. However, the Preprint of [35] was published to arXiv (https://arxiv.org/pdf/2106.02343) on June 4, 2021. analyze the response of discriminator to different-frequency components.

For decomposing and mixing different components of images in frequency domain, we first show the DFT $\mathcal{F}$ of 2D image data $I$ with size $N\times N$:

$$\begin{split}\mathcal{F}(I)(k,l)&=\sum_{m=0}^{N-1}\sum_{n=0}^{N-1}I(m,n)\cdot e^{-2\pi j}\big{(}\frac{k}{N}m+\frac{l}{N}n\big{)},\\ \textit{for}\quad k&=0,\ldots,N-1,\quad l=0,\ldots,N-1.\end{split}$$

(2)

And the Power Spectrum (PS) of DFT is defineds as $PS(I)(k,l)=20\cdot\log\left|\mathcal{F}(I)(k,l)\right|$. At the last, the Radial Power Spectrum (RPS) of DFT via azimuthal integration can be shown as:

$$\begin{split}RPS\left(R\right)&=\int_{0}^{2\pi}PS(I)\left(R\cdot\cos(\phi),R\cdot\sin(\phi)\right)\mathrm{~{}d}\phi,\\ \textit{for}\quad R&=0,\ldots,N/\sqrt{2},\end{split}$$

(3)

where $R$ is the radius of the frequency. Figure 1(a) illustrates the RPS of real and fake images on the CelebA dataset.

In adition, we also introduce a threshold function $\mathcal{T}(;R)$ that separates the low and high frequency components according to the radius $R$. The formal definition of the equation $z_{l},z_{h}=\mathcal{T}(z;R)$ is as follows:

$$\begin{split}\mathbf{z}_{l}(i,j)=\left\{\begin{array}[]{ll}\mathbf{z}(i,j),&\text{ if }d\left((i,j),\left(c_{i},c_{j}\right)\right)\leq R\\ 0,&\text{ otherwise }\end{array}\right.,\\ \mathbf{z}_{h}(i,j)=\left\{\begin{array}[]{ll}0,&\text{ if }d\left((i,j),\left(c_{i},c_{j}\right)\right)\leq R\\ \mathbf{z}(i,j),&\text{ otherwise }\end{array}\right.,\end{split}$$

(4)

where $z(i,j)$ represents the value of $z$ at position index $(i,j)$ and $c_{i},c_{j}$ indicate the index of the center point of the image. We use $d(\cdot,\cdot)$ as the Euclidean distance.

In this paper, frequency decomposition and frequency mixing of the real image $x_{r}$ and the generated image $x_{g}$ are defined as follows:

$$\begin{split}z_{r}=\mathcal{F}(x_{r}),\quad&z^{l}_{r},z^{h}_{r}=\mathcal{T}(z_{r};R),\\ z_{g}=\mathcal{F}(x_{g}),\quad&z^{l}_{g},z^{h}_{g}=\mathcal{T}(z_{g};R),\\ x^{l}_{r}=\mathcal{F}^{-1}(z^{l}_{r}),\quad&x^{h}_{r}=\mathcal{F}^{-1}(z^{h}_{r}),\\ x^{r}_{r}=\mathcal{F}^{-1}(z^{l}_{r},\hat{z}^{h}_{r}),\quad&x^{f}_{r}=\mathcal{F}^{-1}(z^{l}_{r},z^{h}_{g}),\\ x^{l}_{g}=\mathcal{F}^{-1}(z^{l}_{g}),\quad&x^{h}_{g}=\mathcal{F}^{-1}(z^{h}_{g}),\\ x^{f}_{g}=\mathcal{F}^{-1}(z^{l}_{g},\hat{z}^{h}_{g}),\quad&x^{r}_{g}=\mathcal{F}^{-1}(z^{l}_{g},z^{h}_{r}),\\ \end{split}$$

(5)

where $\hat{z}^{h}_{r}$ represents the high-frequency components of another real image that is different from $x_{r}$. $\hat{z}^{h}_{g}$ represents the high-frequency components of another generated image that is different from $x_{g}$.

The equations in Eq (LABEL:Eq:4) define the different frequency components of the image, some are frequency decompositions ($x^{l}_{r}$, $x^{h}_{r}$, $x^{l}_{g}$, and $x^{h}_{g}$) and some are frequency mixtures ($x^{r}_{r}$, $x^{f}_{r}$, $x^{f}_{g}$, and $x^{r}_{g}$). Properties of these frequency components will be discussed in the following sections.

In this subsection, we first analyze the response of discriminator to different frequency components and demonstrate that discriminator in GANs is sensitive to high-frequency components of the images. And then, we propose two frequency operations to eliminate the bias between the generated images and the real images during the training of GANs.

First, we analyze the responses of the discriminator to different frequency components introduced in Eq (LABEL:Eq:4). The mean values of the discriminator output on the CIFAR-10 dataset for different frequency components of real images are shown at the top part of Figure 2, where real, real_low, real_real, real_fake, and real_high represent $x_{r}$, $x^{l}_{r}$, $x^{r}_{r}$, $x^{f}_{r}$, and $x^{h}_{r}$, respectively. Furthermore, the visualization of different frequency components in the real images is also shown at the bottom part of Figure 2. The result illustrates that the trained discriminator has differences in response to different frequency components ($x^{l}_{r}$, $x^{r}_{r}$, and $x^{f}_{r}$), even when $R\geq 16$. It is worth noting that under different threshold radius (R), (1) the output of $\bm{x^{f}_{r}}$ is always lower than that of $\bm{x^{r}_{r}}$; (2) the output of $\bm{x^{l}_{r}}$ is always higher than that of $\bm{x^{r}_{r}}$, which indicates that high-frequency components of images are critical to training the discriminator, and it also demonstrates that the bias between real images and generated images at high frequency is universal.

Furthermore, we demonstrate the response of the discriminator to the generated images on the CIFAR-10 dataset in Figure 3, where fake, fake_low, fake_fake, fake_real, and fake_high represent $x_{g}$, $x^{l}_{g}$, $x^{f}_{g}$, $x^{r}_{g}$, and $x^{h}_{g}$ in the Eq (4). Similar to the response of real images, the output of $x^{f}_{g}$ is lower than that of $x^{r}_{g}$. The output of $x^{l}_{g}$ is always the highest, even higher than $x_{g}$, no matter what $R$ is.

Second, Figure 1(b) illustrates some predicting defects of discriminator in the frequency domain. Original images and high-frequency components have the same prediction by the trained discriminator, while low-frequency components similar to the original image have a different prediction, which is not in line with human expectations. The results further demonstrate that discriminator of GANs is sensitive to high-frequency components of images.

It is reasonable to believe that the discriminator in GANs is sensitive to high-frequency components that is hard to be distinguished by humans. Meanwhile, the high-frequency components of images affect the training of GANs. Based on above analyses, we propose two frequency operations to eliminate high-frequency involvement during the training of GANs, which can improve the fit of the overall distribution, especially low-frequency components. Figure 4 illustrates the overview of the HFF and HFC for updating discriminator and generator during GANs training, the loss functions are:

$$\begin{split}\min_{\phi}\max_{\theta}f(\phi,\theta)&=\mathbb{E}_{x\sim p_{r}}\left[g_{1}\left(D_{\theta}({F}(x))\right)\right]\\ &+\mathbb{E}_{z\sim p_{z}}\left[g_{2}\left(D_{\theta}\left({F}(G_{\phi}(z))\right)\right)\right],\end{split}$$

(6)

where $F$ is required to be the same function (HFF or HFC) and the same threshold radius (R) across the three places illustrated in Figure 4. The framework of HFF and HFC can be seen in Figure 5, where HFF uses a filter to filter out the high-frequency components of the images, while HFC mixes two types of images by replacing the high-frequency components of the generated images with the high-frequency components of the real images. The form can be expressed as:

$$\begin{split}\begin{aligned} HFF(x_{r},x_{g};R)&=x^{l}_{r},x^{l}_{g}\\ HFC(x_{r},x_{g};R)&=x_{r},(x^{l}_{g},x^{h}_{r})\end{aligned}\end{split}$$

(7)

At the end, we also analyze the impact of the HFF for discriminator response. Figure 6 demonstrates the output of the discriminator to real images using the HFF before-and-after, respectively, where HFF indicates that the discriminator is trained with the proposed HFF method. The results illustrate: (1) the discriminator output of HFF_real_low is almost higher (i.e. more confidence) than that of real_low; (2) The gap between HFF_real_real and HFF_real_fake is less than the gap between real_real and real_fake. In particular, when the filter Radius (R) is greater than 16, the gap between HFF_real_real and HFF_real_fake is 0; (3) When the radius is greater than 8, the discriminator outputs of HFF_real_real and HFF_real_fake are also greater than that of real_real and real_fake. These three points demonstrate that the proposed HFF alleviates the sensitivity of discriminator to high-frequency and improves the fit of discriminator to the low-frequency components.

In this subsection, we apply the proposed HFF to different places ((i), (ii), (iii) in Figure 4) of the GANs training. Frechet Inception Distance (FID) and Kernel Inception Distance (KID) with using HFF in different places for training the GANs on the CIFAR-10 dataset are shown in Table I and Figure 7, respectively. The results illustrate: (1) Excessive filtering of high-frequency components makes the training of ”Real only”, ”Fake only”, and ”Dis only” methods fail, and reduces the performance of the ”All” method; (2) When the threshold radius (R) is relatively large, performance of ”Real only”, ”Fake only”, and ”Dis only” methods are similar to that of existing GANs (baseline), while performance of the ”All” method is better than that of baseline. These results show that adding artificial high-frequency bias (using ”Real only” or ”Fake only” methods only filter high-frequency components of real or fake images, which introduces the artificial high-frequency bias during the GANs training) between the real images and the generated images has little effect on the training of GANs, while removing the high-frequency bias between the generated images and the real images (”All”) improves the performance of GANs, which indicates that both natural and artificial high-frequency biases are not beneficial to the training of GANs.

This subsection validates whether our proposed frequency opeartions are robust to different loss functions. Three different loss functions (vanilla GAN [1], LSGAN [36], hinge GAN [17]) are used to train SNGAN on the CIFAR-10 dataset. FID and Inception Score (IS) are shown in Figure 8 and Table II, respectively. Compared to HFF, it is easy to understand that HFC fails to train at small Radius. HFC replaces the high-frequency components of the generated images with the high-frequency components of the real images, which leads to a significant discontinuity in the spectrum domain of the generated images as shown in Figure 5. The spectral discontinuity of the generated images leads to deviations between them and the real images, which leads to the fail of training. When R is large, this spectral discontinuity is minor, and the deviations between real images and generated images are imperceptible. Hence HFC is only suitable for larger radius.

The proposed HFC and HFF both improve the performance of the baseline. Furthermore, the best performances of HFF and HFC are achieved when R equals to 20 (the vast majority of cases), which corresponds to the phenomenon illustrated in the upper and left part of the Figure 10 (High-frequency bias occurs at the radius of 20 on the CIFAR-10 dataset).

In this subsection, HFF and HFC are adopted on some popular models, such as SNGAN, SSGAN, and InfoMAXGAN. Hinge loss is selected for all experiments. The baseline codes are available on Github²²2https://github.com/kwotsin/mimicry. We evaluate our methods on five different datasets at multiple resolutions: CIFAR-10 ($32\times 32$) [37], CIFAR-100 ($32\times 32$) [37], STL-10 ($48\times 48$) [38], LSUN-Bedroom ($64\times 64$) [39], and CelebA ($128\times 128$). Also, three metrics are adopted to evaluate the quality of generated images: Fréchet Inception Distance (FID) [40], Kernel Inception Distance (KID) [41], and Inception Score (IS) [42].

Radius (R) is an important parameter to control the range of frequency during GANs training. Here, different R settings are evaluated on four datasets³³3Both CIFAR-10 and CIFAR-100 have the same resolution, so we only demonstrate the results on CIFAR-10, STL-10, LSUN-Bedroom, and CelebA datasets. to investigate their effect on the training of GANs. First, the bias of Radial Power Spectrum between real and generated images on four datasets are demonstrated in Figure 10. Figure 10 illustrates that the bias between real and generated images on a small dataset (CIFAR-10) mainly exists in the high-frequency components. But for big datasets, such as STL-10, LSUN-Bedroom, and CelebA, all-frequency components exist the bias between real and generated images, which introduces uncertainty into the setting of the optimal Radius (R). To investigate the effect of different settings of Radius (R), Figure 9 illustrates the FID of different Radius on four datasets. It is clear that: the optimal Radius of HFF⁴⁴4The reason we mainly consider HFF is that HFC can cause model collapse when the Radius is small is set at approximately the same location as the appearance of frequency bias (R=20) on the CIFAR-10 dataset. However, due to the underfitting on complex datasets (STL-10, LSUN-Bedroom, and CelebA), the frequency bias exists in the whole frequency domain. There is uncertainty in the optimal Radius for different models.

Furthermore, we also demonstrate the whole results under the optimal Radius (R) in Table III. The results demonstrate that the proposed methods improve the performance of GANs on different architectures and datasets. For instance, compared to SNGAN, the improvement in FID of SNGAN-HFF is $7.5\%$, $9.5\%$, $4.4\%$, $6.8\%$, $42.5\%$ from CIFAR-10 (32*32), CIFAR-100 (32*32), STL-10 (48*48), LSUN (64*64) to CelebA (128*128). Similarly, compared to SSGAN, the improvement in FID of SSGAN-HFF is $12.2\%$, $9.6\%$, $9.4\%$, $10.6\%$, $30.2\%$ from CIFAR-10, CIFAR-100, STL-10, LSUN to CelebA. In particular, the improvement on the CelebA dataset is significant and impressive, which is open to interpretation. The motivation of our work is high-frequency difference between real and fake images reduces the fit of the generator to the low-frequency components, which is not beneficial for the GANs training. Based on this hypothesis, it is reasonable to assume that the proposed method will realize better performance on large resolution datasets. Dzanic et al. [9] show that at higher resolutions, the differences in frequency domain between real and fake images are more easily observed. Similarly, Figure 10 also illustrates this phenomenon.

For qualitative comparisons, we present randomly sampled images generated by SSGAN and SSGAN with HFF for CIFAR-10, CIFAR-100, STL-10, LSUN-Bedroom, and CelebA datasets in Section B of Supplementary Materials.

SSD-GAN [11] is also a method related to frequency domain in GANs training, which reveals that frequency missing in the discriminator results in a significant spectrum discrepancy between real and fake images. To mitigate this problem, SSD-GAN boosts a frequency-aware classifier into the discriminator to measure the realness of images in both spatial and spectral domains, which improves the spectral discrimination of discriminator. However, the frequency biases between real and fake images have not been eliminated. In our study, we hypothesize that high-frequency differences make the discriminator focus on high-frequency components excessively, which hinders generator from fitting the important low-frequency components. The results indicate that our methods can be combined with SSD-GAN, which further boosts the performance of GANs.

Furthermore, we compare the proposed HFF and HFC with SSD-GAN on multiple datasets, such as CIFAR-10, CIFAR-100, and STL-10 datasets. FID scores and the training time on SNGAN are given in Table IV, where we only use the HFF or HFC in spatial discriminator. The results indicate that the proposed methods improve the performance of GANs with little cost and also boost the performance of SSD-GAN. We also show the Radial Power Spectrum of different SNGAN-based methods on the STL-10 dataset in Figure 11. Compared to SNGAN (None), the low-frequency spectrum discrepancy between images generated by the proposed methods and real data is significantly reduced, where the method ”SSD+HFC” achieves the best performance. Although the proposed method reduces the focus on the high-frequency component, this does not affect the high-frequency fit and even reduces the high-frequency bias.