Noise Homogenization
via Multi-Channel Wavelet Filtering
for High-Fidelity Sample Generation in GANs

Improvements to GANs fall into three categories, including carefully selecting a good loss function [7, 5], distinctively considering regularization [8] and/or normalization [3, 9], and painstakingly designing the neural architecture [2, 3, 17], for both the generator and the discriminator. Furthermore, many of the current improvements cover more than one of the categories. For example, regularizations like gradient penalty [9, 18], consensus constraint [19], and spectral normalization [3] are specifically applied to the discriminator, so as to mitigate those gradient vanishing issues in training GANs. Also, there are similar strategies particularly for the generator, like applying spectral normalization [3] to improve training dynamics [11]. This makes things complicated and extremely hard.

As a descendant trend of deep neural networks [1], architecture design for GANs has gained the most attention. Some of the current methods merely focused on discriminator learning [20, 21], while the others were more specific on the generator [22], or both [23]. In correspondence, the GAN architecture with multiple generators has also been proposed [22]. That being said, there are a lot of complicated models proposed by stacking multiple architectures [21, 22]. Among these areas, there is another trend specific on the signal conditions, i.e., conditional training with labels [11] and unsupervised GANs trained without prior labels [10]. More recently, the self-supervised constrains [10, 23] and the self-attention mechanism [11] were also presented to improve GANs.

Besides these, architectures that manipulate the noise generation also attracts a lot of attentions. The multiplicative noise was introduced to reduce the uncertainty in GANs [24]. A decoder-encoder network was particularly designed to map the random noise vectors to informative ones and feed them to the generator of the adversarial networks [25]. A smooth generator was proposed by manipulating the output noise to cope with the perturbation during sample generation [26]. Therefore, we believe that noise manipulation provides a promising potential to produce high-fidelity sample generation in GANs. However, current options have been staying in the pixel space of noise.

Wavelet transformation, known as a well-proven and fully-utilized time-frequency signal analysis tool [27], has also been playing an important role in constructing neural networks [28]. For example, it is able to approximate arbitrary nonlinear functions [28] and replace the radial basis function [29] in neural networks. Besides these, fuzzy wavelet neural networks [30] included a wavelet function to formulate dynamic systems for a better performance. Recently, a deep wavelet network was proposed for image classification [31]. Multi-level wavelet deep CNNs were designed for image restoration [32] and fault diagnosis [33]. Wavelet transform has been integrated into a deep RNN model for natural gas demand forecasting [34]. Furthermore, various types of wavelet-like deep auto-encoders have been proposed to accelerate deep neural networks [35], as well as different applications, including image classification [36], intelligent fault diagnosis [37], medical imaging [38], spherical signal detection [39], etc. Graph wavelets were constructed to sparsely represent a given class of signals in deep learning [40].

However, no exploration has occurred around utilizing wavelets to improve image generation in GANs. It is well-known that wavelet transformation is capable of filtering and performing deconvolution for signal processing [41], for example, capturing the addition frequency information. We believe that it is helpful to homogenize the noise production in the generator, which is likely to simulate the distribution easier and be accepted by the discriminator. Here, we design our proposed WaveletGAN based on this assumption.

The formulation of GANs in the convolutional form [1] is

$$\min_{G}\max_{D}V(G,D)=\mathrm{E}_{\boldsymbol{x}\sim q_{data}}[\log D(\boldsymbol{x})]+\mathrm{E}_{\boldsymbol{z}\sim p_{G}}[\log(1-D(G(\boldsymbol{z})))],$$

(1)

where $q_{data}$ is the data distribution, and $p_{G}$ is the (model) generator distribution to be learned through the adversarial min-max optimization by the fine-tuning game between $G$ and $D$. Given a noise $\boldsymbol{z}$, $\boldsymbol{x}^{\prime}=G(\boldsymbol{z})$ is the fake sample generated by $G$, which is then to be discriminated by $D$. In a simple implementation, the noise $\boldsymbol{z}$ is the output of the generator upon an initial input noise $\boldsymbol{z}_{0}$, a.k.a. $\boldsymbol{z}=G(\boldsymbol{z}_{0})$. Then, the generated sample can be obtained by adding the output noise to the real sample, i.e., $\boldsymbol{x}^{\prime}=G(\boldsymbol{z}_{0})+x$. In this way, the formulation of this kind of simple GANs becomes

$$\min_{G}\max_{D}V(G,D)=\mathrm{E}_{\boldsymbol{x}\sim q_{\text{data }}}[\log D(\boldsymbol{x})]+\mathrm{E}_{\boldsymbol{z}\sim p_{G}}[\log(1-D(\boldsymbol{x}+G(\boldsymbol{z_{0}})))].$$

(2)

Different from the methods aiming at enhancing the discrimination of $D$, a few of the new mechanisms have been revealed to manipulate the output noise $\boldsymbol{z}$ [24, 25, 26]. Here, we are going to take advantage of the frequency decomposed by the wavelet transformations.

The key step (one and only) in our proposed method is an additional wavelet deconvolution layer, which performs a multi-channel wavelet filtering operation to homogenize the generated noise. Typically, the generator network works on a random initialized noise $\boldsymbol{z_{0}}$, i.e., the input to the generator. In conventional GANs, the generated noise was added to the real images to generate the fake samples, before being fed to the discriminator. Technically, the wavelet deconvolution supports random forms of noise, as a signal to be analyzed by wavelets, hence our method supports any GAN models. Indeed, it can be implemented in one line of code²²2See the accompanying code for this work in https://github.com/zengsn/compare_gan. . For simplicity, we give an implementation based on the GAN architecture provided in [3], which used ResNet [2] as the backbone network, for the sake of demonstrating the architecture of WaveletGAN.

We implement two versions of ResNet architectures, according to [3], for two types of input data. The generators accept the same input of the initial noise $\boldsymbol{z}_{0}\in\mathbb{R}^{128}\sim\mathcal{N}(0,I)$. However, the target outputs depend on the datasets. The first one, as shown in Table 1-(a), is for the training data $x\in\mathbb{R}^{28\times 28}$, while the other one supports the RGB images $\boldsymbol{x}\in\mathbb{R}^{32\times 32\times 3}$, as shown in Table 1-(b). Technically, it is all about the backbone networks, i.e., ResNet, in the architecture, which contains 2 or 3 ResBlocks, respectively. However, the backbone network is easily alternated to support other scales of input. We omit this part because our wavelet-based noise homogenization has nothing to do with the network, but only depends on the noise generations. As shown in Table 1, the WaveletDeconv layer is right after the output noise finalized by a $3\times 3$ convolution. We set the number of channels to be a fixed value of $5$, which in turn produces 5 channels of analyzed results. An average operation is then performed upon these resultant signals, as an operation of noise homogenization, to compose the new output noise for the following sample generation.

The noise homogenization is implemented by introducing multi-channel wavelet filtering, right after the outputted noise by the generator. First of all, we introduce the continuous wavelet transformation (CWT) [42] to the signal, i.e., the noise $\boldsymbol{z}$ outputted by the generator (see Figure 1). CWT is defined by a mother wavelet function $\mathbf{\Psi}$, which is scaled to form the wavelet functions. This transformation is usually good at processing a time series signal $x(t)$ (where $t=1\dots T$) [12], convolved with a specific wavelet function. According to the wavelet theory [27], the prerequisite to select the mother wavelet function is making sure it has small local support and satisfies the zero mean and normalization properties. We choose the Mexican Hat wavelet as the mother wavelet, since it is common for combining wavelet transformations with convolutional neural networks [43, 44, 45], whose definition can be derived from a Gaussian:

$$\psi(t)=\frac{2}{\pi^{1/4}\sqrt{3\sigma}}\left(\frac{t^{2}}{\sigma^{2}}-1\right)e^{-\frac{t^{2}}{\sigma^{2}}}.$$

(3)

By scaling and translating, this formulation can be rewritten to a new form of the wavelet function $\psi_{s,b}$:

$$\psi_{s,b}(t)=\frac{1}{\sqrt{s}}\psi\left(\frac{t-b}{s}\right),$$

(4)

where $s>0$ is the scale coefficient and $b$ is the offset to translate. Then, the CWT of a signal, i.e., decomposition of $\boldsymbol{z}$ according to $s$ and $b$, becomes a new formulation as follows:

$$W_{\boldsymbol{z}}(s,b)=\int_{-\infty}^{\infty}\frac{1}{\sqrt{s}}\psi\left(\frac{t-b}{s}\right)\boldsymbol{z}(t)dt.$$

(5)

In this way, the signal $\boldsymbol{z}$ will be transformed from one single dimension of domain ($t$) to two dimensions ($t$ and $s$) by convolution with a wavelet function $\mathbf{\Psi}$ at each scale $s$. This is the basis of how we can implement a multi-channel wavelet filtering layer in a deep convolutional neural network. It is noted that this kind of wavelet deconvolution layer (a.k.a. WaveletDeconv) has been implemented for the time series-based phone recognition task [12], which showed that the gradients of the scales can be learned using back-propagation.

Denote $E$ as a differentiable loss function to be minimized by the neural network, which is typically the categorical cross entropy, the gradient, yielded by the wavelet deconvolution layer, with respect to the scale parameter $s$ for training the network is:

$$\frac{\delta E}{\delta s_{i}}=\sum_{k=1}^{K}\frac{\delta E}{\delta\psi_{s_{i},k}}\frac{\delta\psi_{s_{i},k}}{\delta s_{i}},$$

(6)

where $K$ is the width of the filter that can be learned during training. At this time, the scales can be dynamically updated according to the gradients:

$$s_{i}^{\prime}=s_{i}-\gamma\frac{\delta E}{\delta s_{i}},$$

(7)

where $\gamma$ is the learning rate of the optimizer. We omit the full solution of Eq. 6, which can be found in [12]. We reuse the same implementation of WaveletDeconv from [12] in our model. Consequently, the new formulation for our WaveletGAN becomes:

$$\min_{G}\max_{D}V(G,D)=\mathrm{E}_{\boldsymbol{x}\sim q_{\text{data }}}[\log D(\boldsymbol{x})]+\mathrm{E}_{\boldsymbol{z}\sim p_{G}}\left[\log\left(1-D\left(\boldsymbol{x}+W(G(\boldsymbol{z_{0}}))\right)\right)\right].$$

(8)

However, there are two-fold differences between our WaveletDeconv and the original one. Firstly, we implement WaveletDeconv for image generation in GANs, which has no time series information. Secondly, we support multi-channel filtering, rather than single-channel filtering, followed by an additional averaging operation to combine multiple scales of signals into one target noise signal, and therefore homogenize the output noise. In this way, the noises before and after processing are in an identical shape, and hence our WaveletGAN has no dependency on the network architectures. WaveletGAN simply acts like a novel preprocessing technique, which is therefore very easy to implement and even can be written in one line of code, to support most of the GAN architectures.

The datasets utilized in the experiments include Fashion-MNIST [14] (a.k.a. FMNIST), KMNIST [15], and Street View House Numbers (SVHN) [16], which are all popular for benchmarking GANs. The configurations of all datasets are depicted in Tab. 2.

FMNIST and KMNIST, which are both MNIST-like datasets, contain an equal number and scale of gray images in $28\times 28$ pixels. The fashion styles and the Japanese letters all belong to 10 classes. The training and testing sets are divided in advanced, including 60,000 training samples and 10,000 test samples. Correspondingly, SVHN is an RGB image dataset, in which there are 73,257 training samples and 26,032 test samples in $32\times 32$ pixels and with $3$ channels. Due to the fact that the input images are of different sizes, the GAN architectures are under distinct designs, as previously described in Sec. 3.

No matter using which datasets, the GANs will generate fake images for the test samples after being trained using the training samples. The visualization, as well as the calculation of FIDs, will be conducted based on these generated images.

In order to intuitively evaluate WaveletGAN, we plot the generated samples in Fig. 2. Particularly, the first line shows some real samples (marked as Real) from each dataset for comparison. The following includes 3 lines of the fake samples (marked as RN) generated by the ResNet-based GAN and 3 lines of the generated samples (marked as WG) by our WaveletGAN. In this way, there are 7 lines of samples plotted for each dataset. It is noted that all the samples are directly picked up during the runtime of evaluating the models, which had no fixed order. The random appearance has less influence to judge their visualization in our opinion. For this reason, we do not reorder them manually.

However, we manually highlight some of the visibly occluded samples with the red boxes. It is obviously that more samples generated by ResNet-based GAN (in line 2 to 4) suffer the problem than the ones outputted by our WaveletGAN. For example, many fake samples of shoe (in column 8, 9, 12, and 14 of line 2 to 4) from FMNIST are over occluded, or failed in image generation. The same issue has been found in KMNIST and SVHN as well. That being said, WaveletGAN generates much fewer samples with these problems. This is merely a qualitative judgement, but we can continue to evaluate it through a quantitative metric.

Fréchet Inception Distance (FID) [13] is one of the most significant quantitative metrics to evaluate the performance of GANs. Despite the fact that other measurements, e.g., Inception Score (IS) [46] and GILBO [47], can also be utilized for this quantitative evaluation, FID is known to be popular and effective for the same job [10]. Therefore, we consider FID as the mere metric in this work. Since our WaveletGAN is merely related with the outputted noise, it supports both unsupervised and conditional settings, technically, in most of the GAN architectures. In order to calculate FIDs, we generated the same number of fake samples, as the test samples, 3 times on each dataset, before computing the average FID for each case.

During the benchmarking, we compared our WaveletGAN with most of the state-of-the-art GANs, including DC-GAN [17], WGAN-GP [9], SN-DCGAN [3], and SSGAN [10], under the unsupervised (without labels) and conditional (using labels [48]) settings. Since our benchmarking experiments were conducted on compare_gan, we reused the same configurations from the tool, and set the parameters according to the GAN benchmark settings in [10]. For example, the batch size is set to $64$ for all models, and the dimension of randomly initialized noise $\boldsymbol{z}_{0}$ is 128. The tool provides different structures of DC-GAN and SN-DCGAN, but spectral normalization (SN) [3] is designed as a configurable setting for all models. Therefore, we enabled SN on all models, including DC-GAN and WGAN-GP. For consistency, we set 5 iterations of sub-steps during each discriminator training step, and the hinge loss function was used in all GANs except SN-ResNet. Besides this, Adam was set as the optimizer for all models with a learning rate $\gamma=0.0002$. All models were trained in 100,000 steps. More detailed settings of each models can also be found in our accompanying code.

The lowest FIDs by these selected GANs are recorded in Table 3. Obviously, the smallest FIDs on all datasets are obtained by our WaveletGAN, as aforementioned, which utilizes ResNet as the backbone network. For example, the best FIDs are 4.874 (in bold style) obtained on KMNIST in unsupervised mode. The lowest FIDs on FMNIST and SVHN are 7.945 and 16.106 (in bold style), respectively. All three results are the lowest FIDs on the specific dataset. More impressively, these FIDs are much smaller than many other GANs, like DC-GAN, WGAN-GP and SN-DCGAN, which can only produce FIDs larger than 10. Furthermore, it can be observed that the recently proposed self-supervised GAN (SSGAN) generates FIDs 10.216 and over (on KMNIST), which are inferior to the results by our WaveletGAN.

It is noted that whether using the labels or not this has a less decisive impact on the results. Conditional GANs produce lower FIDs most of the time, as shown in Table 3. However, there are exceptions as well. For example, the unsupervised SN-ResNet outputs lower FIDs on KMNIST (4.983) and SVHN (16.267). This leads to the same results in our WaveletGAN, which obtained FIDs lower than the ones in the conditional setting. We focus on improving sample generation in GANs, rather than the behavior of using labels or not. The results demonstrate that wavelet-based filtering has nothing to do with this. The point is that WaveletGAN is good at generating high-fidelity samples according to the quantitive metric FID.

Technically, the wavelet-based noise homogenization indeed supports any structures of GANs, but our implementations use ResNet as the backbone network (see Table 1). This is because we observed the lowest FIDs in the SN-ResNet-based GAN implementation [3]. Therefore, we calculated the improvement rates between our WaveletGAN and SN-ResNet, as shown in the last line of Table 3.

Obviously, the highest improvements were obtained in conditional settings, where the most promising one is 3.73% on FMNIST. The improvements on the other two datasets are greater than 2.39%, which are equally impressive. Although the rate is slightly less than 1% on SVHN in the unsupervised setting, 2.78% and 2.20% are observed on FMNIST and KMNIST, respectively. This demonstrates that noise homogenization by wavelet-based filtering consistently improves the sample generation in GANs.