Phase-aware music super-resolution using generative adversarial networks

Audio SR with deep learning is usually approached from either frequency or time domain. One of the earliest frequency-domain approaches was proposed in [6] using restricted Boltzmann machines as DNNs for training the mapping between magnitudes of LFC and HFC. A recent study introduced GANs for the purpose and shows the state-of-the-art results [8, 9]. A typical time-domain approach was proposed recently  [7] which applied an auto-encoder network to generate HR waveform from the LR temporal signals.

While the GANs-based spectra-mapping method shows promising results, the HFC phase for HR waveform reconstruction was not fully considered. In the existing works, it is artificially produced by flipping and repeating the LFC phase and adding a negative sign (denoted as FLIP). Since phase has been confirmed to effectively improve subjective perception on speech [12, 13], we attempt to predict HFC magnitudes complemented by the estimated phase in order to recover the HR waveform with improvements in music SR tasks.

Here two common methods for phase estimation are briefly introduced. One is Griffin-Lim algorithm (GLA), and the other one is based on speech vocoder. They are used and compared in our music SR tasks.

GLA has been commonly used as a phase recovery method based on the consistency of spectrogram[16, 17]. GLA method iteratively updates the complex-valued spectrogram while maintains the given magnitude (A). The process of GLA can be explained by the following equation (here we largely followed the notation used in [18]):

$$\textbf{X}^{[m+1]}=P_{C}(P_{A}(\textbf{X}^{[m]})),$$

(1)

where $\textbf{X}^{[m]}$ denotes the complex-valued spectrogram updated through the $m$-th iteration. $P_{A}$ updates the phase and maintains the given magnitudes, i.e., A. $P_{C}$ calculates the corresponding consistent spectrograms. $P_{C}$ and $P_{A}$ are respectively given by:

$$P_{C}(\textbf{X})=\mathbb{F}\mathbb{F^{\dagger}}\textbf{X},$$

(2)

$$P_{A}(\textbf{X})=\textbf{A}\odot\textbf{X}\oslash|\textbf{X}|,$$

(3)

where $\mathbb{F}$ and $\mathbb{F^{\dagger}}$ are the forward short-time Fourier transform (STFT) and inverse STFT, respectively. $\odot$ is element-wise multiplication; $\oslash$ represents element-wise division and division by zero is replaced by zero. Through iterations, complex-valued spectrograms are consistently optimized by minimizing the difference between the energy of X and $P_{C}(\textbf{X})$. To adapt GLA to our music SR tasks, we not only maintain the given LFC magnitude but also the LFC phase in Eq. 3. Although GLA has been widely used for the past decades due to its simplicity, it requires many iterations and resulted reconstruction quality is not good enough. In the experiments of this study, the GLA is implemented with $100$ iterations, following the number of iterations used in [18].

Recently, neural-network-based speech vocoder has been extensively developed for waveform synthesis over the past few years. Although the neural-network-based vocoder has been generally developed for speech applications, it may have promising performance as well when applied to music audio modeling [19]. Here we decide to use a state-of-the-art method proposed in [15], which generates audio waveform from mel-spectrogram using GANs (denoted as MelGAN). The MelGAN model consists of a fully convolutional generator network that generate raw waveform from mel-spectrogram input, and a discriminator network with a multi-scale architecture. In our task, a waveform can be generated using the mel-spectorgram concatenating the LFC of the input LR signal and the estimated HFC. The phase information of HFC can be therefore extracted from the generated audio.

We propose a joint music SR framework, mainly consisting of a “magnitude component” (MC) and a “phase component” (PC). Following the GANs architecture presented in [9], MC predicts the magnitudes of HFC from the LFC. Magnitude prediction is done by a generator network with a convolutional auto-encoder architecture. Besides, a discriminator network is designed to distinguish the estimated magnitudes from real ones. To train MC, we used Wasserstein distance [20] as GAN loss function to stabilize the training. As for PC, we used the the MelGAN model presented in [15] to convert mel-spectrogram into waveform, from which the phase of HFC can be estimated.

Fig. 1 presents the whole joint framework. Both the magnitudes from MC and the phase from PC are used to obtain complex-valued spectrogram of high frequency bands. To estimate phase, PC with MelGAN is used in order to obtain a more accurate phase information for audio reconstruction. It is compared with another two strategies, namely FLIP and GLA. With this joint methods, quality of the reconstructed audio is expected to be improved. A detailed calculation process is illustrated as follows:

• •

  Given the LR waveform input ($\textbf{x}_{LR}$), calculate its LFC spectrogram ($\textbf{X}_{LFC}$) with $L$ time steps and $K$ frequency bins;

• •

  Use MC to predict magnitude of HFC ($\textbf{\^{M}}_{HFC}$) with $N$ frequency bins from the magnitude of $\textbf{X}_{LFC}$, i.e., $\textbf{M}_{LFC}$;

• •

  Concatenate $\textbf{\^{M}}_{HFC}$ and $\textbf{M}_{LFC}$ as the input to PC for generating the corresponding waveform ($\textbf{\^{x}}_{MelGAN}$);

• •

  HFC phase ($\textbf{\^{P}}_{HFC}$) can be extracted from $\textbf{\^{x}}_{MelGAN}$, or estimated using FLIP or GLA;

• •

  Using $\textbf{\^{P}}_{HFC}$ and $\textbf{\^{M}}_{HFC}$, spectrogram of HFC ($\textbf{\^{X}}_{HFC}$) can be obtained;

• •

  Concatenate $\textbf{\^{X}}_{HFC}$ and $\textbf{X}_{LFC}$ to generate HR spectrogram ($\textbf{\^{X}}_{HR}$);

• •

  Finally, HR waveform ($\textbf{\^{x}}_{HR}$) can be obtained from $\textbf{\^{X}}_{HR}$ using iSTFT.

The proposed method is compared with the FLIP and GLA methods as different options for HFC phase estimation as presented in Fig. 1.

For the magnitude components, we adjust the network architecture and configuration in [9] for this task. During the training process, generator of MC takes $\textbf{M}_{LFC}$ with $L=32$ as input. The encoder block contains 4 convolution layers, which have 372,512,512 and 1024 channels with filter size of 7,5,3,3 and stride of 2,2,2,2. The decoder block contains 3 convolution layers, which have 1024, 1024 and 744 channels with filter size of 3,5,7 and stride 1,1,1. To upscale the temporal feature maps in the decoder block, the subpixel layer proposed in Shi et al. [21] is used after each LeakyReLU activation. A skip connection is added between each encoder layer and the subpixel output of the corresponding decoder layer. Besides, batch normalization is used after each convolution layer and before the LeakyReLU activation. Following the decoder block, two convolution layers are used to generate the $\textbf{\^{M}}_{HFC}$. They contain $2N$ channels with filter size of 7 and stride 1 and a subpixel layer, $N$ channels with filter size of 9 and stride 1, respectively. As for the discriminator of MC, it consists of three convolution layers, each of which contain 1024 channels with stride of 7, 5 and 3, respectively. These layers are followed by two fully connected layers with 2048 and 1 neurons respectively, and a Sigmoid activation function for the final output of MC.

For the phase component, we use the network architecture described in [15]. The input of PC is $32\times 160$ log-mel magnitudes transformed from the MC-predicted linear-scaled magnitudes. PC can then output a waveform of 8192 samples. We only make use of the HFC phase information extracted from the waveform. LFC phase is kept the same as the one calculated from LFC spectrogram of the LR waveform.

To solely investigate the importance of phase in the audio SR task, we compare FLIP (i.e., phase flipping), GLA, and MelGAN methods with the assumption that we have knowledge of full-band magnitudes and LFC phase. Given the full-band magnitudes ($0-8$kHz) and LFC phase ($0-4$kHz), the task here is to generate phase information of HFC such that HR waveform can be reconstructed. For the FLIP method, the high-frequency phase is produced by flipping the phase of LFC and adding a negative sign. For the GLA method, as introduced in Sec. 2.2, a modified version of GLA is used to maintain both the magnitudes and phase of LFC through the iteration process. For the MelGAN method, HFC phase is extracted and complemented by the give magnitudes as well as LFC information in order to obtain the final HR output.

The comparison results of the three phase strategies are presented in Table 1. It shows that both the MelGAN and GLA method significantly outperforms the FLIP method used in [6, 9] in terms of the LSD metric. An example of the generated spectrograms is shown in Fig. 2 which indicates that FLIP method weakens the energy of HFC due to the poor consistence between magnitudes and the produced phase, while GLA and MelGAN method generate magnitudes close to the ground truth.

It should be noted that the original LR input yields a better SNR value than any other methods used in the comparison. To explain it, we plot an example of ground-truth HR waveform, along with its LR waveform and the reconstructed waveform by GLA as shown in Fig. 3. It presents that bandwidth expansion can effectively generate finer details that are missing in the LR waveform. These details correspond to audible improvements in the high-frequency range. However, the artifacts introduced by the expansion process may severely impact the SNR value. Therefore, the SNR metric may be not suitable for the frequency-domain music SR task. For this reason, we choose the LSD metric alone in the following experiment.

To evaluate the effectiveness of the proposed joint framework, we implement our approach and compare it with the following state-of-the-art methods:

• •

  F-DNN: Fully-connected network implementation in the frequency domain (Li et al. [6])

• •

  T-CNN: CNN implementation in the time domain (Kuleshov et al. [7])

• •

  F-CNN: CNN (generator network of GAN) implementation in the frequency domain (Eskimez et al. [9])

• •

  F-GAN: GAN implementation in the frequency domain with phase flipping strategy (Eskimez et al. [9])

• •

  Proposed: GAN implementation in the frequency domain for HB magnitude estimation and MelGAN implementation for HB phase estimation.

Fig. 4 plots the predicted spectrograms for each method. Table 2 presents the comparison results regarding the LSD on high-frequency and full range (denoted as HF and Full). It suggests that our proposed method obviously improves LSD. We achieve the best results among all other existing methods. It should be noted that T-CNN performs much worse than the F-DNN method. This conflicts with the conclusion in [7] which conducted the evaluation with speech samples. We believe this is because of the complex nature of polyphonic music, which contains overlapping spectrograms of vocals and various instruments instead of monophonic speech. Thus it is difficult to indirectly expand the frequency bandwidth through temporal interpolation.

We also evaluate the three strategies of phase estimation. This is done by implementing the introduced frequency-domain approaches and replacing FLIP methods with GLA and MelGAN, respectively. In detail, the following methods are evaluated:

• •

  F-DNN+GLA: Fully-connected network implementation for magnitude prediction(Li et al.[6]) and GLA phase method for phase estimation

• •

  F-DNN+MelGAN: Fully-connected network implementation for magnitude prediction (Li et al.[6]) with MelGAN phase method for phase estimation

• •

  F-CNN+GLA: CNN implementation[9] for magnitude prediction with GLA method for phase estimation

• •

  F-CNN+MelGAN: CNN implementation[9] for magnitude prediction with MelGAN method for phase estimation

• •

  F-GAN+GLA: GAN implementation[9] for magnitude prediction with GLA method for phase estimation

• •

  F-GAN+MelGAN: GAN implementation[9] for magnitude prediction with MelGAN method for phase estimation.

The LSD results shown in Table 3 suggests that the MelGAN method for phase estimation achieves the best result. Both the MelGAN and GLA methods consistently improve the LSD results over the FLIP method used in [6, 9]. Therefore, it is valid that a more accurate phase prediction can effectively solve the music SR tasks.