Audio Dequantization for High Fidelity Audio Generation in Flow-based Neural Vocoder

In [23], authors note that optimizing the continuous model $p_{model}(y)$ on the dequantized data $y\sim p_{data}$ can closely optimize the discrete model $P_{model}(x)$ on the origianl data $x\sim P_{data}$ through Jensen’s inequality, which can be formulated as below:

	$\displaystyle\int p_{data}(y)\log p_{model}(y)dy$		(1)
	$\displaystyle=\sum_{x}P_{data}(x)\int_{[0,1)^{D}}\log p_{model}(x+u)du$		(2)
	$\displaystyle\leq\sum_{x}P_{data}(x)\log\int_{[0,1)^{D}}p_{model}(x+u)du$		(3)
	$\displaystyle=\mathbb{E}_{x\sim P_{data}}\left[\log P_{model}(x)\right]$		(4)

Since the uniform noise is bounded to $[0,1)^{D}$, values of audio samples have to be bounded to unsigned integer. For this purpose, we preprocess raw audio with nonlinear companding method called ’mu-law companding’[27]. This method can significantly reduce the range of audio samples while minimizing the quantization error. Redistribution equation of the method can be expressed as:

$\displaystyle\hat{x}=sign(x)(\frac{ln(1+\mu|x|)}{ln(1+\mu)})$

(5)

where $sign$ function represents sign of value x, and $\mu$ represents integers that x values are mapped. We set $\mu$ to $255$ to apply 8-bit mu-law companding. Then, we add random noise from uniformly distributed function formulated as $Unif(0,1)$.

We assume that companding audio with lossy compression can possibly produce noisy output. Therefore, we implement iw(importance-weighted) dequantization proposed in [24] to improve generation quality. In the paper, authors demonstrate that sampling noise multiple times can directly approximate the objective log-likelihood, which can lead to better log-likelihood performance. As a result, we define uniformly dequantized data $y_{u}$ as:

$\displaystyle y_{u}=\hat{x}+\frac{1}{K}\sum_{k=1}^{K}Unif(0,1)_{k}$

(6)

where $Unif(0,1)$ defines noise sampled uniformly from $[0,1)^{D}$. We set K to 10.

To compare the performance of model depending on iw dequantization, we refer to uniform dequantization with iw dequantization as Uniform_IW and only uniform deqauntization model as Uniform.

In flow-based neural vocoder, discrete data distribution is transformed into a spherical Gaussian distribution. In other words, dequantizing data distribution to normal distribution can be more optimal choice. With this though in mind, we formulate Gaussian dequantization motivated from logistic-normal distributions [28]. Random noise samples are generated from normal distribution formulated as $\mathcal{N}(\mu,\sigma^{2})$, where mean and variance are calculated from the given data batch. To properly implement in audio domain, we apply a hyperbolic tangent function to normalize noise boundary at $(-1,1)^{D}$. As a result, normally dequantized data $y_{n}$ can be formulated as:

$$\begin{split}y_{n}=x+\tanh(\mathcal{N}(M(x_{b})),\Sigma(x_{b}))\\ \end{split}$$

(7)

where $M(x_{b})$ and $\Sigma(x_{b})$ represent mean and variance of batch group $x_{b}$.

To compare model performance between conventional and improved method, we refer to the conventional method suggested in [28] as Gaussian_Sig and proposed method as Gaussian_Tanh.

Instead of adding noise from known distribution, noise distribution can be formulated through a neural network such as a flow-based network. Flow++ [25] suggests that if the noise samples $u$ are generated from conditional probability model $q(u|x)$, probability distribution of original data can be estimated as follows:

$\displaystyle P_{model}(x):=\int_{[0,1)^{D}}q(u|x)\frac{p_{model}(x+u)}{q(u|x)}du$

(8)

Then, we can obtain the variational lower-bound on the log-likelihood function by applying Jensen’s inequality as below:

	$\displaystyle\mathbb{E}_{x\sim P_{data}}\left[\log P_{model}(x)\right]$		(9)
	$\displaystyle=\mathbb{E}_{x\sim P_{data}}\left[\log\int_{[0,1)^{D}}q(u|x)\frac{p_{model}(x+u)}{q(u|x)}du\right]$		(10)
	$\displaystyle\geq\mathbb{E}_{x\sim P_{data}}\left[\int_{[0,1)^{D}}q(u|x)\log\frac{p_{model}(x+u)}{q(u|x)}du\right]$		(11)
	$\displaystyle=\mathbb{E}_{x\sim P_{data}}\mathbb{E}_{u\sim q(u|x)}\left[\log\frac{p_{model}(x+u)}{q(u|x)}\right]$		(12)

As a result, dequantized data $y_{p}$ from variational dequantization can be defined as:

$\displaystyle y_{p}=x+q_{x}(\epsilon)$

(13)

where $\epsilon\sim p(\epsilon)=\mathcal{N}(\epsilon;0,I)$.

To implement variational dequantization in flow-based neural vocoder, we modify flow model from FloWaveNet[21]. We set initial input as 1-dimensional noise vector generated from spherical Gaussian distribution $\mathcal{N}(\epsilon;0,I)$, where the length is equal to target audio. In each context block, single squeeze step and multiple flow steps are operated. In each flow step, an affine transformation conditioned on target audio is applied to the half of squeezed vector. At the end, the vector is flattened, and hyperbolic tangent function is applied to fit range of audio domain. Negative log likelihood from the dequantizer is trained jointly with the flow-based neural vocoder.

We set a total of 16 flow stacks as Flow_Shallow and 48 flow stacks as Flow_Dense to examine whether the dept of dequantization model is critical to the model performance.

We set FloWaveNet[21] as our baseline model and trained baseline with 6 different dequantization methods. Each model was trained with VCTK-Corpus[29] containing 109 native speakers English dataset. Since FloWaveNet and WaveGlow[20] evaluated with only a single speaker dataset, we expanded the experiment on the model to a multimodal case where audio generation is much harder due to the larger variation among different speakers. In the dataset, we withdrew some corrupted audio files and used 44,070 audio clips. For each speaker, 70% of data were used as training data, 20% as validation data, and rest of them as test data. All clips were down-sampled from 48,000Hz to 22,050Hz. From each audio clip, 16,000 chunks were randomly extracted.

All models were trained on 4 Nvidia Titan Xp GPUs with a batch size of 8. We used Adam optimizer with a step size of $1\times 10^{-3}$ and set the learning rate decay in every 200K iterations with a factor of 0.5. We trained each model for 600K iterations.

For subjective evaluation, we conducted a subjective 5 scale MOS test on Amazon Mechanical Turk¹¹1https://www.mturk.com/. Each participant was suggested to wear either earbuds or headphones for the eligible testing. Then they had to listen to 5 audio clips at least twice and rated naturalness of audio on a scale of 1 to 5 with 0.5 point increments. We explicitly instructed the participants to focus on the quality of audio. We collected approximately 3,000 samples for the evaluation. In the Table 1, models implemented audio dequantization show higher MOS than the baseline model which show that audio dequantization can improve audio quality. Except for the real audio, variational dequantization with a deeper layer receives the highest MOS. This shows that injecting noise with more complex distribution can produce more natural audio.

Audio generated from the baseline model tended to have digital artifacts such as reverberation, trembling sound, and periodic noise. We assumed that the occurrence of these artifacts was due to the unnatural collapsing from the continuous density model on discrete data-points. To prove that audio dequantization can remove such artifacts, we conducted several quantitative evaluations in signal processing to compare the quality of audio. First we randomly selected 400 sentences in test set. Then, we conducted mel-cepstral distortion (MCD)[30], global signal-to-noise ratio (GSNR)[31], segmental signal-to-noise ratio (SSNR)[11], and root mean square error of fundamental frequency (RMSE_f0)[11]. All equations for the evaluation can be calculated as follows:

	$\displaystyle MCD_{13}[dB]=\frac{1}{T}\sum_{t=0}^{T-1}{\sqrt{\sum_{k=1}^{K}{(c_{t,k}-c_{t,k}^{{}^{\prime}})^{2}}}}$		(14)
	$\displaystyle GSNR[dB]=10log\frac{\sigma_{s}^{2}}{\sigma_{r}^{2}}$		(15)
	$\displaystyle SSNR[dB]=10log_{10}(\frac{\sum_{n=0}^{M}x_{s}(n)^{2}}{\sum_{n=0}^{M}(x_{s}(n)-y_{r}(n))^{2}})$		(16)
	$\displaystyle RMSE_{f0}[cent]=1200\sqrt{(log_{2}(F_{r})-log_{2}(F_{s}))^{2}}$		(17)

where $c_{t,k}$, $c_{t,k}^{{}^{\prime}}$ represent original and synthesized k-th mel frequency cepstral coefficient (MFCC) of t-th frame, $\sigma_{s}^{2}$, $\sigma_{r}^{2}$ represent power of speech signal and noise, $x_{s}(n)$, $y_{r}(n)$ represent raw and synthesized waveform sample at time n, and $F_{r}$, $F_{s}$ represent fundamental frequency of raw and synthesized waveform.

In MCD, we compared all models including Uniform and Gaussian_Sig to see the improvement within the modification. In Table 2, dequantizations with modified methods show better performance than the conventional methods. Gaussian_Tanh and Flow_Dense score relatively lower MCD than baseline, which shows that both models can produce better audio quality than the baseline model. There was no significant performance difference between the two variational dequantization methods. Uniform_IW shows slightly higher MCD than the baseline because of the remaining audible noise generated from mu-law companding.

Table 3 presents the result of SNR and RMSE_f0. All proposed methods show higher SNR than baseline, indicating that audio dequantization can help reducing noise. In addition, Uniform_IW and Gaussian_Tanh dequantization show better performance in modeling fundamental frequency, while Flow_Dense dequantization shows a comparable result with the baseline model. We also visualized test outputs for qualitative evaluation in Figure 2. Figure 2(f) and Figure 2(h) show clearer harmonic structures than Figure 2(b). Although Figure 2(d) shows less clear harmonic structures than other approaches, we can see that periodic noises are reduced significantly. We provide audio results on our online demo webpage.²²2https://claudin92.github.io/deqflow_webdemo/