Parallel WaveNet conditioned on VAE latent vectors

For the neural vocoder, we use a model inspired by Parallel WaveNet. This means we train two models using the Student-Teacher paradigm. The teacher is a WaveNet-like model which provides sequential sampling but parallel likelihood computation via teacher forcing which makes it fast to train but slow for inference. The student uses IAFs where the affine transform parameters are predicted by autoregressive conditioners built from WaveNet-like blocks of dilated convolutions. IAFs allow parallel sampling which makes them efficient during inference. But they require sequential calculations to estimate likelihood and are therefore slow to train. Training first the teacher and then the student to match the teacher’s outputs, results in a model that can be trained and sampled from efficiently.

The WaveNet-like models used for both student and teacher are convolutional networks which are conditioned on textual information via extra bias parameters in the convolutional layers. The original paper proposed the use of linguistic features, f0 and durations. Instead, we condition on mel-spectrograms. As observed in the original paper, Student-Teacher training alone leads to whispering artefacts. Hence, the authors proposed training on a combination of other losses: Power Loss, Phoneme Recognition Style Loss, and Contrastive Loss. We additionally propose conditioning both teacher and student models on the utterance-level latent vector extracted as described in section 2.2. The latent vector is concatenated with the output of the conditioning sub-network (see Figure 1).

To extract the utterance-level latent vector we use to condition the Parallel WaveNet models, we train an acoustic model with a Tacotron 2-like architecture [20] (see Figure 2). It uses a VAE reference encoder [21] to extract a latent representation from the target mel-spectrograms. This VAE reference encoder consists of a stack of convolutional layers followed by a bi-LSTM and two projections for the mean and standard deviation of an n-dimensional Gaussian distribution from which an n-dimensional latent vector is sampled.
Mel-spectrograms (extracted from recordings or predicted by an acoustic model) are input to the VAE reference encoder block to extract an utterance-level bottlenecked representation. We are effectively repurposing the VAE reference encoder trained alongside the acoustic model as a latent vector extractor that is used to condition the neural vocoder.

We train the acoustic model, the Parallel WaveNet teacher and student models on a diverse set of $63$ hours of data from a single female US speaker. It contains $46$ hours of audio in the traditional neutral style and additional $17$ hours of expressive data covering the following domains: news [22], long-form book reading¹¹1https://developer.amazon.com/en-US/blogs/alexa/alexa-skills-kit/2020/04/new-alexa-long-form-speaking-style-and-polly-voices, as well as excited and disappointed emotions²²2https://developer.amazon.com/en-US/blogs/alexa/alexa-skills-kit/2019/11/new-alexa-emotions-and-speaking-styles.

The acoustic model architecture is similar to [22] with the exception of replacing the two-dimensional style ID with a VAE reference encoder that outputs a $64$-dimensional latent vector. This places less constraints on the categories of speech samples since the representation is extracted from the signal rather than a manual label which is potentially inconsistent. The VAE latent space is learned in an unsupervised manner without providing explicit information about the different expressive domains. We follow [20] and increase the Kullback-Leibler divergence (KLD) loss factor from $0$ to $1$ between iterations $25$k and $150$k and additionally only take it into account every $200$ training steps to avoid the well-documented KL loss collapse. Once this model is converged, we use the VAE reference encoder to extract latent vectors for all the training data. These are used as additional conditioning for the training of the Parallel WaveNet teacher and student models.

The Parallel WaveNet teacher model uses a hidden size of $128$ for the two-layer bi-LSTM in the conditioning subnetwork. The output of this is concatenated with the $64$-dimensional VAE latent vector extracted from the acoustic model, before being upsampled to match the audio. The main block of the teacher model is a WaveNet architecture with $30$ layers of dilated convolutions where the dilation value is doubling every layer and reset to $2$ every $10$ layers. The Residual Gated CNN blocks use $256$-dimensional skip and gated channels, the filter activation is tanh and the gate activation is sigmoidal. The output is a mixture of $10$ logistics. The Parallel WaveNet student model shares the conditioning subnetwork with frozen weights with the teacher. The student is using $4$ affine transform flow layers conditioned using a WaveNet like architecture predicting scale and shift. The flow conditioners contain dilated convolutions with $10$, $10$, $10$ and $30$ layers. The last block uses same dilation value growth and reset as the teacher network. The dilated convolutions use $64$-dimensional gated channels with tanh filter activation and sigmoidal gate activation. We are training the vocoder using a batch size of $16$, the Adam optimizer with default parameters and learning rate decay of $0.95$ for the teacher and no decay for the student. We use Polyak averaging with decay of $0.999$.

To evaluate the quality of our proposed model, we conduct a preference test comparing it to a baseline. We use mel-spectrograms predicted by the acoustic model described in section 2.2 which also functions as the latent vector extractor. The baseline model is a Parallel WaveNet student as described in section 2.1, i.e. without additional conditioning, trained on the data described in section 3.1. Our proposed model differs from the baseline only in having the additional conditioning on the latent vector during training and inference. We evaluate $464$ utterances covering all expressive domains described in section 3.1 by asking $30$ native US speakers to ’Please choose the better sounding version.’. The in-domain results in Table 1 show a statistically significant preference for our proposed model ($p<0.05$ in a binomial test). The preference is found across all expressive domains and informal listening shows that more expressive utterances receive a stronger preference and are characterized by a ’cleaner’ sound (i.e. less distortion and buzzing).

We also evaluate our model on another set of data from the same speaker but from an expressive domain unseen during training. The mel-spectrograms are predicted using a separate acoustic model based on the same architecture described in 2.2. We evaluate $100$ utterances by asking $50$ native US speakers to ’Please choose the better sounding version.’. The out-of-domain results in Table 1 again show a statistically significant preference for our proposed model ($p<0.05$ in a binomial test).

To understand if the proposed approach scales to other voices, we train the same acoustic and Parallel WaveNet models on $10$ hours of expressive data of a male US voice. We apply our proposed technique and run a MUSHRA evaluation using 200 utterances comparing Parallel WaveNet with and without the additional conditioning. We ask 6 native US speakers to rate the systems in terms of their naturalness. The recordings are rated 97.48, while the model without extra conditioning received 47.70 and the model with extra conditioning 58.97. The p-values of pairwise two sided t-tests between all systems are 0. The additional conditioning on latent vectors improves the MUSHRA score by 11 points, showing that the approach can be scaled to other expressive voices. The gap between both synthesis systems and recordings is large since it is a very expressive voice with fairly limited amount of data resulting in less accurate predictions from the acoustic model. The results indicate that the additional conditioning can bring a larger uplift on overall lower quality voices.

To test our hypothesis, we perform an evaluation on spectrograms from natural speech rather than predicted ones. We focus on the long-form domain only and evaluate $100$ utterances from a book unseen during training, asking $30$ native US listeners to rate the systems on a scale from $1$ to $100$. We perform two tests, one asking the listeners to ’Please rate the systems in terms of their naturalness.’ and one asking them to ’Please listen to the reference sample on each testing screen and then rate how similar the systems sound to the reference audio.’ providing the recordings as reference audio. The results are shown in the top half of columns 1) and 2) in Table 2. For both questions, a pairwise two sided t-test shows no statistical difference between baseline and proposed system (p-value $>0.05$). The p-values between recordings and both systems are $0.0$. The evaluation on corresponding predicted spectrograms shows a statistically significant preference for the proposed system (see top half of column 3) in Table 2 and section 4.2 for more details).

These results indicate that the additional conditioning helps the model to produce better quality only when vocoding predicted spectrograms. To get a better understanding of the relation between natural and predicted speech in the latent space, we calculate the KLD between clusters as a measure of distances. We assume Gaussian distributions for the different clusters of expressive data (neutral, news, long-form book reading, emotions) and calculate the two-sided averaged KLD between all domain clusters. Table 3 shows that for long-form the natural and predicted clusters are closer to one another than the natural long-form cluster is to any other natural cluster and the predicted long-form cluster is to any other predicted cluster. The results for the other clusters are not shown but follow the same trend. This further supports our hypothesis that the latent vectors of predicted spectrograms carry additional information about corresponding natural spectrograms.

To better understand the importance of the latent vector conditioning during inference, we provide latent vectors other than the one extracted from the predicted spectrogram. We work with the same test set as in section 4.2 but with predicted spectrograms instead of ones extracted from natural speech. The predicted spectrograms are created using an acoustic model different from the one repurposed as the latent vector extractor but also based on the same architecture including a VAE reference encoder, trained on neutral and long-form data. We are comparing the following latent vectors:

• •

  Baseline: without additional conditioning.

• •

  Proposed: latent vector extracted from the spectrogram.

• •

  VAE long-form: latent vector mean of the distribution of training utterances in the long-form domain.

• •

  VAE neutral: latent vector within the distribution of the neutral domain of the training data.

• •

  VAE random: a random latent vector close to but outside of the training distribution.

• •

  VAE outside: a latent vector several magnitudes outside of the training distribution.

Figure 3 shows a two-dimensional PCA projection of the VAE latent space created by the latent space extractor. Blue Xs indicate the latent vectors used for conditioning. Grey points represent the latent vectors of neutral training data, light green training points in the long-form domain and light blue training points from other expressive domains. The brown and pink points indicate the extracted latent vectors for the utterances used in the evaluations; the recordings and their predicted counterparts respectively.

We evaluate the synthesis quality using a MUSHRA tests, asking $30$ native US listeners to’Please rate the systems in terms of their naturalness.’ on a scale from $1$ to $100$. Using the VAE outside vector, the synthesized audio only contained noise and therefore we did not include this in the evaluation. The results of the MUSHRA tests can be seen in column 3) of Table 2. The vocoders conditioned on latent vectors are rated higher than the baseline, by between $1$ and $1.5$ points. The p-values of a pairwise two sided t-test between the baseline and all VAE systems are $<0.05$, showing statistical significance of the results. The p-values between all VAE systems are $>0.05$, indicating that there is no difference in naturalness when conditioning on different latent vectors.

These results show that providing the additional conditioning during training helps to create a model that produces better quality when vocoding predicted spectrograms. Once the model is trained, the actual vector used for conditioning during inference does not seem to affect the quality. However, unreasonable points far outside of the training distribution completely destabilize the model and result in pure noise inference. This finding allows us to train an improved vocoder by providing additional conditioning during training while not having to introduce additional latency during inference for the computation of latent vectors. We are able to provide the mean of the training distribution and still get improved synthesis quality compared to the baseline.