The Academia Sinica Systems of Voice Conversion for VCC2020

To train the ASR model, we followed the Kaldi recipe²²2https://github.com/kaldi-asr/kaldi/tree/master/egs/librispeech of the LibriSpeech corpus [5]. The training process of our ASR system was divided into two parts, data processing, and acoustic modeling.

For data processing of the LibriSpeech corpus, we first extracted two MFCC features with different resolutions. The 13-dimensional MFCCs were used to train the GMM-based acoustic models. The 40-dimensional MFCCs were used to train the i-vector extractor and NN-based acoustic models. Then, we used the CMU pronunciation dictionary [6] to convert the English word transcriptions into the CMU pronunciation format, and mapped the CMU pronunciation symbols into the corresponding IPA symbols [7].

For acoustic modeling, the training set of the LibriSpeech corpus was first used to train the GMM-based acoustic models and the i-vector extractor. Following the model structure and training steps of the recipe, we created the alignment and lattice based on the GMM-based acoustic models, performed the data cleanup procedure, and extracted the 400-dimensional i-vectors to train the NN-based acoustic models. For the NN-based acoustic models, we selected the “chain” model structure (i.e., TDNN-F) [8, 9, 10]. The 40-dimensional MFCCs and the 400-dimensional i-vectors were concatenated as the input of TDNN-F.

To train the Tacotron2 TTS model [11], we followed the ESPNet recipe of the LibriTTS corpus³³3https://github.com/espnet/espnet/tree/master/egs/libritts/tts1 [12]. First, we extracted the 80-dimensional Mel-spectrogram and 512-dimensional x-vector from each utterance in the LibriTTS corpus. The speaker model used to extract the x-vectors was pre-trained with the Kaldi toolkit. As with the training of the Kaldi ASR system, the English word transcriptions were converted into the IPA symbols.

Following the model structure and training steps of the recipe, we obtained the multi-speaker Tacotron2 TTS model, which converts an IPA symbol sequence to the 80-dimensional Mel-spectrogram under the condition of a 512-dimensional x-vector. Lastly, we finetuned the Tacotron2 TTS model with the training data and average x-vector of the target speaker to obtain the speaker-dependent Tacotron2 TTS model. Note that utterance-dependent x-vectors were used to train the multi-speaker Tacotron2 TTS model, while the average x-vector of the target speaker was used to finetune the speaker-dependent Tacotron2 TTS model. In our preliminary experiments, this combination produced the best performance.

For waveform synthesis, we used ParallelWaveGAN [4] as the vocoder. To train the ParallelWaveGAN, similar to the ASR+TTS/ParallelWaveGAN baseline system (T22) [13] (which uses the cascaded seq-to-seq ASR+TTS (Transformer) model for VC and ParallelWaveGAN as the vocoder), we followed the open-source ParallelWaveGAN recipe⁴⁴4https://github.com/kan-bayashi/ParallelWaveGAN. We combined the VCTK corpus [14] and the training set of VCC2020, and extracted the 80-dimensional Mel-spectrogram as the input.

In the conversion phase, the 40-dimensional MFCCs and 400-dimensional i-vectors of each input utterance were first extracted for the Kaldi ASR system to output the IPA symbol sequence. Then, the Tacotron2 TTS model of the target speaker was used to convert the IPA sequence to the 80-dimensional Mel-spectrogram of the target speaker. Finally, the ParallelWaveGAN vocoder was used to convert the 80-dimensional Mel-spectrogram of the target speaker to the waveform.

Through the vector-quantization mechanism, VQVAE can quantize the latent vector representation obtained from the encoder into a discrete phonetic representation. The discrete phonetic representation is the index of the codeword closest to the latent vector in the codebook of the vector quantizer. In this way, the 80-dimensional Mel-spectrogram of an utterance is converted to a phoneme-like phonetic symbol sequence.

As shown in Figure 2 (cf. Step 1), to train VQVAE, we used the W-GAN mechanism with a gradient penalty to make VQVAE perform better. The loss function $L_{G}$ used to update the generator of VQVAE is as follows,

	$\displaystyle L_{G}=$	$\displaystyle\quad p(\bm{{x}}|\bm{{z}}_{q}(\bm{{x}}),\bm{{y}})$		(1)
		$\displaystyle+||\mathrm{sg}[\bm{{z}}_{e}(\bm{{x}})]-\bm{{e}}||^{2}_{2}+||\bm{{z}}_{e}(\bm{{x}})-\mathrm{sg}[\bm{{e}}]||^{2}_{2}$
		$\displaystyle-\mathbb{E}(\mathrm{D}(\mathrm{G}(\bm{{z}}_{q}(\bm{{x}}),\bm{{y}}))),$

where $\bm{{x}}$ denotes the input feature, $\bm{{y}}$ denotes the speaker code, $\bm{{z}}_{e}$ denotes the latent vector representation, $\bm{{e}}$ denotes the closest token of the latent vector representation, $\bm{{z}}_{q}(\bm{{x}})=\bm{{z}}_{e}(\bm{{x}})+\mathrm{sg}[\bm{{e}}-\bm{{z}}_{e}(\bm{{x}})]$ denotes the quantized discrete phonetic representation, $\mathrm{sg}$ denotes the stopping gradient, $\mathrm{G}$ denotes the decoder, and $\mathrm{D}$ denotes the discriminator. The loss function $L_{D}$ used to update the discriminator is as follows,

	$\displaystyle L_{D}=$	$\displaystyle-\mathbb{E}(\mathrm{D}(\bm{{x}}))+\mathbb{E}(\mathrm{D}(\mathrm{G}(\bm{{z}}_{q}(\bm{{x}}),\bm{{y}})))$		(2)
		$\displaystyle+\mathrm{gp}(\bm{{x}},\mathrm{G}(\bm{{z}}_{q}(\bm{{x}}),\bm{{y}})),$

where $\mathrm{gp}$ is the gradient penalty.

We used the ResNet architecture to form the encoder, the decoder, and the discriminator. In the ResNet architecture, as shown in Tables 1 to 3 , there are 4 res-layers, each containing 3 residual blocks. In the residual block of the decoder, we first concatenated the input quantized latent vector representation of size $B\times 128\times T$ with the speaker code of size $B\times 128\times T$ along the time axis as the new input feature, and additionally applied skip-connections, which early output skip features of size $B\times 80\times T$.

After training the VQVAE, we only retained the encoder and vector quantizer to extract the phonetic symbol sequence from the input speech in the subsequent steps.

To train the Transformer TTS model, we followed the ESPNet recipe of the LibriTTS corpus, but replaced the training corpus with a combination of the VCTK corpus and the VCC2020 training set. First, we extracted the 80-dimensional Mel-spectrogram and 512-dimensional x-vector from each utterance in the training corpus. The speaker model used to extract the x-vectors was pre-trained with the Kaldi toolkit. The phoneme-like phonetic symbol sequence of each training utterance was extracted using the VQVAE encoder and vector quantizer.

Following the model structure and training steps of the recipe, we obtained the multi-speaker Transformer TTS model, which convert the phoneme-like phonetic symbol sequence to the 80-dimensional Mel-spectrogram under the condition of a 512-dimensional x-vector. Lastly, we finetuned the Transformer TTS model with the training data and average x-vector of the target speaker to obtain the speaker-dependent Transformer TTS model.

In the conversion phase, the 80-dimensional Mel-spectrogram of each source utterance was first passed to the VQVAE encoder and vector quantizer to generate the phoneme-like phonetic symbol sequence. Then, the Transformer TTS model of the target speaker was used to convert the phoneme-like phonetic symbol sequence to the 80-dimensional Mel-spectrogram of the target speaker. Finally, the ParallelWaveGAN vocoder was used to convert the 80-dimensional Mel-spectrogram of the target speaker to the waveform.