IMPROVING PROSODY FOR UNSEEN TEXTS IN SPEECH SYNTHESIS BY UTILIZING LINGUISTIC INFORMATION AND NOISY DATA

Pre-training provides task-independent general knowledge, and fine-tuning can presumably teach the model to rely more on representations useful for the task at hand. For instance, BERT embeddings fine-tuned on the joint CWS and POS tagging task might acquire more knowledge about phrasing. Therefore, if the fine-tuned embeddings are used in TTS synthesis, the phrasing pattern of generated speech for sentences that are hard to segment is likely to improve. In addition, multi-task learning has demonstrated its effectiveness in sharing the knowledge obtained from several related supervised tasks. Thus with multi-task fine-tuning, BERT representations can gain richer linguistic or contextual knowledge.

During the fine-tuning process, BERT takes the final hidden vector h for the $i^{th}$ input token as the representation of the character $Tok\ i$. A simple FFNN (Feed-Forward Neural Work) with a softmax classifier is added to the top of BERT to predict the probability of label $c$:

$$p(c|h)=softmax(Wh)$$

(1)

where $W$ is the task-specific parameter matrix. We fine-tune all the parameters from BERT as well as $W$ jointly by maximizing the log-probability of the correct label. Specifically, we conduct single-task fine-tuning (BERT-Polyphone; BERT-CWS+POS; BERT-Prosody) and multi-task fine-tuning (BERT-Multi) based on the original character-based Chinese BERT-base model [15], which contains an encoder with 12 Transformer blocks, 12 self-attention heads, and a hidden size of 768.

BERT-Polyphone: BERT encoder with a prediction layer of the polyphone disambiguation task.

BERT-CWS+POS: BERT encoder with a prediction layer of the joint CWS and POS tagging task.

BERT-Prosody: BERT encoder with a predictor layer of the prosody structure prediction task.

BERT-Multi: BERT encoder with three prediction layers attached in parallel. BERT embeddings are fine-tuned in a multi-task learning framework.

We propose a novel fine-tuned BERT-based front-end, which not only performs G2P conversion but also provides linguistic features that contain rich contextual information by combining phoneme embeddings with character embeddings. Specifically, the fine-tuned BERT helps determine the pronunciation of polyphonic characters in the G2P conversion module. Then the converted phonemes are sent to an embedding layer to obtain the phoneme embeddings. Meanwhile, from the fine-tuned BERT we extract the character embeddings, which are then upsampled to match the length of the phonemic symbols of each character. Finally, the character embeddings and the phonemic embeddings are concatenated as the input features to the subsequent layers.

The training data used for this study is the Chinese Standard Mandarin Speech Corpus (CSMSC) [17]. CSMSC has 10,000 recorded sentences read by a female speaker, with the total audio length of about 12 hours of natural speech. We randomly split the dataset into two parts: 9500 samples for training and 500 samples for testing. The clean data used for pre-training the duration predictor is the AISHELL-3 training set (around 60 hours long) [18], which is a high-fidelity multi-speaker Mandarin speech corpus. And the AISHELL-1 training set (around 150 hours) [19], which is primarily used for building speech recognition systems, is selected as the noisy data to pre-train the duration predictor.

We fine-tuned BERT on a text corpus mined from the Chinese Wikipedia, containing 81108 sentences. The corpus was manually annotated by trained linguists. For the joint segmentation and tagging task, 35 tags for word classes and 2 tags for a character’s position within a word were used, totaling 70 categories. For the prosody prediction task, three classes were used for the prosodic break of prosodic word (PW), prosodic phrase (PPH) and intonational phrase (IPH), respectively. The corpus was randomly partitioned into training and test sets with 71853 and 9255 sentences, respectively.

We conduct ablation studies to evaluate the effectiveness of combining a fine-tuned BERT-based front-end with a pre-trained FastSpeech2-based acoustic model in improving the prosody of synthesized speech. The overall speech quality is evaluated on the CSMSC test set using the mean opinion score (MOS) [22]. Ten native speakers were asked to rate the overall prosodic appropriateness of the synthesized speech. We compare the MOS of audio samples generated by our proposed model with other systems, including 1) GT, the ground-truth audios; 2) GT (Mel + MB-MelGAN); 3) FastSpeech 2; 4) w/ BERT: a BERT-based front-end; 5) w/ BERT-Multi: a fine-tuned BERT-based front-end. 6) and 7): clean/noisy means the pre-trained duration predictor with AISHELL3/AISHELL1 dataset. 8) w/ BERT-Multi + noisy.

The results suggest that using a fine-tuned BERT-based front-end and a pre-trained FastSpeech2-based acoustic model can improve the prosody (with 0.14 and 0.06 MOS gains, respectively), as shown in Table 2. In addition, the w/ clean model and the w/ noisy model achieve comparable performance, suggesting that pre-training the encoder and the duration predictor on noisy data is feasible and can effectively improve the data efficiency and prosody at the same time.

We further conduct an ABX test to compare the models pairwisely. Synthesized audio samples from two different systems were presented to the same ten raters. And they were asked to select the one with more appropriate prosody. The evaluation text set contains 150 sentences and each sentence has more than 10 characters. These texts are randomly selected from the test set of the fine-tuning BERT experiments.

The results are summarized in Table 3. All fine-tuned BERT-based models outperform the models without fine-tuning and listeners preferentially choose the audios synthesized by BERT-Multi, suggesting that rich contextual information contained in fine-tuned BERT representations can be injected into the model. The w/ clean and the w/ noisy model perform fairly close, which suggests that pre-training on noisy data or clean data can bring about comparable improvements.

To investigate if the proposed method can generate better prosody, we select a structurally complex sentence for case study, as shown in Figure 3. Based on our linguistic knowledge, there is supposed to be a relatively noticeable pause between the verb of the main clause and the following subordinate clause in the object clause structure. The illustration of generated spectrograms indicates that the w/ BERT-Multi architecture generates speech with more appropriate breaks than the w/ BERT model and FastSpeech 2.