Self-supervised Context-aware Style Representation for Expressive Speech Synthesis

Assume a text dataset $\mathcal{D}=\left\{{U_{i}}\right\}_{i=1}^{D}$ where $U_{i}=\{{C_{i}^{-}},u_{i},{C_{i}^{+}}\}$ represents text utterance $u_{i}$ and its context. $C_{i}^{-}=\{u_{i-m},...,u_{i-1}\}$ is the preceding utterances of $u_{i}$ and $C_{i}^{+}=\{u_{i+1},...,u_{i+m}\}$ is the following utterances of $u_{i}$. Our goal is to learn a style encoding model $s_{i}=g(U_{i})$ from $\mathcal{D}$ which can generate a context-aware style representation for utterance $u_{i}$. This style model will be applied to a TTS system to improve the expressiveness of speech synthesis.

Figure 1 shows our proposed framework. First, we construct positive pairs of $u_{i}$ and its augmented sample $\tilde{u}_{i}$ by replacing the words having the strongest emotion arousal by their synonyms. Based on the data, we design a style encoder and pre-train it via contrastive learning. Second, to optimize the global distribution of our style representation, we further enhance the improved Deep Embedded Clustering method [18] with contrastive learning to train our style encoder further. Through the two stages, we learn $g(U_{i})$ and denote the generated representation as Context-aware Augmented Deep Embedded Clustering (CADEC) style. Finally, we feed it into Transformer TTS as conditioning embedding to generate expressive audio by applying appropriate style to text.

An utterance could be expressed by human in various styles. The appropriate style of utterance $u_{i}$ is highly related to context, its semantic content and conveyed emotion. We propose taking $u_{i}$ and its context together, i.e. $U_{i}$, as input and combine both content feature and emotion feature to model the best-fit style.

We employ a pretrained BERT [23] as backbone to extract content features, and an extra emotion lexicon [24] to extract emotion features. The emotion lexicon starts from a manually annotated English source emotion lexicon. Combining emotion mapping, machine translation, and embedding-based lexicon expansion, the monolingual lexicons for 91 languages with more than two million entries for each language are created. The lexicon provides word-level emotion features including VAD (valance, arousal, dominance) on 1-to-9 scales and BE5 (joy, anger, sadness, fear, disgust) on 1-to-5 scales. Then, we extract our initial style embedding $r_{i}$ by:

$$r_{i}=b(U_{i})\oplus\frac{1}{M}\sum_{j=1}^{M}{e(w_{j})}$$

(1)

where $\oplus$ denotes a concatenation operator, $b(U_{i})$ is the output [CLS] embedding by inputting $U_{i}$ into BERT, $M$ is the total number of words in $U_{i}$ and $w_{j}$ is $j$-th word in $U_{i}$ while $e(w_{j})$ denotes its normalized BE5 feature which is a 5-dimensional vector.

Then we add a fully connected multilayer perceptron (MLP) as encoder to map the initial embedding into hidden features, which are our output style embedding:

$$h_{i}=MLP(r_{i})$$

(2)

We propose augmenting data and using contrastive learning to pre-train the parameters of encoder.

To augment $u_{i}$ to the utterance $\tilde{u}_{i}$ that would have similar speech style, we first split $u_{i}$ into shorter segments not longer than a fixed length, e.g., 10 in our experiments. Then we look up the emotion lexicon to get emotion arousal for each word in a segment and select top $k$%, e.g., 20%, to be replaced by their WordNet synonyms [25]. Take the utterance $u_{i}$ in Figure 1 as an example. We split it into two segments, and select “lucky” and “happy” in the first segment and “annoying” and “crazy” in the second segment. We then replace them with their synonyms to compose $\tilde{u}_{i}$. The aim of splitting a long sentence into segments is to extract emotional words from different segments, thereby avoiding focusing on the dominant emotional words from some segment only. For example, although “fortunate” has higher arousal than “annoying” in the whole sentence of 20 words, we avoid choosing it for the whole sentence by the segment-based selection. This makes our concentrated emotional words more evenly distributed to ensure the expressiveness of the whole sentence.

As for contrastive learning, from a large training dataset $\mathcal{D}$, we randomly sample a minibatch data $\mathcal{B}=\left\{{U_{i}}\right\}_{i=1}^{N}$, and generate its augmented data $\mathcal{\tilde{B}}=\{\tilde{U}_{i}\}_{i=1}^{N}$, where $\tilde{U}_{i}=\{C_{i}^{-},\tilde{u}_{i},C_{i}^{+}\}$. $U_{i}$ and $\tilde{U}_{i}$ are treated as positive pairs while the other $N-1$ pairs $\{<U_{i},\tilde{U}_{k}>\}_{i\neq k}$ are all negative examples in one minibatch. To maximize the agreement between texts with similar emotions and disagreement between texts with different emotions, following simCLR [11], we calculate the sample-wise contrastive loss by

$$\mathcal{l}_{c}^{i}=-log{\frac{exp(cos(h_{i},\tilde{h_{i}})/{\tau})}{\sum_{k=1}^{N}\mathbbm{1}_{k\neq i}exp(cos(h_{i},\tilde{h_{k}})/{\tau})}}$$

(3)

Here $\tau$ is the temperature parameter and $\mathbbm{1}_{k\neq i}$ is the indicator function. The contrastive loss for a minibatch is computed by averaging over all instances in $\mathcal{B}$ and its augmented data $\tilde{\mathcal{B}}$:

$$\mathcal{L}_{contrastive}=\frac{1}{N}{\sum_{i}^{N}\mathcal{l}_{c}^{i}}$$

(4)

As the significant overlap of initial representation, this stage proves useful as the start of Stage 2 in our experiments.

To optimize the global distribution of style representations, we apply deep embedded clustering with autoencoder to train the CADEC style encoder further. The number of clusters K is a prior and each cluster is represented by its centroid $\mu_{k}$. Clustering loss is defined as

$$\mathcal{L}_{clustering}=KL(P\|Q)=\sum_{i}\sum_{k}p_{ik}log\frac{p_{ik}}{q_{ik}}$$

(5)

where $P$ is the target distribution of $Q$. Following  [26], we apply the Student’s t-distribution to compute the probability of assigning $h_{i}$ to the $k^{th}$ cluster $q_{ik}$.

$$q_{ik}=\frac{{(1+{\|{h_{i}-\mu_{k}}\|}_{2}^{2}/\alpha)}^{-\frac{\alpha+1}{2}}}{\sum_{k^{{}^{\prime}}=1}^{K}(1+{\|{h_{i}-\mu_{k^{{}^{\prime}}}}\|}_{2}^{2}/\alpha)^{-\frac{\alpha+1}{2}}}$$

(6)

where $\alpha$ denotes the degree of freedom of the Student’s t-distribution. In this work, we set $\alpha=1$. The target distribution $p_{ik}$ is

$$p_{ik}=\frac{q_{ik}^{2}/\sum_{i}{q_{ik}}}{\sum_{k^{{}^{\prime}}}(q_{ik^{{}^{\prime}}}^{2}/\sum_{i}{q_{ik^{{}^{\prime}}}})}$$

(7)

As this kind of clustering distorts the original space of representation and weakens the representation ability of implicit feature  [18], we add an autoencoder structure with reconstruction loss as in Equation  8, in order to preserve local structure of feature space and avoid corruption:

$$\mathcal{L}_{reconstruction}=\sum_{i=1}^{N}{{\|r_{i}-r_{i}^{{}^{\prime}}\|}_{2}^{2}}$$

(8)

Therefore, the objective of Stage 2 is defined as

$$\mathcal{L_{total}}=\mathcal{L}_{contrastive}+\beta\mathcal{L}_{clustering}+\gamma\mathcal{L}_{reconstruction}$$

(9)

We tune all trainable parameters to optimize $\mathcal{L_{total}}$ and obtain the final style encoder of our CADEC style.

Transformer TTS [27] leverages Transformer-based encoder-attention-decoder architecture. It performs well in synthesizing high-quality audio in fast speed. We extend Transformer TTS architecture by conditioning on CADEC style embedding generated from our proposed style encoder, with phoneme sequence and its style representation as inputs, as shown in Figure 1. Due to space limitation, we refer the reader to the paper [27] for more details about conditioned Transformer TTS.

To train and evaluate CADEC style encoder and TTS model, we use two datasets respectively, a plain text dataset for CADEC style encoder and an audiobook dataset with (text, audio) pairs for Transformer TTS model.

• •

  Text-training set and text-evaluation set. For CADEC encoder, we collect a plain text dataset from e-books. It contains 1.7M conversation utterances with contexts. We split the dataset into three sets, 1,696,000 conversation utterances as text-training set, 2000 utterances as in-domain text-evaluation set for validation and 2000 utterances as out-of-domain text-evaluation set for testing.  ²²2It should be mentioned that for supervised baseline approach training and accuracy evaluation, all conversations utterances are annotated into five emotion categories (calm, joy, serious, fear and depressed) by crowdsourcing, but we do not use these annotations during the CADEC style encoder training.

• $\bullet$

  TTS-training set and TTS-evaluation set. For TTS model, we apply an internal audio corpus as TTS-training set, read by a female speaker and a male speaker in Chinese. It is a complete 14 hours of expressive storytelling book, in which a portion of about 4 hours is conversation audio. Following most previous TTS work, we set in-domain test set from same audiobook and out-of-domain test set respectively to evaluate our model’s performance in different domains. We random select 10,981 utterances as training set, 200 utterances as in-domain TTS-evaluation set and 500 utterances from other books as out-of-domain TTS-evaluation set. Specifically, to evaluate voice quality of different genders, out-of-domain TTS-evaluation set contains both female and male speakers.

We use a supervised system that performed well in industrial as baseline. It contains two components: style prediction model and multi-style TTS model. The style prediction model (denoted as BERT-style) fine-tunes a pretrained BERT ³³3https://huggingface.co/bert-base-chinese with a downstream emotion classification task by leveraging annotated plain text dataset in a supervised manner. The annotated style tag is also applied when training baseline’s multi-style based Transformer TTS model.

In our proposed self-supervised style learning TTS framework, we use the same BERT backbone with content length of 256 as input in the same way as baseline BERT-style. In Stage 1 and Stage 2, we adopt Adam optimizer with mini-batch size of 32 and the initial learning rate as 1e-6. We firstly train 1,000 epoches with the contrastive loss to fine-tune BERT and autoencoder’s encoder in Stage 1 and continue to optimize all modules by minimizing $\mathcal{L_{total}}$ in Stage 2 until convergence. By Grid Search, we set convergence threshold as 0.1%, and both the coefficient $\beta$ and $\gamma$ of the clustering loss as 0.5. For multi-style TTS system training, narrative utterance’s style embedding is zero-initialized in our proposed method. Both the baseline and our proposed method use similar settings including the acoustic feature as 80-dimensional log mel-spectrogram, window shift 12.5 ms. A MelGAN vocoder [28] of 24kHz is used for both.

To evaluate the effectiveness of style embedding in TTS, we conduct subjective listening tests on Microsoft UHRS crowdsourcing platform. Participants are required to focus on speech style and expressiveness in all tests. For sentence-level speech, both MOS (Mean of Opinion Score) and CMOS (Comparative MOS) are conducted on in-domain TTS-evaluation set and out-of-domain TTS-evaluation set. Each audio is judged by at least 10 participants. As shown in Table 1, compared with the baseline, our model achieves better voice quality in terms of both MOS and CMOS, and importantly without any explicit annotation labels. The improvement in different datasets also demonstrates the effectiveness and robustness of our proposed method to cross-domain datasets of different genders.

In the audiobook scenario, a paragraph CMOS is used to evaluate the style expressiveness of multiple continuous utterances, as a paragraph or a session, in a conversation with narrative context. The judges are asked to rate audio considering context, which reflects the coherence and appropriateness of the audios’ expressiveness. We randomly select 20 paragraphs from out-of-domain TTS-evaluation set for each speaker with synthesized speech exceeding 30 seconds. The result shows significant preference over baseline, i.e. 0.22 for female and 0.23 for male speakers respectively. It verifies our assumption that style representation would be a continuous embedding other than a simple tag especially in a long context. Our model can express much more appropriate and diverse styles according to context, and achieves natural emotion transition between sentences.

Compared with BERT-style that requires predefined categories, CADEC style embedding achieves relatively lower accuracy in emotion classification tasks (Table 2). In further analyzing text and human-annotated emotion labels, we find that five discrete labels can not model complex context-aware emotion representation, which leads to low classifier accuracy. However, as our proposed model performs prominently better than the baseline in TTS evaluation experiments, it demonstrates that continuous embedding is more suitable for TTS tasks. Meanwhile, removing Stage 1 leads to a significant drop in accuracy (Table 2) and implicit embedding space overlap (Figure 2(c)). This ablation study shows the importance of Stage 1 and further demonstrates that our proposed CADEC style embedding is effective in learning styles in addition to content. Moreover, removing text’s context reduces the overall accuracy by 7.18% which demonstrates the importance of context in style modeling. More experiment results and audio samples could refer to https://wyh2000.github.io/InterSpeech2022/