Multi-GradSpeech: Towards Diffusion-based Multi-Speaker Text-to-speech Using Consistent Diffusion Models

Text-to-Speech (TTS) systems that use deep learning rely on datasets with one or more speakers to train models that can produce high-quality speech. This approach consists of two key components: an acoustic model that converts text into frame-level acoustic features, such as mel-spectrograms, and a vocoder that converts these features into waveforms. The acoustic model plays an essential role in controlling factors such as prosody, speaker identity, and style, ultimately contributing to the naturalness and expressiveness of generated speech. While it is straightforward to train a single-speaker acoustic model to generate the target speaker’s voice, supporting multiple speakers is critical to creating an economical and efficient multi-speaker TTS system or as pre-trained models for fine-tuning to improve the performance of the generated speech. On the other hand, improving multi-speaker acoustic modeling is a widely researched topic due to the greater complexity of target data distribution in multi-speaker scenarios compared to single-speaker scenarios.

Researchers have long sought to investigate generative modeling approaches [1, 2, 3, 4] to improve multi-speaker acoustic modeling. Auto-regressive models, which include RNNs or Transformers, have been used as generative models for acoustic models [5, 6, 7] . However, these models suffer from issues such as accumulated error and exposure bias [8, 9] during the inference phase, which can reduce the quality of multi-speaker acoustic modeling. Beyond this, some studiess [10, 11, 12] have used non-autoregressive models optimized by L1 or L2 losses as acoustic models for multi-speaker acoustic modeling. However, these models often produce over-smoothed speech, resulting from the assumption that data noise is drawn from Laplace or zero-mean Gaussian distribution [13], which may not be accurate. In recent years, generative adversarial networks (GANs) and normalizing flows (NFs) have been applied as generative modeling approaches to address the over-smoothing problems in multi-speaker acoustic modeling [14, 15].

In recent years, diffusion models have emerged as a potent class of generative networks, particularly in the domain of image generation. These models have also been applied to acoustic modeling, producing impressive speech synthesis quality, as reported in recent researches [16, 17, 18]. However, current efforts primarily focus on single-speaker scenarios and the sampling drift problem of the diffusion models [19] have locked the full potential of these approaches for multi-speaker TTS scenarios due to more complex target data distribution. Specifically, these approaches suffer from quality degradation from single-speaker scenarios to multi-speaker scenarios when there is ample data available for the target speakers. Furthermore, over-smoothing persists in the generated results when target speaker data is scarce.

This paper proposes Multi-GradSpeech, a novel solution to address the challenges mentioned earlier. Multi-GradSpeech utilizes the Consistent Diffusion Model (CDM) [19] for generative modeling, which tackles the sampling drift problem during the inference phase to enhance the performance of diffusion-based multi-speaker acoustic modeling. The experiments conducted on a Mandarin multi-speaker dataset demonstrated that Multi-GradSpeech performs better than Grad-TTS [16] for diffusion-based multi-speaker modeling without the need for fine-tuning.

Basically diffusion models are a class of generative models that gradually add noise to real data through a forward process. While the reverse process which can be approximated using a neural network removes noise from a simple distribution. In the context of stochastic differential equations (SDE) [20], defining $p_{0}$ as the data distribution, the forward and backward processes can be driven by two SDEs respectively. The forward SDE is defined as:

$$dx_{t}=g(t)dB_{t},x_{0}\sim p_{0},x_{t}\sim N\left(x_{0},\sigma_{t}^{2}I_{d}\right).$$

(1)

and the backward SDE is defined as:

$$dx_{t}=-g(t)^{2}\nabla_{x}\log p\left(x_{t},t\right)dt+g(t)d\bar{B}_{t}$$

(2)

where $t\in[0,T]$, $\sigma_{t}$ is the noise schedule of the diffusion process, $B_{t}$ is a Brownian motion, $g(t)^{2}=\frac{d\sigma_{t}^{2}}{dt}$ and $p\left(x_{t},t\right)$ is the probability density of $x_{t}$. It is worth noting that for all backward SDEs there exists a corresponding deterministic process satisfying the following Ordinary Differential Equation (ODE):

$$dx_{t}=-\frac{1}{2}g(t)^{2}\nabla_{x}\log p\left(x_{t},t\right).\vspace{-5pt}$$

(3)

Since this ODE and the corresponding backward SDE share the identical marginal probability density, we can generate samples by solving the ODE using first-order Euler solver or other higher-order solvers. However, for both backward SDE and ODE, $\nabla_{x}\log p\left(x_{t},t\right)$ which is also known as the score function is intractable. Several studies proposed to approximate it by a time-conditional denoiser that is to say predicting the true data $x_{0}$ from corrupted data $x_{t}$. Specifically, we aim to use neural networks to learn the function $h$: $\mathbb{R}^{d}\times[0,1]\rightarrow\mathbb{R}^{d}$:

$$h(x,t)=\mathbb{E}\left[x_{0}\mid x_{t}=x\right]$$

(4)

where the expectation is over $x_{0}$ by running the backward process with the initial condition $x_{t}=x$. By leveraging Tweedie’s formula, we can establish the relationship between the score function and the denoiser $h$:

$$\nabla_{x}\log p(x,t)=\frac{h(x,t)-x}{\sigma_{t}^{2}}$$

(5)

Bringing Eq. (5) into Eq. (2) and Eq. (3) we can use the trained denoiser for sampling. Denoising score matching loss is commonly used to train h:

$$L_{DSM}=\mathbb{E}_{x_{0}\sim p_{0},x_{t}\sim N\left(x_{0},\sigma_{t}^{2}I_{d}\right)}\left\|h_{\theta}\left(x_{t},t\right)-x_{0}\right\|^{2},$$

(6)

where $\theta$ is a neural network.

Perfectly learning the real denoiser $h^{*}$ is commonly unattainable, which leads to the sampling drift challenge. Specifically, during the sampling stage $x_{t}$ is derived from an imperfect $h^{*}_{\theta}$, and each time step produces a drift with respect to the true distribution $p^{*}_{t}$, which ultimately leads to the generation of samples far from the true distribution due to error accumulation. To mitigate sampling drift, Daras et al. proposed Consistent Diffusion Model (CDM). In [19], a denoiser function h is said to be consistent if for all $t\in[0,T]$ and all $x\in\mathbb{R}^{d}$,

$$h(x,t)=\mathbb{E}_{h}\left[x_{0}\mid x_{t}=x\right]$$

(7)

where the expectation is over $x_{t}^{\prime}$ by solving Eq. (2) or Eq. (3) starting at $x_{t}$ with function $h$. Further, a consistency loss is added to the original loss for enforcing consistency property of the denoiser:

$L_{CDM}=\mathbb{E}_{x_{t}\sim p_{t}}\mathbb{E}_{t^{\prime}\sim\mathcal{U}[t-\epsilon,t]}\Bigg{[}\mathbb{E}_{\theta}\left[h_{\theta}\left(x_{t^{\prime}},t^{\prime}\right)\mid x_{t}=x\right]-h_{\theta}(x,t)\Bigg{]}^{2}/2$

(8)

where the innermost expectation is over $x_{t^{\prime}}$. To reduce the computation time of this loss function, $t^{\prime}$ is generally chosen to be close to $t$ and the backward process from $x_{t}$ to $x_{t}{\prime}$ is discretized into a small number of steps.

Although diffusion-based acoustic models, such as Grad-TTS, show good results in single-speaker TTS when adequate data is available, their performance in multi-speaker TTS still has room for improvement. In this study, we propose a CDM-based multi-speaker acoustic modeling approach, Multi-GradSpeech, to enhance the performance of multi-speaker acoustic models. The proposed approach comprises three modules: a prior encoder, an aligner, and a CDM-based decoder (shown in Fig. 1). We describe Multi-GradSpeech’s training and inference in detail.

We introduce Multi-GradSpeech, a diffusion-based acoustic model for multi-speaker modeling. By utilizing Consistent Diffusion Models, we demonstrate significant improvements in our approach compared to Grad-TTS. We anticipate that Multi-GradSpeech will serve as an alternative to prior diffusion-based acoustic models, particularly in large TTS.