An Analysis of the Variance of Diffusion-based Speech Enhancement

The goal of speech enhancement (SE) is to retrieve the clean speech signal from a noisy mixture that has been affected by environmental noise  [1]. Traditional methods attempt to leverage the statistical relationships between the clean speech signal and the environmental noise [2]. Various machine learning techniques have been suggested, treating SE as a predictive learning task, as seen in [3, 4].

Diverging from predictive approaches, which establish a direct mapping from noisy to clean speech, generative approaches focus on learning a prior distribution over clean speech data. Recently, a category of generative models known as diffusion models (or score-based generative models) has been introduced to the realm of SE [5, 6, 7, 8]. The concept involves gradually adding Gaussian noise to the data through a discrete and fixed Markov chain, referred to as the forward process, thereby transforming the data into a tractable distribution like a Gaussian distribution. Subsequently, a neural network is trained to reverse this diffusion process in a so-called reverse process [9]. As the step size between two discrete Markov chain states approaches zero, the discrete Markov chain transforms into a continuous-time stochastic differential equation (SDE) under mild constraints. The use of SDE provides greater flexibility and opportunities compared to methods based on discrete Markov chains [10]. Notably, SDEs enable the application of general-purpose SDE solvers for numerically integrating the reverse process, thereby influencing performance and the number of iteration steps. An SDE can be viewed as a transformation between two specified distributions, with one designated as the initial distribution and the other as the terminating distribution. For the context of score-based generative models for speech enhancement (SGMSE+) [8], recently different SDEs [8, 11, 12] have been introduced, with the initial distribution being the clean speech data and the terminating distribution being centered around the noisy mixture. Hence, these SDEs can be thought of as a stochastic interpolation between the clean speech signal and the noisy mixture. Within mild constraints, it is possible to identify a reverse SDE for each forward SDE, effectively inverting the forward process [13, 14]. This reverse SDE starts from a noisy mixture and ends at an estimate of the clean speech.

Various SDEs [8, 11, 12] with different variance and mean evolutions have been recently introduced for diffusion models for the task of SE. The resulting performances and characteristics of these SDEs vary, posing an open question regarding whether the primary contributing factor to performance disparities lies in the mean evolution or the variance evolution. For instance, the variance preserving schedule in [12] outperforms the variance exploding schedule from [8] with fewer reverse steps needed to solve the reverse SDE. Similar results have been found when comparing the SDE in [11] to the SDE in [8].

To address this open question, our paper aims to analyze the different SDEs regarding their variance schedules. We show that the scale of the variance schedule is a decisive factor in the performance of enhancement. More precisely, we show that a larger variance scale increases the amount of noise attenuation in the enhanced signal at the cost of the speech component quality. Vice versa, a lower variance scale results in less noise attenuation, but a better speech component quality. Moreover, we also show that a larger variance allows using fewer reverse steps when solving the reverse SDE, hence, reducing the computational cost of generating the enhanced signal from the noisy mixture. Finally, we experimentally show that different SDEs can perform virtually the same in terms of PESQ when their variance scales are properly optimized.

The task of SE is to estimate the clean speech signal $\mathbf{S}$ from a noisy mixture $\mathbf{Y}=\mathbf{S}+\mathbf{N}$, where $\mathbf{N}$ is environmental noise. All variables in bold are the coefficients of a complex-valued short-time Fourier transform (STFT), e.g. $\mathbf{Y}\in\mathbb{C}^{d}$ and $d=KF$ with $K$ number of STFT frames and $F$ number of frequency bins. Following [7, 8], we model the forward process of the diffusion model with an SDE defined on $0\leq t<T_{\text{max}}$:

where $\mathbf{w}$ is the standard Wiener process [15], $\mathbf{X}_{t}$ is the current process state with initial condition $\mathbf{X}_{0}=\mathbf{S}$, and $t$ a continuous diffusion time-step variable describing the progress of the process ending at the last diffusion time-step $T_{\text{max}}$. The functions $\mathbf{f}(\mathbf{X}_{t},\mathbf{Y})$ and $g(t)$ are called drift and diffusion coefficient, respectively. The diffusion coefficient $g$ regulates the amount of Gaussian noise that is added to the process, and the drift $\mathbf{f}$ affects mainly in the case of linear SDEs the mean of $\mathbf{X}_{t}$ (see [15, (6.10)]). The process state $\mathbf{X}_{t}$ follows a Gaussian distribution [16, Ch. 5], called the perturbation kernel:

By Anderson [13], each forward SDE as in (1) can be associated to a reverse SDE:

where $\mathrm{d}{\bar{\mathbf{w}}}$ is a Wiener process going backwards in time. In particular, the reverse process starts at $t=T$ and ends at $t=0$. Here $T<T_{\text{max}}$ is a parameter that needs to be set for practical reasons, as the last diffusion time-step $T_{\text{max}}$ is only reached in limit. The score function $\nabla_{\mathbf{X}_{t}}\log p_{t}(\mathbf{X}_{t}|\mathbf{Y})$ is approximated by a neural network called score model $s_{\theta}(\mathbf{X}_{t},\mathbf{Y},t)$, which is parameterized by a set of parameters $\theta$. Assuming that $s_{\theta}$ is available, we can generate an estimate of the clean speech $\mathbf{X}_{0}$ from $\mathbf{Y}$ by solving the reverse SDE.

Originally in [7], it was proposed to use a drift term for the SDE resulting in a mean evolution interpolating between clean and noisy signals. Following this line of research, other SDEs [11, 12] interpolating between clean and noisy signals were proposed for the task of SE. To analyze the SE performance of trained score models with different underlying SDEs, here we focus on the SDEs proposed by [8, 11].

The OUVE and BBED SDEs differ in many aspects which can be observed from Fig. 1, where we plotted the mean evolutions of OUVE and BBED with shaded areas indicating the standard deviations. First, the variance evolutions are different, as the variance of OUVE is a strictly increasing function and the variance of BBED first increases and decreases again. This can be seen also in Fig. 1 as the standard deviation of the OUVE SDE (red shaded area) increases and the standard deviation of the BBED SDE (blue shaded area) vanishes at $t=0,T$. Second, the mean evolution of the OUVE SDE is exponential, whereas it is linear for BBED. In [11] the construction of BBED was motivated by reducing the prior mismatch. The prior mismatch is defined as the difference of the SDE's mean evolution $\bm{\mu}(T)$ to $\mathbf{Y}$. In [11] it was shown the BBED SDE achieves improved reconstructions over the OUVE SDE and that the BBED SDE has significantly lower prior mismatch than the OUVE SDE. The successful reduction of the prior mismatch by BBED can clearly seen in Fig. 1, where the red line does not approach $Y$ for $t=T=1$. However, in Section 5 we will show that the dominant factor for the improvement is not the reduction of the prior mismatch, but the amount of Gaussian noise injected during the reverse trajectory as controlled by the variance scale of the SDE. For this, we analyze the variance scale $c$ of the OUVE and BBED SDE and show that with the right choice of the variance scale $c$, the two SDEs perform virtually the same in terms of PESQ in our experiments. Specifically, we reveal the insights:

1. 1.

   When solving the reverse SDE for inference, an SDE parameterized with low variance will not remove as much background noise as an SDE parameterized with a larger variance. In addition, when the variance is too large then it will overaggressively remove energy from the noisy mixture and as a result, the speech component quality of the estimate degrades. This means that the amount of variance of the SDE controls the tradeoff between noise attenuation and speech component quality of the enhanced signal.

2. 2.

   An SDE parameterized with a larger variance can reduce the number of network evaluations when solving the reverse SDE for inference. This is because the step size for solving the reverse SDE can be increased, and the reverse starting point $t_{\text{rsp}}$ can be reduced to values lower than $T$. Both effects result in fewer steps for solving the reverse SDE and therefore speeding up the inference process.

3. 3.

   Even two very different SDEs can lead to similar overall performance for an optimized variance scale.

In Tab. 1, we report enhancement results of the trained score model using either BBED or OUVE for the diffusion process with scales $c$ as discussed in Section 4. For brevity, we write $\text{BBED}_{c}$ or $\text{OUVE}_{c}$ when we refer to BBED or OUVE with variance scale $c$. For generating the enhanced signal with BBED or OUVE in Tab. 1, we choose $t_{\text{rsp}}=T$ and $N=60$ reverse diffusion steps for the EuM method.

To tackle the first contribution point from Section 3, we observe from Tab. 1 that the speech component quality of the enhanced signal decreases when the variance scale is increased. More precisely, $\text{BBED}_{\text{0.01}}$ achieves $2.30$ in Speech-PESQ. Increasing the variance scale to $c=0.08$ and $c=0.51$ reduces this value to $2.21$ and $2.14$ in Speech-PESQ, indicating increased speech distortions. In contrast, we observe that the amount of noise attenuation increases when the variance scale increases. More concretely, $\text{BBED}_{\text{0.01}}$ achieves $20.9$ dB in NA, and BBED with larger variance scale $c=0.08$ and $c=0.51$ increases NA to a much larger value of $29.6$ dB and $33.0$ dB. Likewise, this tradeoff between noise attenuation and speech component quality can also be observed for OUVE in Tab. 1. Moreover, in Fig. 2 we see that the spectrogram produced by $\text{BBED}_{\text{0.51}}$ overaggressively removes energy from the clean speech signal, especially in the high frequencies. In contrast, $\text{BBED}_{\text{0.01}}$ preserves the energy of the clean speech signal but also does not perform as much denoising as $\text{BBED}_{\text{0.51}}$. A reasonable tradeoff between $\text{BBED}_{\text{0.01}}$ and $\text{BBED}_{\text{0.51}}$ is obtained by $\text{BBED}_{\text{0.08}}$.

For the second contribution point from Section 3, we argue that an SDE with a larger variance scale is more robust against errors in the reverse trajectory than an SDE with a lower variance scale, as it may help to disguise these errors with a large amount of Gaussian noise. We now show that this is the case when the error source is the discretization of the reverse SDE or the prior mismatch. First, we discuss how changing the variance scale $c$ affects the performance when the discretization error is increased. To increase the discretization error, we decrease $N$ from 60 to 30 (or equivalently $\Delta t$ increases from $1/60$ to $1/30$) and fix $t_{\text{rsp}}=T=0.999$. Thus, for Fig. 3 we used the same trained models as in Tab. 1 but with a fixed discretization step size of $\Delta t=1/30$ instead of $\Delta t=1/60$. From Tab. 1 we show that $\text{BBED}_{\text{0.01}}$ with $N=60$ achieves $2.03$ in PESQ. However, in Fig. 3, we show that $\text{BBED}_{\text{0.01}}$ with $N=30$ achieves only $1.83$ in PESQ. Therefore, we conclude that when the variance scale $c=0.01$ is relatively low, enhancement performance is sensitive when the discretization error increases. Conversely, we show from Fig. 3 that $\text{BBED}_{\text{0.51}}$ with $N=30$ achieves $2.45$ in PESQ which is similar to $2.43$ when $\text{BBED}_{\text{0.51}}$ uses $N=60$ reverse steps in Tab. 1. Hence, we conclude that an SDE with a higher variance scale is less sensitive to discretization errors than an SDE with a lower variance scale. Second, we show that an SDE with a larger variance scale remains more robust when the prior mismatch increases. To increase the prior mismatch, we lower $t_{\text{rsp}}$ with fixed $\Delta t=1/30$. Our experiments in Fig. 3 verify the claim as $\text{BBED}_{\text{0.51}}$ (thick blue curve) remains much more stable than $\text{BBED}_{\text{0.01}}$ (dotted blue curve) when lowering $t_{\text{rsp}}$.

Finally, we observe that the two different SDEs achieve similar performance on the DNS4 dataset when the variance is suitably scaled by comparing $\text{BBED}_{\text{0.08}}$ and $\text{OUVE}_{\text{0.18}}$ in all evaluated metrics in Tab. 1. Our finding indicates that BBED from [11] outperformed OUVE from [8] in terms of PESQ because the variance of $\text{BBED}_{\text{0.08}}$ in [11] was simply much larger than the variance of $\text{OUVE}_{\text{0.18}}$ (see Fig. 1). While in [8] $\text{OUVE}_{\text{0.01}}$ achieved $2.92$ in PESQ on the VoiceBank-Demand [25], using the insights of this paper we here report that OUVE with an increased $c=0.08$ achieves $3.11$ in PESQ on the VoiceBank-Demand benchmark. This improvement is due to a higher noise attenuation at the cost of slightly increased speech distortions. Likewise, $\text{BBED}_{\text{0.08}}$ achieves $3.08$ in PESQ on the VoiceBank-Demand dataset.

This paper contributes to the understanding of diffusion models for speech enhancement by emphasizing the critical role of the variance scale of the Gaussian noise. Our findings reveal an interesting tradeoff, namely, that a larger variance scale removes more environmental noise at the cost of the speech component quality. Conversely, a smaller variance scale removes less environmental noise but increases in speech component quality. Moreover, we experimentally verified that a larger variance helps to reduce the effect of discretization errors and the prior mismatch. Therefore, when solving the reverse SDE fewer reverse steps are required, hence reducing the computational footprint. Last, we showed that with optimized variance scales, OUVE and BBED perform similarly, with OUVE reaching a PESQ of 3.11 on the VoiceBank-Demand benchmark.

The authors gratefully acknowledge the scientific support and HPC resources provided by the Erlangen National High Performance Computing Center (NHR@FAU) of the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU). The hardware is funded by the German Research Foundation (DFG).