Improved Vocal Effort Transfer Vector Estimation for Vocal Effort-Robust Speaker Verification

Given a particular speaker, let $\tilde{\mathbf{x}}\in\mathbb{R}^{D}$ be a $D$-dimensional speaker embedding extracted from an utterance with normal vocal effort. Furthermore, let $\tilde{\mathbf{y}}\in\mathbb{R}^{D}$ be a non-neutrally-phonated counterpart of $\tilde{\mathbf{x}}$. Then, we assume the following additive model:

$$\tilde{\mathbf{y}}=\tilde{\mathbf{x}}+\tilde{\mathbf{v}},$$

(1)

where $\tilde{\mathbf{v}}\in\mathbb{R}^{D}$ represents a vocal effort transfer vector between the normal and non-neutrally-phonated modes.

For vocal effort-robust speaker verification purposes, previous work [4, 5] proposed to estimate $\tilde{\mathbf{x}}$ from an estimate of the vocal effort transfer vector, $\hat{\tilde{\mathbf{v}}}$, by following the additive model of Eq. (1):

$$\hat{\tilde{\mathbf{x}}}=\tilde{\mathbf{y}}-\hat{\tilde{\mathbf{v}}}.$$

(2)

Several MMSE compensation techniques where studied in [4, 5] to realize Eq. (2), and, among them, MEMLIN [6] showed to be the best performing one. Assuming that the non-neutrally-phonated embedding domain is modeled by a $K$-component Gaussian mixture model (GMM), MEMLIN —which also models the normal embedding space by another $K$-component GMM— approximates $\tilde{\mathbf{x}}$ from a weighted combination of $K$ partial estimates $\left\{\hat{\tilde{\mathbf{v}}}^{\{k\}};\;k=1,...,K\right\}$:

$$\hat{\tilde{\mathbf{x}}}=\tilde{\mathbf{y}}-\underbrace{\displaystyle\sum_{k=1}^{K}P(k|\tilde{\mathbf{y}})\hat{\tilde{\mathbf{v}}}^{\{k\}}}_{\mbox{$\hat{\tilde{\mathbf{v}}}$}},$$

(3)

where $\left\{P(k|\tilde{\mathbf{y}});\;k=1,...,K\right\}$ are the combination weights.

As introduced in Section 1, a limitation of MEMLIN is that, unlike the combination weights, the set of partial estimates $\left\{\hat{\tilde{\mathbf{v}}}^{\{k\}};\;k=1,...,K\right\}$ is independent of the observed non-neutrally-phonated embedding $\tilde{\mathbf{y}}$, which constrains the potentials of the Bayesian estimation framework²²2In MEMLIN, the set of partial estimates is pre-computed during an offline training stage [6].. In the next subsection, we propose a different MMSE compensation approach where also the partial estimates exploit $\tilde{\mathbf{y}}$, which yields significant speaker verification performance improvements in Section 4.

Inspired by classical noise-robust speech recognition methods such as front-end joint uncertainty decoding (FE-Joint) [8] and stereo-based stochastic mapping (SSM) [9], we explore jointly modeling the vocal effort transfer vector and non-neutrally-phonated embedding domains by means of a $K$-component GMM $p(\tilde{\mathbf{z}}=(\tilde{\mathbf{v}},\tilde{\mathbf{y}}))$. However, estimating $p(\tilde{\mathbf{z}}=(\tilde{\mathbf{v}},\tilde{\mathbf{y}}))$ requires the computation of $2D\times 2D$ covariance matrices that are ill-conditioned under our non-neutrally-phonated speech data scarcity scenario (see Subsection 3.1). To deal with this issue, we propose to use PCA as detailed below.

Let $\mathbf{W}_{L}$ be a $D\times L$ PCA transform matrix calculated from normal and non-neutrally-phonated embeddings. This matrix is comprised, column-wise, of $L$ principal eigenvectors, where $L\ll D$. We can express $\tilde{\mathbf{v}}$ and $\tilde{\mathbf{y}}$ in the PCA domain as

$$\mathbf{v}=\mathbf{W}_{L}^{\top}\tilde{\mathbf{v}},\hskip 28.45274pt\mathbf{y}=\mathbf{W}_{L}^{\top}\tilde{\mathbf{y}}.$$

(4)

Then, the joint variable $\mathbf{z}=(\mathbf{v},\mathbf{y})$, $\mathbf{z}\in\mathbb{R}^{2L}$, is modeled by a $K$-component GMM:

$$p\left(\mathbf{z}\right)=\displaystyle\sum_{k=1}^{K}P(k)\mathcal{N}\left(\mathbf{z}\left|\boldsymbol{\mu}_{z}^{\{k\}},\boldsymbol{\Sigma}_{z}^{\{k\}}\right)\right..$$

(5)

In Eq. (5), $\left\{P(k);\;k=1,...,K\right\}$ is the set of prior probabilities, whereas the mean vector $\boldsymbol{\mu}_{z}^{\{k\}}$ and covariance matrix $\boldsymbol{\Sigma}_{z}^{\{k\}}$ of the $k$-th Gaussian density can be partitioned as

$$\boldsymbol{\mu}_{z}^{\{k\}}=\left(\begin{array}[]{c}\boldsymbol{\mu}_{v}^{\{k\}}\\ \boldsymbol{\mu}_{y}^{\{k\}}\end{array}\right),\;\;\;\boldsymbol{\Sigma}_{z}^{\{k\}}=\left(\begin{array}[]{cc}\boldsymbol{\Sigma}_{vv}^{\{k\}}&\boldsymbol{\Sigma}_{vy}^{\{k\}}\\ \boldsymbol{\Sigma}_{yv}^{\{k\}}&\boldsymbol{\Sigma}_{yy}^{\{k\}}\end{array}\right),$$

(6)

where all $\boldsymbol{\Sigma}_{vv}^{\{k\}}$, $\boldsymbol{\Sigma}_{vy}^{\{k\}}=\left(\boldsymbol{\Sigma}_{yv}^{\{k\}}\right)^{\top}$ and $\boldsymbol{\Sigma}_{yy}^{\{k\}}$ are $L\times L$ diagonal matrices.

Given Eq. (5), the MMSE estimate of $\mathbf{v}$, $\hat{\mathbf{v}}$, is calculated as follows:

$$\begin{array}[]{lll}\hat{\mathbf{v}}&=&\mathbb{E}(\mathbf{v}|\mathbf{y})=\displaystyle\int_{\mathbf{v}}\mathbf{v}p(\mathbf{v}|\mathbf{y})d\mathbf{v}\vskip 4.26773pt\\ &=&\displaystyle\sum_{k=1}^{K}\int_{\mathbf{v}}\mathbf{v}p\left(\mathbf{v},k|\mathbf{y}\right)d\mathbf{v}\vskip 4.26773pt\\ &=&\displaystyle\sum_{k=1}^{K}P(k|\mathbf{y})\int_{\mathbf{v}}\mathbf{v}p(\mathbf{v}|\mathbf{y},k)d\mathbf{v}\vskip 4.26773pt\\ &=&\displaystyle\sum_{k=1}^{K}P(k|\mathbf{y})\underbrace{\mathbb{E}\left(\mathbf{v}\left|\mathbf{y},k\right)\right.}_{\mbox{$\hat{\mathbf{v}}^{\{k\}}$}},\end{array}$$

(7)

where $\mathbb{E}(\cdot)$ denotes the expectation operator. On the one hand, the combination weights in Eq. (7) are obtained, by means of the Bayes’ rule, according to

$$P(k|\mathbf{y})=\frac{p(\mathbf{y}|k)P(k)}{\sum_{k^{\prime}=1}^{K}p(\mathbf{y}|k^{\prime})P(k^{\prime})},\;\;\;\;\;k=1,...,K,$$

(8)

where $p(\mathbf{y}|k)=\mathcal{N}\left(\mathbf{y}\left|\boldsymbol{\mu}_{y}^{\{k\}},\boldsymbol{\Sigma}_{yy}^{\{k\}}\right)\right.$. On the other hand, given that the joint density $p(\mathbf{z}=(\mathbf{v},\mathbf{y})|k)$ is Gaussian, the conditional density $p(\mathbf{v}|\mathbf{y},k)$ is also Gaussian, and, therefore, $\mathbb{E}\left(\mathbf{v}\left|\mathbf{y},k\right)\right.$, i.e., the partial estimates in Eq. (7), can be expressed, $\forall k\in\{1,...,K\}$, as [10]

$$\mathbb{E}\left(\mathbf{v}\left|\mathbf{y},k\right)\right.=\boldsymbol{\mu}_{v}^{\{k\}}+\boldsymbol{\Sigma}_{vy}^{\{k\}}\left(\boldsymbol{\Sigma}_{yy}^{\{k\}}\right)^{-1}\left(\mathbf{y}-\boldsymbol{\mu}_{y}^{\{k\}}\right).$$

(9)

Finally, an estimate of the normal embedding $\tilde{\mathbf{x}}$ is achieved by means of Eq. (2) along with the application of the inverse PCA transform to the result of Eq. (7), namely,

$$\hat{\tilde{\mathbf{x}}}=\tilde{\mathbf{y}}-\underbrace{\mathbf{W}_{L}\hat{\mathbf{v}}}_{\mbox{$\hat{\tilde{\mathbf{v}}}$}}.$$

(10)

Note that, in order to apply this method in Section 4, both the PCA transform matrix $\mathbf{W}_{L}$ and the GMM $p(\mathbf{z})$ are calculated from a training set comprising paired normal and non-neutrally-phonated embeddings (see Subsection 3.1).

For the sake of reproducibility, a Python implementation of this speaker embedding compensation methodology has been made publicly available³³3https://ilopezes.files.wordpress.com/2023/06/mmsev.zip.

For experimental purposes, we consider the vocal effort modes shouted and whispered in addition to normal. To this end, we employ two different (i.e., disjoint) corpora: the speech corpus informed in [12], which comprises paired shouted-normal speech utterances in Finnish from 22 speakers, and CHAINS (CHAracterizing INdividual Speakers) [13], which contains paired whispered-normal speech utterances in English from 36 speakers. Due to speech data scarcity, all the embedding compensation experiments in Section 4 are performed —as in [5]— by following a leave-one-speaker-out cross-validation strategy, which serves to split the corpora into training and test sets.

We consider the following 4 test conditions (trial lists) under the shouted-normal scenario: A${}_{\mbox{s}}$-A${}_{\mbox{s}}$ (all shouted and normal utterances vs. all shouted and normal utterances; 557,040 trials), N${}_{\mbox{s}}$-N${}_{\mbox{s}}$ (normal utterances vs. normal utterances; 139,128 trials), S-S (shouted utterances vs. shouted utterances; 139,128 trials) and N${}_{\mbox{s}}$-S (normal utterances vs. shouted utterances; 278,784 trials). Furthermore, we similarly examine 4 equivalent test conditions under the whispered-normal scenario, namely, A${}_{\mbox{w}}$-A${}_{\mbox{w}}$ (2,821,498 trials), N${}_{\mbox{w}}$-N${}_{\mbox{w}}$ (705,078 trials), W-W (704,950 trials) and N${}_{\mbox{w}}$-W (1,411,344 trials).

For further details about these corpora, the reader is referred to [12, 13] and [5].

The used ECAPA-TDNN back-end was trained, employing the additive angular margin (AAM) loss [14], on an augmented version of the VoxCeleb2 [15] dataset to extract $D=256$-dimensional speaker embeddings. Considering an AAM loss margin of 0.2, first, WavLM —which was pre-trained on 94k hours of unlabeled speech data— was fixed and the ECAPA-TDNN parameters were trained for a total of 20 epochs. Second, WavLM and the ECAPA-TDNN back-end were jointly fine tuned for 5 epochs. Finally, by following the large margin fine-tuning strategy reported in [16], WavLM and the ECAPA-TDNN back-end were jointly trained for 2 more epochs by considering an AAM loss margin of 0.4. Notice that, for the sake of reproducibility, the model corresponding to this speaker verification system is publicly available⁴⁴4https://github.com/microsoft/unilm/tree/master/wavlm. The reader is referred to [3] for further information on this speaker verification system.

Tables 1 and 2 show speaker verification results in terms of EER under the shouted-normal and whispered-normal scenarios, respectively. The left part of these tables compare, when no embedding compensation is considered, the use of WavLM speech representations (as in Section 3), E-T+WavLM, with the use of traditional speech features, E-T+MFCC (note that E-T stands for ECAPA-TDNN). Specifically, the speaker verification system E-T+MFCC, which is publicly available⁵⁵5https://huggingface.co/speechbrain/spkrec-ecapa-voxceleb, employs 80-dimensional Mel-frequency cepstral coefficients [17]. In line with [3], we can see from these tables that E-T+WavLM generally outperforms E-T+MFCC. That being said, we can also observe that there is still a large room for improvement in the presence of vocal effort mismatch (all conditions except N${}_{\mbox{s}}$-N${}_{\mbox{s}}$ and N${}_{\mbox{w}}$-N${}_{\mbox{w}}$) that will be addressed by embedding compensation in the next subsections. Bear in mind that all the embedding compensation experiments in this section are carried out by employing E-T+WavLM as the baseline system.

Figure 2 plots the EER performance of the estimation methodology proposed in Section 2, MMSE${}_{\mbox{v}}$, as a function of the PCA dimension $L$. For comparison, these bar plots also show results from MEMLIN (applied in the PCA domain) as well as from an MMSE estimator equivalent to that of Section 2 that directly estimates the normal embedding $\tilde{\mathbf{x}}$ from $\mathbb{E}[\mathbf{x}|\mathbf{y}]$, MMSE${}_{\mbox{x}}$. From this figure, we can see that MEMLIN’s performance tends to drop when decreasing $L$ as a result of the information loss caused by PCA compression, which can be particularly harmful when the estimation relies on a small set of pre-computed and fixed partial estimates.

On the other hand, MMSE${}_{\mbox{v}}$ involves the computation of $2L\times 2L$ covariance matrices, $\boldsymbol{\Sigma}_{z}^{\{k\}}$, under a data scarcity scenario. Given our small sample size, reducing $L$ helps to achieve better-conditioned covariance matrices to be used in Eqs. (8) and (9). This, together with the fact that MMSE${}_{\mbox{v}}$ exploits the observed non-neutrally-phonated embedding $\tilde{\mathbf{y}}$ for partial estimate calculation, can explain why EER decreases up to $L=16$ for MMSE${}_{\mbox{v}}$ (see Figure 2). Keeping decreasing $L$ beyond this point harms speaker verification performance due to the information loss entailed by PCA compression.

In relation to MMSE${}_{\mbox{x}}$, an internal analysis revealed that estimating the normal embedding $\tilde{\mathbf{x}}$ from $\mathbb{E}[\mathbf{x}|\mathbf{y}]$ yields target and non-target score probability masses that are poorly separated as a result of compensated embeddings $\hat{\tilde{\mathbf{x}}}$ where the specific-speaker information is significantly distorted. Interestingly, we also observed that the vocal effort transfer vector $\tilde{\mathbf{v}}$ has a weak speaker-dependence. Therefore, estimating $\tilde{\mathbf{x}}$ as $\tilde{\mathbf{y}}-\hat{\tilde{\mathbf{v}}}$ according to MMSE${}_{\mbox{v}}$ better preserves the specific-speaker information contained in $\tilde{\mathbf{y}}$, which, in turn, leads to better-separated target and non-target score probability masses.

The right part of Tables 1 and 2 compare standard MEMLIN (i.e., without PCA) with MMSE${}_{\mbox{v}}$, MMSE${}_{\mbox{x}}$ and MEMLIN applied in the PCA domain (MEMLIN+PCA). Note that, in these tables, the three latter techniques process, after PCA application, $L=16$-dimensional embeddings. Under the shouted-normal scenario (Table 1), MMSE${}_{\mbox{v}}$ outperforms MEMLIN in the presence of vocal effort mismatch (i.e., in A${}_{\mbox{s}}$-A${}_{\mbox{s}}$, S-S and N${}_{\mbox{s}}$-S). Furthermore, while MEMLIN is on par with MMSE${}_{\mbox{v}}$ in A${}_{\mbox{w}}$-A${}_{\mbox{w}}$ under the whispered-normal scenario (Table 2), MMSE${}_{\mbox{v}}$ achieves in N${}_{\mbox{w}}$-W a 22.7% EER relative improvement with respect to MEMLIN which actually worsens the baseline system E-T+WavLM (as in the S-S condition).