PHASE-INCORPORATING SPEECH ENHANCEMENT BASED ON COMPLEX-VALUED GAUSSIAN PROCESS LATENT VARIABLE MODEL

Unlike in previous works [1, 2], in which prior knowledge about the instance of speech and noise is used to generate a mask, in this work, a binary mask is estimated without training. Noise power spectral density (PSD) is utilized to determine whether an STFT bin is reliable or not. As in Eq. (3), an element of the mask $\mathbf{M}$ is $1$, meaning that the corresponding SFTF bin is reliable. Otherwise, the STFT bin is unreliable. The masked spectra ${\mathbf{\widetilde{S}}}$ are then regarded as incomplete observations. Figure 1 displays an example of how speech is estimated using a missing data mask. As in a previous work [4], the goal here is to reconstruct speech spectra from the incomplete observations.

First, we investigate the feasibility and applicability of nonlinear probabilistic model, named GPLVM [6], for speech enhancement. In this subsection, the reconstruction is performed on magnitude spectrum. Each frequency band is independently regarded as a GP. GPLVM [6] is utilized to learn the T-F pattern of clean speech. Given training frames $\mathbf{Y}=[\mathbf{y}_{1},...,\mathbf{y}_{QT}]\in{\mathbb{R}^{F\times QT}}$ which comprise $Q$ clean speech spectrograms, each frequency band $\mathbf{Y}_{f}\in{\mathbb{R}^{QT}}$ can be modelled as

$$\mathbf{Y}_{f}=g_{f}(\mathbf{Z})+\bm{\epsilon}_{f}$$

(7)

where $\mathbf{Z}=[\mathbf{z}_{1},...,\mathbf{z}_{QT}]\in\mathbb{R}^{K\times QT}$ with $K\ll F$, $\bm{\epsilon}_{f}{\sim}{\mathcal{N}}(\mathbf{0},{\beta^{-1}}{\mathbf{I}})$. The mappings $g_{f},f=1,...,F$ are drawn from an independent and identically distributed (i.i.d.) GP, i.e. $g_{f}(\mathbf{Z}){\sim}{\mathcal{N}}(\mathbf{0},\mathbf{K})$, where $\mathbf{K}$ is a covariance matrix in which the element of the $n$-th row and the $m$-th column is determined by a kernel function, $[\mathbf{K}]_{nm}=k(\mathbf{z}_{n},\mathbf{z}_{m})$, $n,m\in\left\{1,...,QT\right\}$. The form of a nonlinear mapping depends on the choice of a kernel function. For example, a radial basis function (RBF) kernel $k({\mathbf{z}}_{n},{\mathbf{z}}_{m})=\theta_{1}\exp(-\theta_{2}\left\|{{\mathbf{z}}_{n}-{\mathbf{z}}_{m}}\right\|^{2})$, where $\theta=\{\theta_{1},{\rm{}}\theta_{2}\}$ are hyperparameters in the model, yields a smooth mapping. The marginal likelihood of $\mathbf{Y}$ can be calculated as

	$\displaystyle p(\left.{{\mathbf{Y}}}\right|{\mathbf{Z}})=$	$\displaystyle{\prod_{f=1}^{F}}\int{p(\left.{{\mathbf{Y}}_{f}}\right|{\mathbf{g}}_{f})}p(\left.{{\mathbf{g}}_{f}}\right|{\mathbf{Z}})d{\mathbf{g}}_{f}$		(8)
	$\displaystyle=$	$\displaystyle{\prod_{f=1}^{F}}\mathcal{N}(\left.{{\mathbf{Y}}_{f}}\right|{\bf{0}},{\mathbf{K}}+{\beta}^{-1}\mathbf{I})$

where ${\mathbf{g}}_{f}=g_{f}(\mathbf{Z})\in{\mathbb{R}^{QT}}$. The hyperparameters $\theta$ and the low-dimensional representations $\mathbf{Z}$ can be estimated by maximizing Eq. (8) by the gradient descend method [6]. Accordingly, the spectral patterns of the clean speech are then stored in the kernel.

Given estimation of $\theta$ and $\mathbf{Z}$, incomplete observations $\mathbf{\widetilde{S}}$ can be enhanced by calculating the corresponding low-dimensional representation. To reconstruct the speech spectrogram, the standard GP prediction is utilized [17]. The reconstructed spectrogram is then combined with the noisy phase.

To incorporate the estimation of phase into the reconstruction, rather than modifying only the magnitude spectra, the complex-valued STFT coefficients are directly enhanced from the masked spectra ${\mathbf{\bar{S}}}={\mathbf{M}}\otimes{\mathbf{X}}\in{\mathbb{C}}^{F\times T}$. Let ${\mathbf{U}}=[\mathbf{u}_{1},...,\mathbf{u}_{QT}]^{\top}\in{\mathbb{C}^{F\times QT}}$ be the complex-valued STFT coefficients of $Q$ training data from clean speech signals. Similar to GPLVM, each frequency band ${\mathbf{U}}_{f}$ can be viewed as a complex GP. To learn the nonlinear mapping between the complex-valued spectrum and its low-dimensional representation, this work proposes the CGPLVM.

$$\mathbf{U}_{f}=h_{f}(\mathbf{V})+\mathbf{e}_{f}$$

(9)

where ${\mathbf{V}}=[\mathbf{v}_{1},...,\mathbf{v}_{QT}]^{\top}\in{\mathbb{C}^{K\times QT}}$, $\mathbf{e}_{f}$ has a complex Gaussian distribution ${\mathcal{CN}}(\mathbf{0},{\beta^{-1}}{\mathbf{I}},\mathbf{0})$ and $h_{f},f=1,...,F$ are drawn from an i.i.d. proper complex GP, so $h_{f}(\mathbf{V}){\sim}{\mathcal{CN}}(\mathbf{0},\mathbf{K}_{c},\mathbf{0})$. $\mathbf{K}_{c}$ is a kernel matrix that expresses the relationships among the complex-valued low-dimensional frames $\mathbf{v}_{1},...,\mathbf{v}_{QT}$. Based on the work of Boloix-Tortosa et al. [18], a kernel that is used in a complex GP framework can be defined as

	$\displaystyle k_{c}({\mathbf{v}}_{n},{\mathbf{v}}_{m})=$	$\displaystyle k_{rr}({\mathbf{v}}_{n},{\mathbf{v}}_{m})+k_{jj}({\mathbf{v}}_{n},{\mathbf{v}}_{m})$		(10)
		$\displaystyle+j(k_{rj}({\mathbf{v}}_{m},{\mathbf{v}}_{n})-k_{rj}({\mathbf{v}}_{n},{\mathbf{v}}_{m}))$

where $k_{rr}$, $k_{jj}$ and $k_{rj}$ are real kernel functions. In this work, a kernel that is the sum of an exponentiated quadratic kernel and a bias term is used.

$$k({\mathbf{v}}_{n},{\mathbf{v}}_{m})=\theta_{1}\exp(-\frac{({\mathbf{v}}_{n}-{\mathbf{v}}_{m})^{\mathrm{H}}({\mathbf{v}}_{n}-{\mathbf{v}}_{m})}{\theta_{2}})+\theta_{3}$$

(11)

where $\left(\cdot\right)^{\mathrm{H}}$ is the Hermitian matrix. The hyperparameters and the low-dimensional representations can be learned by maximizing the log marginal likelihood as in Eq. (8).

	$\displaystyle\ln p(\left.{\mathbf{U}}\right|{\mathbf{V}})$	$\displaystyle=-FQT\ln\pi-F\ln\left|{{\mathbf{K}}_{c}+\beta^{-1}\mathbf{I}}\right|$		(12)
		$\displaystyle-{\rm{trace}}(({{\mathbf{K}}_{c}+\beta^{-1}\mathbf{I}})^{-1}{\mathbf{UU}}^{\mathrm{H}})$

By introducing the low-dimensional representations $\mathbf{V}$ that are associated with the training spectra $\mathbf{U}$ and the $t$-th frame $\mathbf{\bar{s}}_{t}$ from the masked spectra ${\mathbf{\bar{S}}}$, the corresponding low-dimensional representation $\mathbf{\bar{v}}_{t}$ can be obtained by solving the equation,

$$\widehat{\mathbf{v}}_{t}=\underset{\bar{\mathbf{v}}_{t}}{\arg\max}\ln p(\mathbf{U},\mathbf{\bar{s}}_{t}\mid\mathbf{V},\mathbf{\bar{v}}_{t})$$

(13)

Finally, the $t$-th enhanced spectrum ${\breve{\mathbf{s}}_{t}}$ can be reconstructed using a predictive approach, which is given by ${\breve{\mathbf{s}}_{t}}=$ ${\mathbf{U}}^{\mathrm{H}}({{\mathbf{K}}_{c}+\beta^{-1}\mathbf{I}})^{-1}{\mathbf{k}}$, where ${\mathbf{k}}=[k_{c}({\mathbf{v}}_{1},{\widehat{\mathbf{v}}_{t}}),k_{c}({\mathbf{v}}_{2},{\widehat{\mathbf{v}}_{t}}),...,k_{c}({\mathbf{v}}_{QT},{\widehat{\mathbf{v}}_{t}})]^{\mathrm{T}}$.

In this work, the performances of the proposed methods when applied to the CHTTL database [19], were evaluated. The CHTTL database includes 100 speakers (50 males and 50 females) who said the numbers zero to nine consecutively in Mandarin only once. Each complete utterance lasted 5-6 seconds, and was sampled at 8 kHz. In the experiments herein, 60 speakers (30 males and 30 females) were randomly selected from the CHTTL database. Data from ten (five males and five females) of them were used as training data. White noise was added with SNRs of 5, 10, 15 and 20 dB to the utterances of the remaining 50 speakers.

The performance of the tested enhancement method was evaluated in terms of segmental SNR (SSNR) [20], as

$${\rm{SSNR}}=\frac{1}{T}\sum_{t=1}^{T}P\left\{10\log_{10}\frac{\left\|\mathbf{s}_{t}\right\|^{2}}{\left\|\mathbf{s}_{t}-\breve{\mathbf{s}}_{t}\right\|^{2}}\right\}$$

(14)

where $\mathbf{s}_{t}$ and $\breve{\mathbf{s}}_{t}$ represent the clean speech and the enhanced speech in the $t$-th frame. $P(x)=\min\left\{\max(x,\rm{-10}),\rm{35}\right\}$ confines the SNR in each frame to a perceptually meaningful range. The perceptual evaluation of speech quality (PESQ) [21] was used to measure the quality of speech.

The spectrograms were generated using a 512-point STFT with Hamming windows to transform the speech into the time-frequency domain ($F=$ 257). The windows were shifted relative to each other by one half of the window length to cause them to overlap.

The performance of the proposed method was compared with the following baselines.

• •

  SR: A method of Williamson et al. [1] was selected as the first baseline. The SR was operated on magnitude spectra and the noisy phase was used to resynthesize the estimated speech signal. Using the settings of [1], an overcomplete dictionary is formed by concatenating the spectra of clean speech with the sparsity set to 5. ($L=5$ in Eq. (5)). The maximum number of iterations was set to 50.

• •

  NMF: Another method of Williamson et al. [2] was selected as the second baseline. As in SR, only the magnitude was modified. Using the settings of [2], the number of vectors in the basis matrix was 80 and the Euclidean distance was used to calculate the reconstruction error. The maximum number of iterations was set to 40.

Notably, the above methods [1, 2] use a DNN-based mask to extract speech components. To ensure a fair comparison, the statistical model-based mask was utilized to generate the masked spectra.

In this experiment, the performance that is obtained using different values of the latent dimension ($K=$ 10, 20, 30, 40) is studied. The threshold was set to 0.95. Figure 2 shows the relevant experimental results. For the CGPLVM, with a high noise level (15 and 20 dB), a larger $K$ implies better performance and the highest SSNR is obtained at $K<40$ when the noise level is low (5 and 10 dB), perhaps because the speech components of the masked spectra are highly corrupted, reflecting the fact that the learned model with the highest latent dimension ($K=40$) was overfitting. Additionally, the GPLVM underperforms the CGPLVM except in the case of $K=10$.

For various values of threshold $c$, the proposed methods (GPLVM, CGPLVM) were compared with the baselines (SR, NMF) in terms of SSNR and PESQ. There are three values of the threshold ($c=$ 0.75, 0.85, and 0.95) were investigated. The latent dimension $K$ of the proposed methods was set to 30. Tables 1-3 present the experimental results. For all methods, the SSNR increases with the threshold perhaps because a higher threshold reflects the fact that the masked spectra contain less noise. In the other word, the masked spectra mainly consist of speech components that are consistent with the assumptions of the model. Experimental results reveal that the CGPLVM outperforms the baselines for various noise levels and thresholds.

Figure 3 displays the reconstructed spectrogram using the proposed methods and baseline methods. Due to limitations of space, the following experiments are not conducted using all values of threshold. The threshold $c$ was set to 0.95. The latent dimension $K$ of the proposed methods was set to 30. For the CGPLVM, we take the magnitude of the reconstructed complex-valued STFT coefficients. Referring to the spectrogram of the clean speech (Fig. 1 (a)), nonlinear model-based methods (GPLVM and CGPLVM) outperformed linear model-based methods (SR and NMF).

To demonstrate the superiority of the proposed methods, which jointly estimate the magnitude and phase of a speech, the PESQs that were obtained using the proposed methods and baselines were evaluated. The results in Figure 4 demonstrate that the CGPLVM achieved a better PESQ than the other methods which do not consider the phase information of a speech at any noise level.