An empirical study on speech restoration guided by self-supervised speech representation

Self-supervised learning methods aim to learn useful data representations without relying on human annotations. Recently, these methods have demonstrated remarkable results on various speech-related tasks such as automatic speech recognition [16], speaker recognition [15], and emotion recognition [14]. One prominent example is the work by [14], where the model is trained to predict the pseudo-labels of masked regions using unmasked speech parts. By leveraging this pre-training task, the model learns to capture the global structure and long-term dependencies of speech signals. Another noteworthy approach is WavLM [15], a variant of HuBERT [20] that jointly learns masked speech prediction and denoising during pre-training. This method has achieved remarkable performance on the SUPERB challenge [21].

We expect that incorporating contextualized information into speech restoration, particularly in recovering speech components lost due to various degradation factors such as clipping, limited bandwidth, and attenuation, will be beneficial since training objectives in [15, 16, 22] encourage models to capture the global structure of the signal [17, 18]. We define the degraded speech signal $x$ as $x=f\left(s\right)+n\in{\mathbb{R}^{T}}$, where $f$ represents various degradation factors, and $s$ and $n$ are the original speech and noise signals, respectively. The goal of speech restoration is to find a mapping function $g:{\mathbb{R}^{T}}\to{\mathbb{R}^{T}}$ that transforms the degraded speech signal $x$ into a high-quality restored speech signal $\hat{s}$. To achieve this, we utilize the SSL representation as a conditioning signal for the speech restoration network $g$, and we formulate the problem as $\hat{s}=g\left({x|h\left(x\right)}\right)$, where $h$ represents the SSL network.

In speech enhancement, spectral mask estimation in the time-frequency domain is a common approach that attenuates the magnitude spectrogram using the estimated ratio mask but neglects the distorted phase information. This method is effective for noise and interference that are modeled as additive relationships. In [17], SSL representations are incorporated into this mask estimation method as shown in Figure  1(a). A simple bi-directional recurrent neural network (BLSTM) is designed where the SSL representation is injected before the BLSTM layer to integrate the degraded speech feature and its corresponding SSL feature. The model’s noise suppression performance has improved compared to previous models.

In recent years, deep learning has enabled the restoration of signals in the waveform domain. These methods typically involve a 1-dimensional convolution layer with a large kernel and stride to simulate spectral analysis in STFT, while a transposed convolution layer is used to synthesize acoustic embedding into the waveform. This approach reduces the artifacts caused by STFT, resulting in signals with improved perceptual quality.

In [19], the authors used SSL representations on the DEMUCS structure to improve the overall quality of speech enhancement, demonstrating superiority over phonetic representations. We applied the conditioning method from [19] to both DEMUCS and SE-Conformer models as shown in Figure  1(b). Both models have the same U-Net-like structure except for the Conformer bottleneck in SE-Conformer. SSL representations are incorporated with distorted speech features before the bottleneck for both models, as this approach showed the best performance among methodologies. In addition, we investigated the Conv-TasNet model for speech enhancement in the waveform, with several modifications for speech restoration based on [23]. In our approach, the SSL embedding is integrated with the inputs of the first convolution layer in every ConvBlock of Conv-TasNet, as illustrated in Figure  1(c), where a ConvBlock is comprised of a point-wise convolution, a depth-wise convolution, and a residual connection.

We used the VCTK-DEMAND dataset [24], which is a widely used benchmark for speech enhancement. The dataset comprises premixed noisy speech and includes a training set of 11,572 utterances from 28 speakers mixed with noise samples at 4 SNRs (0, 5, 10, and 15 dB). The test set contains 824 utterances from 1 male and 1 female speaker with 4 SNRs (2.5, 7.5, 12.5, and 17.5 dB). Speakers p286 and p287 were separated from the training set and used for validation as in [1]. We degraded the VCTK-DEMAND test set by simultaneously applying several speech degradation factors, which are described in the next subsection. We evaluated speech quality using PESQ¹¹1https://github.com/schmiph2/pysepm [25] and NISQA²²2https://github.com/gabrielmittag/NISQA [26], and intelligibility using STOI1 [27] with open-source toolkits.

We evaluated 4 types of speech degradation factors: background noise, audio clipping, limited bandwidth, and speech attenuation. All these factors were applied simultaneously in all evaluations, unless otherwise stated.

We trained all the compared models within our speech restoration framework, ensuring that our models obtained similar scores to those reported in [1, 17, 19] using the original configurations. In our experiments, we adjusted the depth ($L$) of the U-Net structure to 4 to align the temporal resolution with SSL representations and set the number of channels in the first convolution layer to 48. For the remaining model parameters, we followed the original configurations.

To utilize SSL representation, we used the pre-trained WavLM-Large model [15], which has shown good results among compared SSL methods in [18]. We followed the setup of SUPERB [21] and froze the parameters of the SSL model. We took the weighted average of representations from all transformer encoder layers with learnable weights to yield the final SSL embedding. To match the frame rates of SSL with those of the intermediate representation of the backbone network, we repeated frames. For conditioning with SSL, we simply concatenated the intermediate features with the SSL representation, as we found no significant difference compared to applying [28].