Optimizing Shoulder to Shoulder: A Coordinated Sub-Band Fusion Model for Real-Time Full-Band Speech Enhancement

The overall diagram of the proposed approach is shown in Figure 1(a), which is comprised of three sub-networks, namely DSLB-Net, MBM-Net, and HBM-Net. As speech contains more harmonics and semantic information in the frequency range of 0 to 8 kHz, we first employ DSLB-Net to eliminate the noise and recover the clean complex spectrum in the LB regions. Inspired by the efficacy of decoupling-style SE methods [8, 9, 11, 12], we decouple the original complex spectrum estimation into spectral magnitude and phase optimization, and adopt a dual-stream network to collaboratively estimate the magnitude and residual complex components of the low-band (0-8 kHz) spectrum in parallel, which consists of a magnitude estimation network (ME-Net), a complex spectrum purification network (CP-Net), and a reconstruction module (RM), as illustrated in Figure 1(b), (c) and (d), respectively. The input features of ME-Net and CP-Net are denoted as $\lvert{{X}^{lb}}\rvert\in\mathbb{R}^{T\times F\times 1}$ and $X_{com}^{lb}=Cat({X}^{lb}_{r},{X}^{lb}_{r})\in\mathbb{R}^{T\times F\times 2}$, where $\left\{T,F\right\}$ denote the number of frames and that of frequency bins, respectively.

In ME-Net, given noisy low-band spectral magnitude $\left|X^{lb}\right|$, the network estimates a real-valued gain function $G^{lb}$, which aims at coarsely filtering out the dominant noise. Then the denoised LB spectral magnitude is coupled with the original noisy phase to obtain the coarse-estimated real and imaginary (RI) spectrum. As the supplementation, we leverage CP-Net to purify the spectral structures and also recover the phase information. Rather than explicitly estimating the complex spectrum from scratch, CP-Net is designed for residual mapping, which alleviates the overall burden of this network. Taking the above estimations as the input, we reconstruct the estimated spectrum via the proposed RM. The whole procedure is thus formulated as:

	$$\displaystyle\lvert\widetilde{X}^{lb_{me}}\rvert=\lvert{X^{lb}}\rvert\otimes G^{lb},$$		(1)
	$$\displaystyle\widetilde{X}^{lb_{me}}_{r}=|\widetilde{X}^{lb_{me}}\rvert\otimes\cos\left(\theta_{X^{lb}}\right),$$		(2)
	$$\displaystyle\widetilde{X}^{lb_{me}}_{i}=\lvert\widetilde{X}^{lb_{me}}\rvert\otimes\sin\left(\theta_{X^{lb}}\right),$$		(3)
	$$\displaystyle\widetilde{X}^{lb}_{r}=\widetilde{X}^{lb_{me}}_{r}+\widetilde{X}^{lb_{cp}}_{r},$$		(4)
	$$\displaystyle\widetilde{X}^{lb}_{i}=\widetilde{X}^{lb_{me}}_{i}+\widetilde{X}^{lb_{cp}}_{i},$$		(5)

where $\left\{\widetilde{X}^{lb_{cp}}_{r},\widetilde{X}^{lb_{cp}}_{i}\right\}$ denote the output residual RI components of CP-Net and $\left\{\widetilde{X}^{lb}_{r},\widetilde{X}^{lb}_{i}\right\}$ denote the final merged estimation of clean RI components. $G^{lb}$ and $\theta_{X^{lb}}$ denotes the estimated gain of ME-Net and the LB noisy phase, respectively. $\otimes$ is the element-wise multiplication operator.

Based on the fact that the frequency bands ranging from 8-24 kHz tend to contain less speech information, we further employ two light-weight sub-networks, namely MBM-Net and HBM-NET, as the noise suppressors for the middle-band and high-band, respectively. To reduce the computational burden of the network and evade the implicit compensation effect between magnitude and phase [13], we only consider the magnitude and retain the phase unaltered in these two bands. Besides, we model the correlations among LB, MB, and HB via the proposed interaction module, where the estimated LB features are employed to guide the spectrum recovery of MB and HB.

To be specific, the estimated LB spectral magnitude by DSLB-Net is fed into MBM-Net and HBM-Net along with the noisy MB and HB spectral magnitude. Taking MBM-Net as an example, as shown in Figure 1(e), we adopt two encoders to extract the magnitude features from noisy MB spectra and the pre-estimated LB spectra. After that, the features are concatenated together and fed into the mask block to derive the gain function, which aims to automatically learn to filter and preserve different areas of the guided LB feature. Then we sum the extracted FB feature and the filtered version from LB to yield the interacted representation and feed for the latter modules. For HB regions, likewise, the denoised LB and MB spectra are concatenated in the channel axis to obtain the guided input features. With the proposed interaction module, external knowledge from DSLB-Net can effectively propagate to MBM-Net and HBM-Net regions and gradually guide the spectrum recovery.

In summary, the operation stream of the middle- and high-band modeling can be formulated as:

	$$\displaystyle|\tilde{X}^{mb}|=\lvert{X^{mb}}\rvert\otimes\mathcal{G}^{mb}\left(|X^{mb}|,\lvert\widetilde{X}^{lb}\rvert;\Phi_{1}\right),$$		(6)
	$$\displaystyle|\tilde{X}^{hb}|=\lvert{X^{hb}}\rvert\otimes\mathcal{G}^{hb}\left(|X^{hb}|,\lvert\widetilde{X}^{lb}\rvert,\lvert\widetilde{X}^{mb}\rvert;\Phi_{2}\right),$$		(7)
	$$\displaystyle\tilde{X}^{mb}=|\tilde{X}^{mb}|\exp(j\theta_{X^{mb}}),\tilde{X}^{hb}=|\tilde{X}^{hb}|\exp(j\theta_{X^{hb}}),$$		(8)

where $|\tilde{X}^{mb}|$ and $|\tilde{X}^{hb}|$ denote the denoised MB and HB outputs, respectively. $\mathcal{G}^{mb}$ and $\mathcal{G}^{hb}$ denote the mapping functions of MBM-Net and HBM-Net with parameter set $\Phi_{\left(\cdot\right)}$. $\theta_{X^{mb}}$ and $\theta_{X^{hb}}$ denote the noisy MB and HB phase, respectively. Finally, we stack the estimated three sub-band estimation along the frequency axis to obtain the full-band spectrum. Note that we average the overlapped bands in the MB and HB regions.

The detailed architectures of DSLB-Net, MBM-Net, and HBM-Net are shown in Figure 1. Similar to [8, 9], we employ a classical convolutional encoder-decoder topology [14] for these sub-networks, where multiple temporal convolution modules (TCMs) and a causal adaptive hierarchical attention module (cAHAM) [11] are stacked in the bottleneck for sequence modeling. More specifically, the detailed structures of ME-Net and CP-Net in DSLB-Net are shown in Figure 1(b) and (c), in which ME-Net utilizes a magnitude encoder and a dual-path masking decoder and CP-Net utilizes a complex encoder and two decoders to recover both RI components.

Taking ME-Net as an example, the encoder consists of five downsampling blocks, each of which consists of a convolutional layer, a normalization layer, and PReLU, with kernel size being (2, 3) in the time and frequency axes except (2, 5) in the first block. The number of channels remains 64 by default, and the stride is set to (1, 2) to gradually halve the frequency size. To enable streaming inference, the features are normalized with cumulative layer norm (cLN) [15], where the statistics are adaptively updated in a frame-wise manner. The dual-path masking decoder consists of five symmetrical deconvolutional layers and a dual-path mask module, which is performed to obtain the magnitude spectral gain by a 2-D convolution and a dual-path tanh/sigmoid nonlinearity operation similar to [11].

Inspired by [16], four groups of squeezed TCMs (S-TCMs) are employed for sequence modeling, each of which stacks six S-TCMs with increasing dilation rates $d$ to obtain a large temporal receptive field, i.e., $d=\{1,2,4,8,16,32\}$. To reduce the computational burden, the parameter weights of S-TCMs are shared in ME-Net and CP-Net. Similar to our previous study [11, 17], we utilize a casual AHAM [11] to integrate all intermediate features and global hierarchical contextual information during sequence modeling, where the average pooling layer is adopted along the frequency axis and retain the time resolution unaltered to satisfy the real-time requirement. For MBM-Net and HBM-Net, we leverage a similar structure as ME-Net, which aims at filtering out the dominant noise in MB and HB regions, respectively.

In our SF-Net, we adopt a two-stage training pipeline to recover low-band and full-band spectra progressively. First, we pretrain DSLB-Net with MSE until convergence. The loss function involves both RI and magnitude estimation, given as:

	$$\displaystyle\mathcal{L}_{lb}^{RI}=\left\|\widetilde{X}^{lb}_{r}-S^{lb}_{r}\right\|_{F}^{2}+\left\|\widetilde{X}^{lb}_{i}-S^{lb}_{i}\right\|_{F}^{2},$$		(9)
	$$\displaystyle\mathcal{L}_{lb}^{Mag}=\left\|\lvert\tilde{X}^{lb}\rvert-\lvert S^{lb}\rvert\right\|^{2}_{F},$$		(10)
	$$\displaystyle\mathcal{L}_{lb}=\mu\mathcal{L}_{lb}^{RI}+(1-\mu)\mathcal{L}_{lb}^{Mag},$$		(11)

where $\mathcal{L}_{lb}^{Mag}$ and $\mathcal{L}_{lb}^{RI}$ denote the loss terms toward magnitude and RI components, respectively. $\lvert S^{lb}\rvert$ denotes the target LB spectral magnitude. $S^{lb}_{r}$ and $S^{lb}_{i}$ represent the RI components of target LB regions. With the internal trial, we empirically set $\mu=0.5$ in the following experiments.

In the second stage, we couple the pretained DSLB-Net with MBM-Net and HBM-Net, and train them jointly. The overall loss can be given by:

$$\displaystyle\mathcal{L}_{full}=\alpha\mathcal{L}_{lb}+\mathcal{L}_{mb}^{Mag}+\mathcal{L}_{hb}^{Mag}$$

(12)

where $\mathcal{L}_{mb}^{Mag}$ and $\mathcal{L}_{hb}^{Mag}$ denote the loss functions for MF-Net and HF-Net in the magnitude domain, while $\mathcal{L}_{full}$ represents the full loss function of the second stage. We empirically find that $\alpha=0.1$ suffices in our evaluation.

To verify the effectiveness of the proposed model, we conduct extensive experiments on two public full-band benchmarks, namely VoiceBank + DEMAND dataset [18] and the ICASSP 2022 DNS-Challenge (DNS-2022) dataset [19], where all the utterances are sampled at 48000 Hz.

VoiceBank + DEMAND: The dataset is a selection of the VoiceBank corpus [20] with 28 speakers for training and another 2 unseen speakers for testing. The training set consists of 11,572 mono audio samples, while the test set contains 824 utterances by 2 speakers (one male and one female). For the training set, all the audio samples are mixed with one of the 10 noise types, including two artificial noise processes and eight real noise recordings taken from the Demand database [21].

DNS-Chanllenge: We further train and evaluate our model on the DNS-2022 dataset¹¹1https://github.com/microsoft/DNS-Challenge, which consists of various clean speech, noise clips, and room impulse responses (RIRs) to simulate practical acoustic scenarios. For this dataset, we totally generate around 600 hours of noisy-clean pairs. To generate reverberant-noisy training data, we use 248 real and 60,000 synthetic RIRs from openSLR26 and openSLR28 datasets [22]. As only late reverberation degrades the speech quality/intelligibility [23], we preserve both anechoic and early reverberation components as the training target. During each mixing process, the clean speech is convolved with a randomly selected RIR, and is then mixed with the noise in the SNR range of $\left(-5\rm{dB},15\rm{dB}\right)$. We evaluate the model performance upon the DNS-2022 blind test set, which includes 859 real test clips.

The 20ms Hanning window is utilized, with 50% overlap between adjacent frames. To extract the features, 960-point FFT is utilized and 481-dimension spectral features are obtained. Due to the efficacy of power compression in both dereverberation and denoising tasks [24], we conduct the power compression toward the spectral magnitude while remaining the phase unaltered, and the compression factor is set to $\beta=0.5$. All the models are optimized using Adam ($\beta_{1}=0.9$, $\beta_{2}=0.999$) [25]. In the first stage, the initialized learning rate (LR) is set to 1e-3 for DSLB-Net. In the second stage, DSLB-Net is fine-tuned with LR of 1e-4, while LR is set to 1e-3 for MBM-Net and HBM-Net. The batch size is set to 16 at the utterance level. The processed samples are available online, where the source code will be released soon.²²2https://github.com/yuguochencuc/SF-Net

We first conduct ablation studies to investigate the effects of the proposed sub-band fusion strategy, dual-stream structure, cAHAM and the sub-band interaction module, including a) three one-stage full-band SE approaches, i.e., the magnitude estimation network (dubbed ME-Net (full)), the complex purification network (dubbed CP-Net (full)), and the dual-stream network (dubbed DS-Net (full)), b) three sub-band fusing-based methods, namely ME-SF-Net, CP-SF-Net and the proposed coordinated sub-band fusion model (SF-Net), c) SF-Net without the causal adaptive hierarchical attention module (-cAHAM) and sub-band interaction module (- Inter.). Due to the lack of PESQ evaluation for full-band speech, we downsample the outputs of all models at 16000 Hz and measure wide-band PESQ and STOI, while SSNR and SDR are measured on the full-band SE.

Quantitative results are presented in Table 1, one can have the following observations. First, among the one-stage full-band approaches, DS-Net dramatically outperforms other single-stream approaches in terms of wide-band and full-band SE performance. For example, DS-Net provides average 0.21, 1.2%, 3.38dB and 1.78dB improvements than CP-Net in PESQ, STOI, SDR and SSNR, respectively. This reveals the effectiveness of the proposed dual-stream SE topology in improving speech quality. Second, compared with the one-stage full-band SE, when employing the sub-band fusion strategy, considerable performance improvements in terms of both wide-band and full-band cases can be obtained. Besides, we also provide the model size and the number of multiply-accumulate operations (MACs) per second, as shown in Table 1. Note that although SF-Net suffers more parameters than DS-Net (full), it achieves fewer MACs and remarkably better performance. This indicates the effectiveness of the proposed sub-band splitting and fusion strategy. Finally, without using cAHAM and sub-band interaction module, consistent performance degradations in both wide-band and full-band speech are observed, which emphasize the significance of cAHAM and sub-band interaction in improving speech quality.

Moreover, spectrograms of noisy utterance, clean utterance and enhanced utterances by DS-Net and SF-Net are presented in Figure 2 (a)-(d). Focusing on the red and green boxes in Figure 2 (b) and (c), one can see that SF-Net can better suppress background noise than DS-Net in the high-frequency regions, while more spectral details can be restored in the low-frequency regions. This indicates the superiority of SF-Net in improving speech quality in both low- and high-band cases.

The best configuration of SF-Net in Table 1 is chosen to compare with other SOTA baselines, whose results are presented in Table 2. For VoiceBank + DEMAND dataset, six full-band and one super-wideband SE approaches are selected as the baselines, namely GCRN (full-band version) [30], RNNoise [3], PercepNet [4], CTS-Net (full-band version) [8], DeepFilterNet [5], DMF-Net [31] and S-DCCRN (super-wide band) [32]. Note that we re-implement GCRN and CTS-Net (full) with power compression for a fair comparison, while we directly use the reported results of other baselines. From Table 2, several observations can be made. First, compared with the compressed psycho-acoustically feature-based methods, SF-Net provides considerably better performance in all objective metrics. Second, the proposed system dramatically outperforms the previous one-stage full-band baselines, demonstrating the effectiveness of the proposed sub-band fusion strategy. Third, compared with our preliminary decoupling-style multi bands fusion model (i.e., DMF-Net), SF-Net achieves further improvements with fewer trainable parameters, especially in the background noise suppression (CBAK) and overall quality (COVL). This indicates that the utilization of the proposed dual-stream network can realize better speech recovery and noise suppression.

We further conduct evaluations on the DNS-2022 blind test set, whose results are presented in Table 3. Due to the lack of clean speech as the reference, we utilize the non-intrusive subjective evaluation metrics to evaluate the subjective speech performance, namely DNSMOS P.808 [33] and P.835 [34], which are based on ITU-T P.808 [35] and P.835 [36], respectively. Compared with NSNet2, a standard baseline system for DNS-2022 [37], the proposed approach yields consistently better performance in speech distortion (SIG), background noise (BAK) and overall quality (OVRL). Besides, SF-Net outperforms DMF-Net in terms of speech distortion and overall quality, while a similar BAK DNSMOS score is observed. This further verifies the superiority of our approach in recovering the speech components in practical acoustic scenarios.