A Perceptual Weighting Filter Loss
for DNN Training in Speech Enhancement

In AMR speech coding, the perceptual weighting filter, employed in the analysis-by-synthesis search of the codebooks, is expressed according to [24] as

$$W_{\ell}(z)=\frac{1-A_{\ell}(z/\gamma_{1})}{1-A_{\ell}(z/\gamma_{2})},\vskip-5.69046pt$$

(2)

where $A_{\ell}(z/\gamma)\!=\!\sum_{i=1}^{N_{p}}\!a_{\ell}(i)\gamma^{i}z^{-i}$, $a_{\ell}(i)$ are the linear prediction (LP) coefficients of frame $\ell$, $N_{p}$ is the prediction order, and $\gamma_{1}$, $\gamma_{2}$ are the perceptual weighting factors. As the search of the codebooks is done by minimizing the weighted error between the clean speech and the coded speech, the weighted error becomes spectrally white, meaning that the final (unweighted) coding error is proportional to the inverse weighting filter $1/W_{\ell}(z)$. This inverse weighting filter has similarities to the structure of the clean speech spectral envelope which exploits the masking property of the human ear: More energy of the quantization error will be in the speech formant regions, as $1/W_{\ell}(z)$ is somewhat below the spectral envelope there.

A variant of the above weighting filter, originally used in AMR-WB (also used in the recent EVS codec) [25, 26] is proposed as

$$W^{\prime}_{\ell}(z)=1-A^{\prime}_{\ell}(z/\gamma_{1}),\vskip-2.84544pt$$

(3)

where the LP coefficients $a^{\prime}_{\ell}(i)$ are computed based on the speech being preemphasized by a filter $H_{\text{pre}}(z)\!=\!1\!-\!\beta z^{-1}$. Note that the weighting filter in the codecs is originally of the form $\big{(}1\!-\!A^{\prime}_{\ell}(z/\gamma_{1})\big{)}/H_{\text{pre}}(z)$ and a de-emphasis filter $H_{\text{pre}}^{-1}(z)$ is applied in the decoder. Therefore, the quantization error is actually proportional to $1/\big{(}1\!-\!A^{\prime}_{\ell}(z/\gamma_{1})\big{)}$ [25], resulting in the equivalent weighting filter (3). Both filter types being employed as potential weighting filter losses will be evaluated and discussed in the next section.

To intuitively understand this shaping effect, we exemplarily show the clean speech spectral envelope, the pedestrian (PED) noise amplitude spectrum, and the noise shaped by the inverse weighting filter (2) for an example frame in Figure 3. This example frame is from the speech file bbaf1s of the Grid corpus [30] downsampled to 16 kHz, the pedestrian noise is from the CHiME-3 dataset [31], and the SNR is 5 dB. The AMR weighting filter (2) is used in this example with $N_{p}\!=\!16$, $\gamma_{1}\!=\!0.92$, and $\gamma_{2}\!=\!0.6$. It shows nicely, how the noise is shaped to be “hidden” under the speech spectral envelope, so that the shaped noise will be less perceivable by the human ear.

Now that the perceptual weighting filter from CELP speech coding has been revisited, we show how a weighting filter loss is constructed for DNN training in Figure 1. We derive the loss from the AMR weighting filter, while the AMR-WB weighting filter is obtained straightforward. First, LP analysis is performed based on the clean speech frame $s_{\ell}(n)$ to obtain the frame-wise LP coefficients $a_{\ell}(i)$. Then, the weighting filter $W_{\ell}(z)$ is calculated by (2) and the corresponding frequency amplitude response is obtained as

$$W_{\ell}(k)=W_{\ell}(z)\!\Bigm{|}_{z=e^{j\!\frac{2\pi k}{K}}},\ \ \ \ \ k\in\{0,1,\ldots,K/2\}.\vskip-2.84544pt$$

(4)

Finally, the loss function is obtained as

$$J_{\ell}=\big{(}E^{\text{w}}_{\ell}(0)\big{)}^{2}+\big{(}E^{\text{w}}_{\ell}(\tfrac{K}{2})\big{)}^{2}+2\cdot\!\!\sum_{k=1}^{K\!/2\!-\!1}\big{(}E_{\ell}^{\text{w}}(k)\big{)}^{2},\vskip-5.69046pt$$

(5)

where $E_{\ell}^{\text{w}}(k)\!=\!|W_{\ell}(k)|\cdot E_{\ell}(k)$ is the weighted error, and $E_{\ell}(k)\!=\!|S_{\ell}(k)|-\widehat{|S_{\ell}(k)|}$ is the training error. As $E_{\ell}^{\text{w}}(k)$ becomes spectrally white, we find that $E_{\ell}(k)\!\sim\!1/|W_{\ell}(k)|$. As a result, the residual noise is expected to be less audible. It is worth noting that in the enhancement stage, the inference process of this DNN is the same as for the reference DNN.

The speech material used in this paper is from the Grid corpus [30] and is downsampled to 16 kHz. We randomly select 8 female and 8 male speakers, and each speaker includes 100 sentences with a duration of 3 seconds per sentence. Thus, the total amount of clean speech used for training and validation is 80 minutes long. Another 2 female and 2 male speakers with 20 sentences each are used for testing. The additive background noise is from the CHiME-3 dataset: Pedestrian noise (PED), café noise (CAF), and street noise (STR) are used for the training, validation, and test, while bus noise (BUS) is used additionally as an unseen noise type for testing. Training and validation material contains the noisy data at six SNR levels (from -5 dB to 20 dB with 5 dB step size), and these two sets are obtained by a split with a ratio of 4 $:$ 1 with a proportional number of sentences covered by all the speakers, noise types and SNR levels. Note that the noise material used in the test set is also disjoint from that for training and validation.

Regarding the DNN training, the input is first normalized to have zero mean and unit variance based on the statistics of the training data. Then, the weights of the DNN are trained by minimizing the loss function, applying the Adam algorithm with the learning rate being $5\!\cdot\!10^{-4}$. The weights are updated after each minibatch containing 128 input and target pairs, which are randomly selected from the training data.

In order to evaluate the speech enhancement system including both, speech and noise components, PESQ [32] and STOI [10] are used to measure the quality and the intelligibility of the enhanced speech. Also, SNR improvement (SNRI) [33] is used to measure the SNR improvement achieved by the speech enhancement system.

Regarding the evaluation metric for the speech component, segmental speech-to-speech-distortion ratio (SSDR) is calculated after [34] as

$$\text{SSDR}=\frac{1}{\left|\mathcal{L}\right|}\sum_{\ell\in\mathcal{L}}\text{SSDR}(\ell),\vskip-2.84544pt$$

(6)

where $\mathcal{L}$ is the set of frame indices for speech active frames and $\text{SSDR}(\ell)$ is limited from $R_{\text{min}}\!=\!-10$ dB to $R_{\text{max}}\!=\!30$ dB by $\text{SSDR}(\ell)\!=\!\text{max}\left\{\text{min}\left\{\text{SSDR}^{\prime}(\ell),\!R_{\text{max}}\right\}\!,\!R_{\text{min}}\right\}$. The term $\text{SSDR}^{\prime}(\ell)$ is actually calculated as

$$\text{SSDR}^{\prime}(\ell)=10\text{log}_{10}\left[\frac{\sum_{n\in\mathcal{N}_{\ell}}s(n)^{2}}{\sum_{n\in\mathcal{N}_{\ell}}\left(\tilde{s}(n)-s(n)\right)^{2}}\right]\;[\text{dB}],\vskip-2.84544pt$$

(7)

where $\mathcal{N}_{\ell}$ is the set of sample indices $n$ belonging to frame $\ell$, $s(n)$ is the clean speech, and $\tilde{s}(n)$ is the time-aligned filtered speech component. Note that this filtered speech component $\tilde{s}(n)$ and the following filtered noise component $\tilde{d}(n)$ are obtained on the basis of (1), but replacing the noisy speech spectral amplitudes $|Y_{\ell}(k)|$ by either the speech component spectral amplitudes $|S_{\ell}(k)|$, or the noise component spectral amplitudes $|D_{\ell}(k)|$, respectively [35].

Another evaluation metric we use in this paper to measure the SNR improvement is $\Delta\text{SNR}\!=\!\text{SNR}_{\text{out}}\!-\!\text{SNR}_{\text{in}}$, where $\text{SNR}_{\text{in}}$ is the SNR level for the noisy mixture $y(n)$, and $\text{SNR}_{\text{out}}$ is calculated based on $\tilde{s}(n)$ and $\tilde{d}(n)$.

We first search for the optimal parameter $\gamma_{1}$ of the perceptual weighting filter applied to the loss function by conducting experiments on a development dataset. This development dataset is a subset of the validation data, which uses a quarter of the data from the validation dataset covering all speakers, noise types, and SNR levels. It is used to decrease the amount of data for optimal parameter search, thus improving efficiency. As shown in Figure 4, the two abovementioned perceptual weighting filters (see (2) and (3), with $N_{p}\!=\!16$) are evaluated with various perceptual weighting factors. Regarding the weighting filter from AMR, we search the optimal $\gamma_{1}$, as different $\gamma_{1}$ values are also applied for different bitrates in the AMR codec [24]. In the meanwhile, we keep $\gamma_{2}\!=\!0.6$ unchanged because an informal search on $\gamma_{2}$ with limited values shows that the performance has a rather weak dependency on $\gamma_{2}$. The same search range of $\gamma_{1}$ is also investigated for the weighting filter from AMR-WB, and we keep the preemphasis filter factor $\beta\!=\!0.68$ as in [25]. The factor combinations actually define the spectral shape and the spectral tilt of the weighting filter. It can be seen that under the investigated parameters, the weighting filters with various $\gamma_{1}$ from AMR shows generally better performance compared to those from AMR-WB. The weighting filters of AMR with $\gamma_{1}$ in the selected range (left part in Figure 4) can outperform the reference DNN except for some outlier values of $\gamma_{1}$. The optimal settings are the weighting filter from AMR (2) with $\gamma_{1}\!=\!0.92,\,\gamma_{2}\!=\!0.6$. The DNN model optimized by the loss function with the optimal weighting filter settings will be used in the following experiments (denoted as Weighted DNN).

In Table 1, we report the results on the test dataset and compare them to the reference DNN. The results are collected in four noise conditions and all values are averaged over six SNRs from -5 to 20 dB. We can see the PESQ score improvements from the noisy speech $y(n)$ to the enhanced speech $\hat{s}(n)$ for both approaches. Compared to the reference DNN, the proposed method achieves improved PESQ performance in the range of 0.07…0.11 points for the three seen noise types (PED, CAF, and STR), and 0.05 points improvement for the unseen BUS noise condition. The STOI measures are comparable for the two approaches. Regarding the SNR improvement, the proposed approach consistently outperforms the reference DNN in terms of SNRI and $\Delta$SNR by more than 3.5 dB and 0.6 dB, respectively, for all four noise conditions. The results support the effectiveness of the proposed loss function, showing that the difference between the enhanced speech and the clean speech is less perceptually significant.

We notice that the SSDR is decreased by using the weighting filter loss function, which is easy to explain: The trained DNN with the weighting filter loss is biased to focus on the spectrum, i.e., more focus is put on the clean speech spectral valley regions and less on the formant regions. This effectively improves the perceptual quality of the enhanced speech as shown in the above PESQ scores, however, it introduces some measurable distortions to the speech component. This proves that the weighting filter loss does what it is expected to do: It does not excel the reference DNN in terms of MSE (or SSDR), but only subjectively (shown by PESQ).

We further show some results in various SNR levels in Figure 5, where all the values are averaged over the four noise types. The PESQ score improvement for the proposed approach over the reference DNN is more significant in higher SNR levels than in lower SNR levels. As already shown in Table 1, the STOI measures for the two approaches are quite comparable also across various SNR levels. Regarding the SNRI metric, the proposed method clearly excels the reference DNN in all SNR conditions, obtaining 4.17 dB SNRI improvement on average of SNR conditions and noise types.