ANALYZING THE IMPACT OF SPEAKER LOCALIZATION ERRORS ON SPEECH SEPARATION FOR AUTOMATIC SPEECH RECOGNITION

Speech captured by a distant microphone is corrupted by reverberation and noise. In a typical home scenario, it is often further distorted by interfering speakers. This problem, referred to as the speech separation problem or the cocktail party problem, has been studied for more than 20 years [1, 2]. With the advent of neural networks, it has regained the attention of the community.

In presence of multiple speakers different time-frequency bins are dominated by different speakers and the goal is to estimate a time-frequency mask for each speaker. The problem has been addressed in both single-channel and multichannel contexts. Single-channel approaches include clustering-based methods such as deep clustering [3] and deep attractor networks [4] where a neural network is trained to cluster together the time-frequency bins dominated by the same speaker. In another approach, the speakers are estimated iteratively [5] using neural networks with permutation-invariant training criteria [6].

In multichannel scenarios, the usual approach is to estimate the second-order statistics (covariance matrices) of all speech and noise sources and to derive a beamformer to separate the speakers [7, 8]. The separation quality will therefore depend on the covariance matrix estimates. Different methods have been proposed to estimate the target speech and noise covariance matrices. In [9], the phase differences between the microphones encoding speaker location information are exploited as input features to train a deep clustering based neural network for speech separation. Explicit speaker location estimates have also been employed. In [10] and [11], the speaker is first localized and the microphone signal is beamformed towards the speaker. The beamformed signal is used to estimate a mask corresponding to the localized speaker which is then used to estimate the covariance matrices. A similar approach is proposed in [12] where the so-called speech presence probability (SPP) is estimated using speaker location information with a minimum Bayes risk detector. The speech and noise statistics are then derived from the SPP.

Speech separation algorithms are often evaluated using speech enhancement metrics such as the signal-to-distortion ratio (SDR) and the perceptual estimation of speech quality (PESQ) metric [9, 12] and, in limited cases, using automatic speech recognition (ASR) metrics [13, 14, 11]. There are very few studies analyzing the impact of localization errors on WER in large vocabulary speech recognition systems. We found one closely related study in [14], but under limited acoustic conditions and vocabulary size.

In this paper, we provide the following contributions. We create a new multichannel, multispeaker, reverberated, noisy dataset which extends the original WSJ0-2mix single-channel, non-reverberated, noiseless dataset [3] to the strong reverberation and noise conditions and the Kinect-like microphone array geometry used in CHiME-5 [15]. This allows us to use the real noise captured as part of the CHiME-5 dataset, thereby making the simulated dataset quite realistic and challenging. On this dataset, we perform speech separation using either the ground truth location of the speakers or the location estimated by the established generalized cross-correlation phase transform (GCC-PHAT) algorithm [16], and we evaluate the resulting ASR performance on the separated speech. To the best of our knowledge, this is the first evaluation of ASR performance on a multichannel, reverberated, noisy version of WSJ0-2mix by contrast with the ASR evaluation reported for the original single-channel, non-reverberated, noiseless WSJ0-2mix in [17].

The rest of the paper is organized as follows. Section 2 introduces the proposed framework for speech separation using speaker localization information. Section 3 explains the procedure used to simulate the dataset. Section 4 describes the experimental procedure and the obtained results. We conclude in Section 5.

The data for this work is based on a multichannel, reverberated, noisy version of the WSJ0-2mix dataset ¹¹1The code to recreate the dataset can be found here: https://github.com/sunits/Reverberated_WSJ_2MIX. The original WSJ0-2mix dataset introduced in [3] was created by mixing pairs of speakers from the WSJ0 corpus, and contains 20 k, 5 k, and 3 k training, development, and test mixtures, respectively. Each mixture contains two different speakers speaking for a variable duration. In this work, the “max” version of the dataset is used where the length of mixed signals is the maximum of the length of individual signals.

In our experiments we emulate the recording conditions of the CHiME-5 corpus which was recorded using Microsoft Kinect devices. For each pair of speech signals in WSJ0-2mix, we simulate room impulse responses (RIRs) using the RIR Simulator [23] for two distinct spatial locations with a minimum DOA difference of 5^∘. The room dimensions and the reverberation time (RT60) are randomly chosen in the range of $[3-9]$ m and $[0.3-1]$ s. The two speech signals are convolved with these RIRs and mixed at a random signal-to-interference ratio (SIRs) in the range of $[0-10]$ dB.

Real multichannel noise captured as part of the CHIME-5 dataset is then added with a random SNR in the range of $[0-10]$ dB. To obtain noise segments, the ground truth speech activity detection (SAD) labels from Track 3 of the DIHARD-II speaker diarization challenge [24] are used, as these are more reliable than the SAD labels originally provided in CHiME-5. The noise signals in the training, development, and test sets are taken from different CHiME-5 sessions. The noise is realistic and non-stationary in nature and makes the speech separation task very challenging. A reverberated dataset based on WSJ0-2mix was reported earlier in [25], but it does not contain any noise.

DNN to estimate the mask: Mask estimation is done in the time-frequency domain. The short time Fourier transform (STFT) of the 4-channel signal was computed using a sine window of length 100 ms and a shift of 50 ms resulting in a frequency dimension of 801. The input to the mask estimation network was of dimension $2403$: it comprises the magnitude spectrum of the DS signal, as well as the cosine and sine of the phase differences as detailed in Section 2.3, each of which is of dimension 801. A 2-layer Bi-LSTM network was trained to estimate the mask corresponding to the reverberated component of the localized speaker. No dereverberation was performed. Adam was used as the optimizer.

ASR system: For each separation method tested, the ASR system was trained on the enhanced training set using accurate senone alignments obtained from the underlying clean single-speaker utterances. The acoustic model (AM) was a 15-layer time-delayed neural network (TDNN) trained using the lattice-free maximum mutual information criterion [26]. $40$ dimensional Mel frequency cepstral coefficients along with 100-dimensional i-vectors were used as input features.

Computing WER metrics for the mixture: In every mixture we perform ASR only for the speaker who spoke for the longest duration. This was done so that the insertion errors corresponding to the speaker who spoke for a shorter duration do not effect the ASR performance. In a typical ASR system, this is handled by a voice activity detector / endpointing system which is not the focus of this work.

Estimating location using GCC-PHAT: Experiments were conducted using both ground truth DOA values as well as using the DOA values estimated by GCC-PHAT. In the case of GCC-PHAT, peaks in the angular spectrum are assumed to correspond to the DOAs of the sources. The top two peaks are chosen and the peak which is closest to the true DOA is taken as the estimated DOA. Since GCC-PHAT works using 2 microphones, only the first and the last microphone of the array which are placed at a distance of 0.226 m are used.

Results: Table 1 shows the baseline ASR performance before separation. It can be observed that background noise and overlapping speech severely degrade performance.

Table 2 shows the ASR results obtained on noisy two-speaker mixtures after speech separation. An average WER of $29.4\%$ was obtained using the ground truth DOA, a relative improvement of $49$% with respect to the system without source separation. This is close to the ASR performance for a single speaker with noise ($23.6$%) as shown in Table 1, showing that DOA information can indeed help source separation. The performance dropped to $42.4\%$ when the DOA was estimated using GCC-PHAT, indicating that erroneous DOA estimates decrease the separation quality. The DOA difference between the speakers was found to have an impact on the ASR performance. When the speakers are located at close angles with respect to the array (DOA difference $<10^{\circ}$), a WER of $37.4\%$ was obtained. When the speakers are well separated in space (say $>50^{\circ}$) the performance improved to $28.8\%$, a relative gain of $23\%$.

The SIR had an even bigger impact on the ASR performance as shown in Table 3. SIRs below $-5$ dB resulted in a WER of $59.7\%$ which improved to $24.3\%$ when the SIR was above $10$ dB, a relative improvement of $59\%$.

Finally, note that the R1-MWF beamformer outperformed the widely used GEV beamformer in all our experiments.

We conducted the first analysis of the impact of speaker localization accuracy on speech separation performance in challenging two-speaker, reverberated, noisy scenarios, as measured by the resulting ASR performance. To do so, we created a new dataset by reverberating WSJ0-2mix and mixing it with real CHiME-5 noise, and made the corresponding code publicly available. We found that the ASR performance depends more on the SIR of the speakers, with lower WERs for signals with higher SIR. The angular distance between the DOAs of the speakers was also found to have an impact, with better WERs for signals whose speakers exhibit a larger difference in DOAs. As future work we would like to investigate methods to make the mask estimation network robust to localization errors.