Multiple Speaker Separation from Noisy Sources in Reverberant Rooms using Relative Transfer Matrix
^††thanks: Thanks to Australian Research Council (ARC) Discovery Project Grant DP200100693 for funding.

Consider a reverberant environment with concurrently active $\mathcal{L}_{S}$ speech and $\mathcal{L}_{N}$ background noise sources. Let $\mathcal{L}=\mathcal{L}_{S}+\mathcal{L}_{N}$. In the short time Fourier transform (STFT) domain, we denote $S_{\ell}^{(S)}(f,t)$, $\ell={1,\cdots,\mathcal{L}_{S}}$ and $S_{\ell}^{(N)}(f,t)$, $\ell={1,\cdots,\mathcal{L}_{N}}$ as the speech $(S)$ and background noise $(N)$ signals, respectively.

Let there be $Q$ arbitrary distributed microphones in the room. We divide them to two groups of microphones, $\{A\}$ and $\{B\}$ with $Q_{A}$ and $Q_{B}$ microphones, respectively ($Q=Q_{A}+Q_{B}$). We denote $\mathbf{M}_{A}(f,t)$ and $\mathbf{M}_{B}(f,t)$ as the vector of received signals at microphone groups A and B, respectively. Then the received signals at each microphone group in matrix form as

where $\mathbf{S}(f,t)=[S_{1}^{(S)},\ldots,S_{\mathcal{L}_{S}}^{(S)},S_{1}^{(N)},\ldots,S_{\mathcal{L}_{N}}^{(N)}]^{T}$, and $\{\cdot\}^{T}$ is the matrix transpose. Here, $\mathbf{H}_{A}(f)\in\mathbb{C}^{Q_{A}\times\mathcal{L}}$ and $\mathbf{H}_{B}(f)\in\mathbb{C}^{Q_{B}\times\mathcal{L}}$ are the matrices with elements defined by the acoustic transfer functions. The microphone thermal noise vector $\mathbf{V}_{A}(f,t)\in\mathbb{C}^{Q_{A}\times 1}$ and $\mathbf{V}_{B}(f,t)\in\mathbb{C}^{Q_{B}\times 1}$ are similarly defined.

The aim of this paper is to separate each individual speech signal $S_{\ell}^{(S)}(f,t)$, $\ell={1,\cdots,\mathcal{L}_{S}}$, from the concurrent speakers $\mathcal{L}_{S}$ and the background noise sources $\mathcal{L}_{N}$.

The ReTM, $\bm{\mathcal{R}}_{AB}(f)$, is define as in [17]

where $(\cdot)^{\dagger}$ is Moore-Penrose inverse, assuming the validity, i.e., $Q_{B}\geq\mathcal{L}$. Thus, we can relate the received signal at group $\{A\}$ and $\{B\}$ using

Note that $\bm{\mathcal{R}}_{AB}(f)$ is defined by the spatial properties of the sound sources such that it is independent of the sound source signals. In applications, the ReTM is invariant for a stationary environment.

The next section shows how to separate the desired speakers by estimating the ReTM of the undesired sources in practice.

We utilize an open-source toolbox [21] to model the room impulse response (RIR) from the sound sources to irregularly distributed microphones in a $6\times 7\times 3$ m rectangular room ($T_{60}=500$ ms). We consider four speech sources, three background noise sources, and $27$ microphones. Four speaker locations are: (1): (3 m, 4.5 m, 1.2 m), (2): (3 m, 2.5 m, 1.2 m), (3): (4.5 m, 3.5 m, 1.2 m), and (4): (1.5 m, 3.5 m, 1.2 m) and three background noise sources locations are: (1): (2 m, 0.9 m, 1.8 m), (2): (1 m, 6 m, 2.5 m), and (3): (3.5 m, 5 m, 1.8 m) with respect to the origin position in the left corner of the room. We convolve the speech sources RIRs with both male and female speech utterances from the TIMIT dataset [22] and noise sources RIRs with wall air-conditioner noise, vacuum noise, and music signal. The received signals are down-sampled to $16$ kHz and ranged from $0$ to $-20$ dB SNR of background noise and added with $60$ dB SNR of white Gaussian noise at each microphone. Here, we define the SNR by averaging SNR at each receiver over all $27$ receivers.

We assume this undesired sound sources-only signal $\mathbf{\bar{S}}_{\ell}$ has been obtained when each $\ell^{th}$ target speaker $\ell\in\{1,\cdots,\mathcal{L}_{S}\}$ is inactive and use a $60$ second segment of this recording for $\bm{\mathcal{R}}_{AB}^{({\ell})}$ training. Then, the recordings are short-time-Fourier-transformed with an 8192-point window size that was long relative to the length of the RIR to satisfy the multiplicative transfer function [23]. We assign $Q_{A}\in\{5,7,10\}$ varying number of receivers to group $\{A\}$, and $Q_{B}=17$ a fixed number of receivers to group $\{B\}$.

We use two qualitative measures to assess the separation performance: (i) Signal-to-Interference Ratio (SIR), and (ii) Signal-to-Distortion Ratio (SDR), using the BSS-Eval toolbox [24]. We also measure speech intelligibility through Short-Time Objective Intelligibility (STOI) [25]. The SNR is defined with respect to all sources in the mixture, whereas SIR is defined considering one speaker as the target signal and the rest of the interfering speakers as the noise signal.

We first evaluate performance in a varying number of $Q_{A}$ microphones with fixed $Q_{B}$ at $-15$ dB SNR level in Table I. We observe that both group A sizes of $7$ and $10$ yield the best results, however, $Q_{A}=7$ obtains the highest SDR. The total number of sources in the mixture is $\mathcal{L}=7$, suggesting a lower bound for the groups $\{A,B\}$ size for the improved separation capabilities. The results confirm that with the higher $Q_{A},Q_{B}(\geq\mathcal{L})$ accurately separated the speech signals in noisy reverberant environments.

We will now examine the spectrogram plots of the mixture signal, separated speech signal outputs at the microphone $1$ of $Q_{A}=10$ in Fig. 1. We observe that the input signal is contaminated by background noise, however, the separated output for each speaker is less noisy. We share a link to the mixture signal and the separation speech signals to listen.¹¹1https://github.com/wnilmini/SS-ReTM

Next, we examine the performance of the proposed algorithm with different SNR levels in Fig. 2. Given that the proposed method works better with more microphones in groups A and B than the number of sound sources, we set $Q_{A}=10$ and $Q_{B}=17$ with varying SNR levels from $0$ dB to $-20$ dB.

Results of Fig. 2 show that the proposed method is able to successfully separate all speakers in the mixture, especially with the different input speaker configurations, i.e., STOI, SIR, at low SNR levels. Three key observations are made from Fig. 2. First is in terms of intelligibility improvement, where we observe the highest output STOI values for the same speaker with the highest input STOI while the rest achieve comparable results. The second observation is on separation results, where we observe improved SIR, and SDR for individual speakers with a performance gap that the first speaker is dominated which has the highest SIR value if four speakers are with different SIR levels. Our third observation is that the speaker separation results are seen to be nearly constant for each speaker over the low input SNR conditions as focusing constant input SIR of the individual speaker.

Finally, we consider intelligibility performance with different SIR levels from $-15$ dB to $0$ dB and a fixed SNR level of $-15$ dB to explain both the second and the third observations on speaker separation. In Fig. 3 we present four scenarios with the assumption that the target speaker’s SIR levels are varied given that the other three speakers and the background noise sources in the mixture are kept the same. This is similar to Fig. 2(a), where the first speaker with the highest SIR among all four speakers in the mixture is well-separated by the proposed method achieving the highest STOI performance. It can be seen in Fig. 3 that the proposed method has indistinguishable intelligibility performance for the other three speakers in the mixture. Note that the algorithm’s robustness to room reverberation is not included in Section IV, since it provided similar performance.

Now, we conduct a preliminary study of this work in a real-life scenario with highly overlapped speakers. The real recordings are measured in an office room at the Australian National University with dimensions $2.94\times 4.4\times 3.04$ m, and $T_{60}\approx 410$ ms. We consider $2$ speakers, $2$ background noise sources (fan and room air cooler), and $15$ randomly distributed microphones over the room (as shown in Fig. 4). We assigned $5$ microphones (channels $\{3,4,10,12,15\}$) to group A, and $10$ microphones (channels $\{1,2,5:9,11,13,14\}$) to group B. We examined the proposed method’s performance in separating the desired speakers inside a real room. We share a link to the audio files.²²2https://zenodo.org/records/15009085 By listening to the output signals, we observe that the leakage of the interfering sources into the separated desired speakers. However, the desired speakers are clearly audible to understand their utterances. Note that the separation results are inferior to the results obtained in simulations. We attribute this discrepancy can be modified by using Wiener filtering techniques to enhance the separation performance, which we will continue to discuss in future work.

The machine learning-based methods [3, 4, 11, 12, 13, 14, 15, 16] for audio speaker separation consider data-driven approaches and exploit either speech features or sparsity in their models/algorithms, perhaps both in a supervised [11, 14] or unsupervised [3] manner whereas in the proposed algorithm only requires speaker spatial location or environment-based blind (or semi-blind) training among the microphone groups. Both DNN-based methods and NMF-based methods require greater computational power which aggravates with the number of microphone channels in the mixture whereas our proposed method alleviates the computational cost of multichannel speaker separation using one to two minutes of the ReTM training. Moreover, DNN-based methods rely on numerous assumptions about the acoustic environment or the background noise sources and essential rigorous training for specific microphone array structures and/or a given number of speakers in the mixture. However, the proposed method is independent of the microphone placement in the specific acoustic scenario, however, requires a more spatially diverse microphone configuration to learn the ReTM. Furthermore, we note that our algorithm is independent of the source signals, and the number of sources prior to the separation task as well as, speech-based features like pitch and sparsity which are unreliable under noisy environments.

As our method requires multiple microphones (groups A and B should have microphones more than the number of sound sources in the mixture), it is not direct and easy to compare fairly with existing datasets and tasks to reproduce with baseline methods with appropriate configurations. But in the future, we seek to have a fair comparison with the baseline speaker separation methods.

For now, we will present the performance evaluation of the DNN-based method, SepFormer [16]. To fairly evaluate the algorithms, we use the same settings given in IV-A with SNR $=0$ dB. Table II depicts the speaker separation results for $\mathcal{L}_{S}=3$. The results confirm that both methods effectively separate the speaker with the highest loudness. Although ’SepFormer’ exhibits slightly higher performance than the input signal, it fails to effectively separate the concurrent speakers when their loudness levels vary, whereas our method successfully achieves this separation. Note that the results for low SNR levels are not reported here, since it is severely impacted by the noise. We share a link to the audio files.³³3https://github.com/wnilmini/SS-ReTM/tree/main/Baseline