Subband Splitting: Simple, Efficient and Effective Technique for Solving Block Permutation Problem in Determined Blind Source Separation

Determined BSS can be formulated as an optimization problem of the demixing matrices in the time-frequency domain. Let $\mathbf{s}_{ft}=[s_{ft1},\ldots,s_{ftN}]^{\mathsf{T}}\in\mathbb{C}^{N}$ be a vector of $N$ source signals at the $(f,t)$th bin, where $1\leq f\leq F$ and $1\leq t\leq T$ are frequency and time indices, respectively. The $M$-channel observed signal $\mathbf{x}_{ft}=[x_{ft1},\ldots,x_{ftM}]^{\mathsf{T}}\in\mathbb{C}^{M}$ is approximated using the frequency-wise mixing matrix $\mathbf{A}_{f}\in\mathbb{C}^{M\times N}$ as $\mathbf{x}_{ft}=\mathbf{A}_{f}\mathbf{s}_{ft}$. In a determined situation (i.e., $N\leq M$), the $n$th separated signal at the $(f,t)$th bin is extracted using the $n$th demixing vector $\mathbf{w}_{\!fn}\in\mathbb{C}^{M}$ as follows:

Then, the $N$ separated signals $\mathbf{y}_{\!ft}=[y_{ft1},\ldots,y_{ftN}]^{\mathsf{T}}\in\mathbb{C}^{N}$ are obtained by

where

is the demixing matrix. For notational simplicity, we omit the indices to represent all the components altogether as $\mathbf{x}=((\mathbf{x}_{ft})_{f=1}^{F})_{t=1}^{T}$, $\mathbf{y}=((\mathbf{y}_{\!ft})_{f=1}^{F})_{t=1}^{T}$ and $\mathbf{W}=(\mathbf{W}_{\!f})_{f=1}^{F}$. Then, the linear operation in Eq. (2) for all frequencies $f=1,\ldots,F$ and times $t=1,\ldots,T$ is shortly represented as $\mathbf{y}=\mathbf{W}\mathbf{x}$ for brevity.

To separate the sources using Eq. (2), addressing the permutation problem is essential. Namely, the extraction target of demixing vector $\mathbf{w}_{\!fn}^{\mathsf{H}}$ in Eq. (1) must be consistent across all frequencies $f=1,\ldots,F$. A standard approach to the permutation problem is to model the co-occurrence of the frequency components. For instance, IVA [10, 11] considers the frequency-directional group structure, which results in the minimization problem of the following objective function:

where $y_{ftn}=\mathbf{w}_{\!fn}^{\mathsf{H}}\mathbf{x}_{\!ft}$. ILRMA[23] utilizes nonnegative matrix factorization (NMF) [42] to model low-rankness of the power spectrograms of sources. The typical objective function is

where $\boldsymbol{\Theta}=((\mathbf{T}_{n})_{n=1}^{N},(\mathbf{V}_{n})_{n=1}^{N})$ represents the set of auxiliary variables for NMF, $\mathbf{T}_{n}=[\mathbf{t}_{1n},\ldots,\mathbf{t}_{Fn}]^{\mathsf{T}}\in\mathbb{R}_{+}^{F\times K}$ and $\mathbf{V}_{n}=[\mathbf{v}_{1n},\ldots,\mathbf{v}_{Tn}]\in\mathbb{R}_{+}^{K\times T}$ are the basis and activation matrices, respectively, and $K$ is the number of bases.

Despite the great success of the above methods, they occasionally fail in separation due to the block permutation problem[16, 28, 29, 8, 36]. Figure 2 illustrates such a failure by an example of signals separated by IVA, where $M=N=2$ and each of the two source signals is colored either green or red. As in Fig. 2 (b), three frequency blocks appeared in this case: the lower- and higher-frequency blocks extracting the green source and the middle-frequency block for the red source. Consequently, IVA did not keep consistent permutations among these frequency blocks, resulting in poor separation performance.

However, even when the block permutation problem arises, the demixing matrices obtained for each frequency block are often optimized correctly, except for their permutations. As an example, Fig. 3 displays the frequency-wise separation performance of the result shown in Fig. 2 (b). We calculated the frequency-wise version of the scale-invariant signal-to-distortion ratio (SI-SDR) of the separated signals $\mathbf{y}$ in the time-frequency domain as follows:

where $\tilde{s}_{ftn}\in\mathbb{C}$ is the scaled version of the $n$th true source signal at the $(f,t)$th bin, i.e.,

and complex conjugation is denoted by $\bar{(\cdot)}$. The area filled with light gray in Fig. 3 corresponds to the separation result of IVA in Fig. 2 (b). The dotted line shows IVA + IPS, where IPS ideally resolved the permutation problem of the result of IVA so that the correlation between the separated signal and the true source is maximized (see Eq. (26)). For reference, the blue line indicates FDICA with IPS (FDICA + IPS). In this example, the separation performance of IVA was extremely bad within 0.5 to 1.5 kHz, which corresponds to the middle-frequency block in Fig. 2 (b), where the red source is extracted. However, IVA + IPS was comparable to that of FDICA + IPS across all frequencies, which highlights that if the block permutation was circumvented, then IVA performs effectively. This observation motivates us to propose some specialized techniques to avoid the block permutation problem for IVA or other existing BSS algorithms.

While the block permutation problem arises from various factors, one major factor is considered to be the difficulty in handling the complicated structure of audio signals. In speech signals, for example, there are two main components: vowels dominating the lower-frequency bands and consonants appearing in the higher-frequency bands. When a BSS algorithm attempts to separate the sources, it must align the permutations of these two components, which is not straightforward since they appear in different frequency bands and at different times. At the same time, locally looking at a narrower frequency band (e.g., the mid-frequency block in Fig. 2 (b)), only a few components are dominant and have a simple structure. Therefore, it should be easy for existing BSS algorithms to resolve permutations inside such narrower bands. If we split the BSS problem into a set of several BSS problems in the narrower frequency bands, then we can focus on how to align the permutations between the multiple bands. The block permutation solvers can be used for this approach with some additional computational costs, but we can resolve the block permutation problem without an additional cost, as described in the next section.

In the proposed method, the entire frequencies are split into $I$ subbands $(\mathcal{F}_{i})_{i=1}^{I}$, where the $i$th subband $\mathcal{F}_{i}$ is the set of frequency indices,

$F$ is the number of frequency bins, and $L_{i}$ and $H_{i}$ $(L_{i}\leq H_{i})$ are the lower and upper bounds for the $i$th subband $\mathcal{F}_{i}$, respectively. All frequency indices must be contained in at least one subband, and hence $(\mathcal{F}_{i})_{i=1}^{I}$ must satisfy

Additionally, the adjacent subbands must overlap, i.e.,

For notational convenience, let the separation procedure of a BSS algorithm (e.g., IVA and ILRMA) be written as

where the BSS algorithm receives an observed signal $\mathbf{x}$, an initial value of the demixing matrix $\mathbf{W}^{(\text{init})}$, and initial values of other auxiliary variables $\boldsymbol{\Theta}^{(\text{init})}$ (e.g., NMF variables $(\mathbf{T}_{n})_{n=1}^{N}$ and $(\mathbf{V}_{n})_{n=1}^{N}$ of ILRMA in Eq. (5) or any other variables necessary for the BSS algorithm). After running the BSS algorithm, it returns separated signals $\mathbf{y}$, the corresponding demixing matrix $\mathbf{W}$, and the corresponding auxiliary variables $\boldsymbol{\Theta}$.

Using the above notation, the proposed subband splitting for any BSS algorithm $\mathsf{BSS}(\cdot)$ is given as follows:

where $\mathsf{extract}_{\mathcal{F}_{i}}(\cdot)$ is the operator that extracts the part of variables corresponding to the $i$th subband $\mathcal{F}_{i}$. The extracted part is indicated by the subscript $(\cdot)_{\mathcal{F}_{i}}$ as

The operator $\mathsf{substitute}_{\mathcal{F}_{i}}(\cdot)$ substitutes the extracted part of the variables back into their original location as

namely, the part of optimization variables associated with the subband $\mathcal{F}_{i}$ are replaced with the latest optimization result provided by $\mathsf{BSS}(\cdot)$. The proposed method repeats Eq. (12) for $i=1,\ldots,I$ so that $\mathsf{BSS}(\cdot)$ is applied to all the subbands.

Note that $\mathsf{extract}_{\mathcal{F}_{i}}(\cdot)$ and $\mathsf{subtitute}_{\mathcal{F}_{i}}(\cdot)$ for the auxiliary variables $\boldsymbol{\Theta}$ should be defined similarly, depending on the BSS algorithms used in each subband. Specific definitions of the extraction/substitution operators for SS-IVA and SS-ILRMA are described in Section 4.1

The most important aspect of the proposed method is the existence of the overlap of the subbands. In the overlapping part $\mathcal{F}_{i}\cap\mathcal{F}_{i+1}$, the output of $\mathsf{BSS}(\cdot)$ within the $i$th subband is carried over to the $(i+1)$th subband as the initial value for $\mathsf{BSS}(\cdot)$. Therefore, within the $(i+1)$th subband, the BSS algorithm is induced to obtain the same permutation as that obtained in the $i$th subband. This initialization strategy can keep consistent permutation among the subbands if there are sufficient overlaps.

To generate the subbands, we define shift rules that constantly shift the subbands upward or downward:

where $\Delta>0$ is the amount of shift.

To easily compare the settings for subbands, we introduce subband parameter $(\theta_{W},\theta_{\Delta})$. The parameter $\theta_{W}\geq 1$ determines the width of subband $W$ $(=H_{i}-L_{i}+1)$ and the parameter $\theta_{\Delta}\geq 1$ controls the amount of shift $\Delta$ as follows:

where $\lceil\cdot\rceil$ is the ceiling function. That is, the width of each subband $W$ is $1/\theta_{W}$ times the entire number of frequency bins $F$. The subband is shifted by the amount of another $1/\theta_{\Delta}$ times the bandwidth $W$.

To ensure that all the frequencies, including the highest and the lowest, are updated by the same number, we set the first bounds as follows:

Using Eq. (17), all the frequency bins are included in $\theta_{\Delta}$ subbands. Namely, for every frequency (including the lowest and the highest), BSS methods are applied by the same number, i.e., $\theta_{\Delta}$ times²²2 This may not be true when $\theta_{\Delta}$ is neither an integer nor a divisor of $W$. For example, when $\theta_{\Delta}=2.5$, BSS methods are applied either 2 $({}=\lfloor\theta_{\Delta}\rfloor)$ or 3 $({}=\lceil\theta_{\Delta}\rceil)$ times for each frequency bin, where $\lfloor\cdot\rfloor$ denotes the flooring function. . Note that the first and the last bounds overflow the entire frequency band $\{1,\ldots,F\}$, i.e., $L_{i}<1$ or $H_{i}>F$ holds for smaller and larger $i$. Such bounds are clipped to the range of $1$ to $F$ when each subband $\mathcal{F}_{i}$ is calculated (see Eq. (8)), and then the corresponding subbands become narrower than $W$. For instance, the first subband $\mathcal{F}_{1}$ and (when $\theta_{W}\theta_{\Delta}$ is a divisor of $F$) the last subband $\mathcal{F}_{I}$ have only $\Delta$ frequency bins.

Figure 4 illustrates an example of the subbands generated by the above rule. Here, the subband parameter is set to $(\theta_{W},\theta_{\Delta})=(3,2)$, i.e., the bandwidth is $W=F/3$, and the amount of shift is $\Delta=F/6$. All the frequency bins are included in 2 $({}=\theta_{\Delta})$ subbands. The subband splitting with the shift rule is summarized in Alg. 1, where any BSS method can be used for $\mathsf{BSS}(\cdot)$.

Two iterations appear in the proposed method: (i) the outer iteration $i=1,\ldots,I$, where $\mathsf{BSS}(\cdot)$ is sequentially applied to the subbands $(\mathcal{F}_{i})_{i=1}^{I}$, and (ii) the inner iteration that corresponds to the iterative updates in $\mathsf{BSS}(\cdot)$.

Here, we confirm the relationship between the parameter $\theta_{\Delta}$, the number of inner iterations $J_{\textrm{inner}}$, and the total number of updates $J_{\textrm{total}}$, i.e., how many times the demixing matrix $\mathbf{W}_{f}$ is updated for each frequency ³³3 The total number of updates of the demixing matrix for each frequency will be the same because every frequency bin is included in the same number of subbands, as described in Section 3.3. . Since the proposed method runs BSS algorithms $\theta_{\Delta}$ times duplicated for each frequency, the total number of updates increases as $\theta_{\Delta}$ increases, i.e.,

For example, in Fig. 4, all the frequency bins are updated twice because $\theta_{\Delta}=2$.

With fixing the number of inner iterations $J_{\mathrm{inner}}$, the overall computational cost becomes $\theta_{\Delta}$ times larger due to the duplicated run of the BSS algorithm. This increase in computational cost can be canceled by dividing the number of inner iterations $J_{\mathrm{inner}}$ by $\theta_{\Delta}$. Namely, given a total number of updates $J_{\textrm{total}}$, set the number of inner iterations as follows⁴⁴4 The increase in computational cost due to the duplicated runs of BSS algorithms is canceled using Eq. (19) when the cost of BSS algorithms depends linearly on the number of frequency bins. This condition is satisfied with AuxIVA[12] and ILRMA[23]. :

Note that the proposed method can improve memory efficiency because each subband is narrower than usual.

To compare our subband splitting with conventional subband-aware methods, we tested OC-IVA, which is an extension of IVA[18]. Similar to the proposed method, OC-IVA considers overlapping subbands to estimate demixing matrices. The significant difference with our SS-IVA is that OC-IVA separates all frequency bands simultaneously using an algorithm derived from the following objective function:

where we generate the subbands $(\mathcal{F}_{i})_{i=1}^{I}$ in the same way with our method. Note that the first term models the subband-wise group structure (see Eq. (4) for comparison to the original IVA). Comparison with OC-IVA helps to see whether optimizing the subbands simultaneously or sequentially yields better performance.

We evaluated BSS methods using two-channel mixtures of two sources, i.e., $N=M=2$. The speech and music signals were obtained from SiSEC 2011[40] dataset. For speech signals (Section 4.4 and 4.5), 8 speech signals were obtained from dev1 in underdetermined speech and music mixtures task[40]. Then, all 56 possible pairs (considering their permutations) were used as source signals. Their duration was 10 s, and the sampling frequency was 16 kHz. For music signals (Section 4.6), we used 4 songs from professionally produced music recordings task[40]. For each song, two instruments (including vocals, guitars, violins, and synth) were chosen as in [23, Table 3] (the paper proposed ILRMA). They were downsampled to 16 kHz.

The observed signals were generated by convolving room impulse responses in [41] with downsampling to 16 kHz. The reverberation time was 160 ms. The pair of source directions were $(-45^{\circ},30^{\circ})$, $(-75^{\circ},30^{\circ})$, $(-45^{\circ},60^{\circ})$ and $(-75^{\circ},60^{\circ})$. The spacing between the two microphones was 8 cm, and the distance between the sources and the center of the microphones was 1 m.

The demixing matrices $\mathbf{W}_{\!f}$ were initialized with identity matrices. All elements of the initial value for $(\mathbf{T}_{n})_{n=1}^{N}$ and $(\mathbf{V}_{n})_{n=1}^{N}$ were drawn from the uniform distribution $\mathcal{U}(0,1)$, where five different seeds were used.

The window length and the hop size of STFT were set to 2048 and 1024 samples, respectively. The Hann window was used for the window function. The Nyquist frequency was omitted from the optimization target, i.e., the number of frequency bins $F$ was $1024$.

The source-to-distortion ratio improvement (SDRi) [43] was used to evaluate separation performance. In addition, we define permutation consistency, a weighted accuracy of permutation, as follows:

where $\gamma_{f}$ is the frequency-wise weight given by the power of each frequency, i.e.,

$\delta_{ij}$ is Kronecker’s delta,

$q_{f}$ is the index of correct permutation that achieves the highest correlation between the source and separated signal at each frequency, i.e.,

where $\sigma_{qn}\in\{1,\ldots,N\}$, $\boldsymbol{\sigma}_{q}=(\sigma_{q1},\ldots,\sigma_{qN})\in S_{N}$ is the $q$th permutation (i.e., $\{\boldsymbol{\sigma}_{1},\ldots,\boldsymbol{\sigma}_{N!}\}=S_{N}$), and $S_{N}$ is the set of all permutations of $N$ elements. Permutation consistency takes the values between $1/(N!)$ and $1$, and the higher, the better. The experiments were performed with MATLAB R2024b on AMD Ryzen 9 5950X (3.40 GHz, 16 core).

We evaluated the separation performance of BSS methods using speech signals. The total number of updates $J_{\textrm{total}}$ was fixed to 100. The parameters $\theta_{W}$ and $\theta_{\Delta}$ were set to 2 or 4 to ensure that $\theta_{\Delta}\in\{2,4\}$ is a divisor of 100 (= $J_{\textrm{total}}$) and that $\theta_{W}\theta_{\Delta}\in\{2,4,8\}$ is a divisor of 1024 (= $F$). The number of bases for ILRMA $K$ was set to 2 and 10. We also tested FDICA + IPS, the oracle method that individually separates each frequency via FDICA and solves the permutation via IPS, where IPS finds the ideal permutation using Eq. (26).

Figure 5 and Table 1 summarize their separation performance, where Vanilla refers to the original IVA or ILRMA handling all frequencies simultaneously. The experimental results show the notable improvements brought by the proposed subband splitting.

Both the conventional OC-IVA and our proposed SS-IVA outperformed Vanilla IVA, confirming that band splitting enhances separation performance. In all the settings, our SS-IVA (sequential separation) consistently outperformed OC-IVA (simultaneous separation). Notably, the permutation consistency of SS-IVA was significantly high compared to OC-IVA, which demonstrates that the sequential procedure contributes to aligning permutation.

For ILRMA, the proposed SS-ILRMA obtained the highest performance for almost all subband parameters. Downward SS-ILRMA with $(K,\theta_{W},\theta_{\Delta})=(2,4,2)$ achieved the best result in the experiment. Remarkably, its separation performance was comparable to FDICA + IPS, and the permutation consistency was almost perfect (99.60 on average). Moreover, even under the worst case (indicated by a large marker), it resulted in $6.22$ dB in SDRi, which was $7.88$ dB higher than Vanilla ILRMA ($-1.66$ dB). Note that the average performance of ILRMA and SS-ILRMA tended to be better when $K=2$ rather than when $K=10$. At the same time, when $(\theta_{W},\theta_{\Delta})=(4,4)$, the downward SS-ILRMA with $K=10$ achieved similar performance as $K=2$ (see Fig. 5), indicating that splitting into smaller subbands reduces the size of the problem within each subband and helps the algorithms to work stably.

The downward shift tended to be superior to the upward shift, which was particularly noticeable in SS-ILRMA. That is, starting with the separation of higher frequency bands was more effective. This is likely due to the sparsity (an important cue for many BSS algorithms) of the speech signals in higher frequency bands. To illustrate this, Fig. 6 shows a speech signal used in the experiment along with the (A) highest and (B) lowest subbands, where the subband parameter was set to $(\theta_{W},\theta_{\Delta})=(2,2)$. When using the upward shift, the lowest band (B) must be separated first. However, this subband is relatively dense, making it difficult for sparsity-based BSS methods. Failures at this subband negatively affect the separation of subsequent subbands. On the other hand, when the downward shift is used, the highest subband (A) is separated first. Owing to its sparsity and simple structure (i.e., components concentrated in specific time frames and appearing as vertical patterns), this subband can be successfully separated using IVA and ILRMA. The results from the higher subbands are then propagated to the lower (i.e., more challenging) subbands and help their separation, leading to better outcomes.

Next, we investigate the computational costs of the proposed technique using the same mixtures as in Section 4.4 We run each BSS method 20 times to measure the real-time factor (RTF), i.e., the normalized runtime required to process one second of the signal. To check the implementations, we also tested SS-IVA and SS-ILRMA with $(\theta_{W},\theta_{\Delta})=(1,1)$, which agrees with the Vanilla version of IVA and ILRMA. The total number of updates $J_{\textrm{total}}$ was fixed to 100. The downward shift was used, and the number of bases $K$ was set to $2$ since SS-IVA and SS-ILRMA were found to perform better with this setting (see Section 4.4).

The results are shown in Fig. 7. The conventional OC-IVA and the proposed SS-IVA took slightly longer runtime than Vanilla IVA due to band splitting. Their RTF increased as the subband parameters $\theta_{W}$ and $\theta_{\Delta}$ grew. For ILRMA, the subband splitting was found to reduce the computation time. It was also observed that the runtime in SS-ILRMA was primarily influenced by $\theta_{W}$, which determines the bandwidth $W$, and was less dependent on $\theta_{\Delta}$, which controls the amount of shift $\Delta$.

We also examined the relationship between computational time and separation performance. The total number of updates $J_{\textrm{total}}$ was varied from 0 to 100 in increments of 4. For each method, the best subband parameters $(\theta_{W},\theta_{\Delta})$ in Table 1 were used: $(2,2)$ for OC-IVA, $(2,4)$ for downward SS-IVA, and $(4,2)$ for downward SS-ILRMA with $K=2$.

Figure 8 (a) shows the average separation performance against the total number of updates $J_{\textrm{total}}$. The separation performance of conventional IVA and ILRMA grew gradually, which is due to some mixtures that require many iterations for resolving the permutation problem. On the other hand, the proposed SS-IVA and SS-ILRMA rapidly converged in fewer iterations. SS-IVA reached the ceiling of the separation performance around $J_{\textrm{total}}=12$ $({}=\theta_{\Delta}\times J_{\textrm{inner}}=4\times 3)$, and SS-ILRMA around $J_{\textrm{total}}=8$ $({}=\theta_{\Delta}\times J_{\textrm{inner}}=2\times 4)$. The required number of updates $J_{\textrm{total}}$ (i.e., 12 and 8) is less than half compared to that in the previous literature, e.g., [23], thanks to the ability to avoid erroneous permutation.

Figure 8 (b) shows the average separation performance versus RTF. The proposed SS-IVA and SS-ILRMA were confirmed to converge with shorter runtimes than conventional methods.

From the experimental results so far, we have confirmed that, for speech signals, SS-ILRMA is highly effective. This might be a surprising outcome because ILRMA is known to be unsuitable for speech signals but suitable for music signals. Therefore, we additionally evaluated SS-ILRMA on music signals. The parameters are the same as in Section 4.4

Figure 9 and Table 2 show the separation performance of ILRMA, SS-ILRMA, and FDICA+IPS. When the number of basis $K$ was set to 10, the Vanilla ILRMA performed worse in some cases⁵⁵5 When only evaluating the song ID4 in [23, Table 3] (i.e., guitar and synth in [23, Fig. 12] that investigates the relationship between $K$ and SDRi), the average SDRi of Vanilla ILRMA was 16.24 dB ($K=2$) and 16.62 dB ($K=10$), i.e., $K=10$ outperformed $K=2$, which agrees with [23, Fig. 12]. The poor performance of Vanilla ILRMA with $K=10$ in Fig. 9 is due to the other mixtures, especially ID2 and ID3. . The upward SS-ILRMA was ineffective for music signals, while the downward SS-ILRMA performed robustly for both $K=2$ and $K=10$. In particular, the downward SS-ILRMA with $(K,\theta_{W},\theta_{\Delta})=(10,2,2)$ yielded the best results in terms of the median SDRi, where the worst-case SDRi was improved by 7.51 dB compared to Vanilla ILRMA with $K=10$. Note that downward SS-ILRMA with $(K,\theta_{W},\theta_{\Delta})=(2,2,2)$, $(2,4,2)$, and $(10,2,2)$ even outperformed FDICA+IPS on average, which might be because SS-ILRMA can more precisely model the structure of the power spectrogram (including the relationship among the time-frequency bins) by using NMF, while FDICA handles the frequency bins independently.

The difference between upward and downward shifts was more pronounced in music signals compared to that in speech signals (Section 4.4). This should be because the separation of lower frequency bands in music signals can be significantly challenging for some instruments. As an example, Fig. 10 shows the signal of the guitar used in the experiment. Similar to speech signals, the highest subband (A) is sparse. On the contrary, the lowest subband (B) is even denser than that of speech signals. Therefore, the downward shift, i.e., leveraging the separation result obtained from higher subbands to assist in the separation of lower subbands, is indispensable when using SS-ILRMA for music signals.