Multi-ACCDOA: Localizing and Detecting Overlapping Sounds from the Same Class with Auxiliary Duplicating Permutation Invariant Training

The multi-ACCDOA format is an extension of the single-ACCDOA format [11] to a track dimension; thus, it is a class- and track-wise format. While each track in the track-wise format only detects one event class and a corresponding location, each track in the multi-ACCDOA format is equivalent to the single-ACCDOA format, where a track predicts activities of all target classes and their corresponding locations.

The $N$-track $C$-class $T$-frame multi-ACCDOA format, $\textit{\boldmath$P$}\in{\mathbb{R}}^{3\times N\times C\times T}$, is formulated with activities and Cartesian DOAs. Each sound event class of a track is represented by three nodes corresponding to the sound event location in the $x$, $y$, and $z$ axes. Let $\textit{\boldmath$a$}\in{\mathbb{R}}^{N\times C\times T}$ be activities, whose reference value is ${a}_{nct}^{*}\in\{0,1\}$, i.e., it is 1 when the event is active and 0 when inactive. $n,c,t$ indicates an output track number, a target class, and a time frame. Also, let $\textit{\boldmath$R$}\in{\mathbb{R}}^{3\times N\times C\times T}$ be Cartesian DOAs, where the length of each Cartesian DOA is 1, i.e., $||\textit{\boldmath$R$}_{nct}||=1$ when a class $c$ is active. $||\cdot||$ is the L2 norm. An ACCDOA vector in the multi-ACCDOA format is formulated as follows:

$\displaystyle\textit{\boldmath$P$}_{nct}={a}_{nct}\textit{\boldmath$R$}_{nct}.$

(1)

In this study, to estimate the ACCDOA vectors in the multi-ACCDOA format $\textit{\boldmath$P$}_{nct}$, the CRNN architecture is used as in [11].

Similarly to the track-wise approaches, the multi-ACCDOA format also suffers from the track permutation problem. To overcome this issue, we adopt PIT for the training process. The frame-level PIT [14] is used in this study. Assume all possible permutations constitute a permutation set $\mathrm{Perm}$. $\alpha\in\mathrm{Perm}(t)$ is one possible frame-level permutation at frame $t$. A PIT loss for the multi-ACCDOA format can be straightforwardly written as follows:

	$\displaystyle\mathcal{L}^{\mathrm{PIT}}$	$\displaystyle=\frac{1}{T}\sum_{t}^{T}\min_{\alpha\in\mathrm{Perm}(t)}{l}^{\mathrm{ACCDOA}}_{\alpha,t},$		(2)
	$\displaystyle{l}^{\mathrm{ACCDOA}}_{\alpha,t}$	$\displaystyle=\frac{1}{NC}\sum_{n}^{N}\sum_{c}^{C}\mathrm{MSE}(\textit{\boldmath$P$}_{\alpha,nct}^{*},\hat{\textit{\boldmath$P$}}_{nct}),$		(3)

where $\textit{\boldmath$P$}_{\alpha,nct}^{*}$ is an ACCDOA target of a permutation $\alpha$, and $\hat{\textit{\boldmath$P$}}_{nct}$ is an ACCDOA prediction at track $n$, class $c$, and frame $t$. We refer this PIT as a non-class-wise PIT because the PIT loss in Eq. (3) exclusively assigns one sound event to one track regardless of the sound event class [14]. We use MSE as a loss function for the multi-ACCDOA format [11]. The left in Fig. 2 shows a possible permutation of a 3-track 2-class multi-ACCDOA format with the non-class-wise PIT. The blue and orange vectors indicate ACCDOA vectors of a barking dog and a footstep, respectively.

Because the multi-ACCDOA format has a class dimension, it is reasonable to set a permutation for each class. Therefore, we extend the non-class-wise PIT to a class-wise formulation, namely class-wise PIT. $\beta\in\mathrm{Perm}(ct)$ is one possible class-wise frame-level permutation at class $c$ and frame $t$. The class-wise PIT loss for the multi-ACCDOA format can be written as follows:

	$\displaystyle\mathcal{L}^{\mathrm{PIT}}$	$\displaystyle=\frac{1}{CT}\sum_{c}^{C}\sum_{t}^{T}\min_{\alpha\in\mathrm{Perm}(ct)}{l}^{\mathrm{ACCDOA}}_{\alpha,ct},$		(4)
	$\displaystyle{l}^{\mathrm{ACCDOA}}_{\alpha,ct}$	$\displaystyle=\frac{1}{N}\sum_{n}^{N}\mathrm{MSE}(\textit{\boldmath$P$}_{\alpha,nct}^{*},\hat{\textit{\boldmath$P$}}_{nct}).$		(5)

The center in Fig. 2 shows a permutation example for the class-wise PIT, where a permutation is set for each class.

The class-wise PIT assigns one active event to only one track for each class. When there are fewer active events than tracks in a event class, we need to assign zero vectors as auxiliary targets for the inactive tracks in the class. The class-wise PIT cannot enable every track of the multi-ACCDOA format to learn with the same target as the single one, i.e., a unit vector. As a result, the auxiliary targets for the class-wise PIT, i.e., zero vectors, might interfere the optimization of multi-ACCDOA training.

To enable every track of the multi-ACCDOA format to learn with the same target as the single one, we duplicate the original target as the auxiliary targets of the other tracks instead of zero vectors. We refer to the PIT framework with the duplicated targets as ADPIT. The multi-ACCDOA format tries to output ${M}_{ct}(\leq N)$ original targets and $N-{M}_{ct}$ duplicated targets in class ${c}$ and frame ${t}$, and the PIT framework is applied to find the best permutations. The class-wise ADPIT loss is the same formula as the class-wise PIT in Eq. (4). The difference lies in their permutation set. We show a possible permutation in the right side in Fig. 2 for the class-wise ADPIT. Each track is trained with an original target or a duplicated target.

The possible number of permutations for the class-wise ADPIT in the permutation set, ${K}_{ct}$, is calculated as follows:

$\displaystyle{K}_{ct}=\begin{cases}{}_{N}\mathrm{P}_{{M}_{ct}}\times{{M}_{ct}}^{N-{M}_{ct}}&\text{if ${M}_{ct}>0$,}\\ 1&\text{if ${M}_{ct}=0$,}\end{cases}$

(6)

where ${}_{N}\mathrm{P}_{{M}_{ct}}$ denotes ${M}_{ct}$-permutations of $N$. Fig. 3 shows all permutations for the class-wise ADPIT. In practice, original targets and duplicated targets cannot be distinguished. Therefore, the number of actual permutations is smaller than ${K}_{ct}$.

After the training with the class-wise ADPIT, during inference, duplicated outputs are unified in three steps. First, we use threshold processing to determine whether each output is active or not. Second, if there are two or more active outputs from the same class, we calculate the similarity between the active outputs. In this study, we use angle difference as the similarity. Third, if an angle difference between the outputs is smaller than a threshold, i.e., the outputs are similar, we take the average of the outputs to a unified output.

We used the development set of TAU Spatial Sound Events 2021 - Ambisonic with the suggested setup for DCASE 2021 Task 3 [12]. The dataset contained 600 one-minute sound scene recordings: 400 for training, 100 for validation, and 100 for testing. The sound scene recordings were synthesized by adding sound event samples convolved with room impulse response (RIR) to spatial ambient noise. The sound event samples consisted of 12 event classes such as footsteps and a barking dog. The RIRs and ambient noise recordings were collected at 15 different indoor locations. Each event had an equal probability of being either static or moving. The moving sound events were synthesized with 10, 20, or 40 degrees per second. Up to three overlapping sound events are possible, temporally and spatially. In addition, there are simultaneous directional interfering sound events with their temporal activities, static or moving. Signal-to-noise ratios ranged from 6 to 30 dB.

Following the setup, five metrics were used for the evaluation [19]. The first was the localization error ${LE}_{CD}$, which expresses the average angular distance between the same class’s predictions and references. The second was a simple localization recall metric ${LR}_{CD}$, which tells the true positive rate of how many of these localization estimates were detected in a class out of the total number of class instances. The following two metrics were the location-dependent error rate ${ER}_{20^{\circ}}$ and F-score ${F}_{20^{\circ}}$, where predictions were considered true positives only when the distance from the reference was less than $20^{\circ}$. We also adopted an aggregated SELD error, $\rm{\mathcal{E}_{SELD}}$, which is defined as

$\displaystyle\rm{\mathcal{E}_{SELD}}=\frac{\rm{{ER}_{20^{\circ}}}+(1-\rm{{F}_{20^{\circ}}})+\frac{\rm{{LE}_{CD}}}{180^{\circ}}+(1-\rm{{LR}_{CD}})}{4}.$

(7)

The sampling frequency was set to 24 kHz. The short-term Fourier transform (STFT) was applied with 20 ms frame length and 10 ms frame hop. Input features are segmented to have a fixed length of 1.27 seconds. The shift length was set to 0.2 seconds during inference. We used a batch size of 32, and each training sample was generated on the fly [20]. We used the Adam optimizer with a weight decay of $10^{-6}$. We gradually increased the learning rate to 0.001 with 50,000 iterations [21]. After the warm-up, the learning rate was decreased by 10% if the SELD error of the validation did not improve in 40,000 consecutive iterations. We validated and saved model weights in every 10,000 iterations up to 400,000 iterations. Finally, we applied stochastic weight averaging (SWA) [22] to the last 10 models. The threshold for activity is 0.5 to binarize predictions during inference.

We compared the multi-ACCDOA format with the single one. The number of tracks in the multi-ACCDOA format was fixed at the maximum number of overlaps in the dataset, i.e., $N=3$. The difference in network architecture between the multi- and single-ACCDOA formats was only the final fully-connected layer. We compared three PITs for the multi-ACCDOA format: the non-class-wise PIT, the class-wise PIT, and the class-wise ADPIT. We also investigated the threshold of unification from 15 to 45 degrees for inference. Other configurations mostly followed our DCASE 2021 challenge settings [23] in all methods. Multichannel amplitude spectrograms and inter-channel phase differences (IPDs) were used as features. Three data augmentation methods were applied: equalized mixture data augmentation (EMDA) [24, 25], rotation in the first-order Ambisonic (FOA) [26], and SpecAugment [27]. A conventional CRNN [10] was used for the experiments.

The multi-ACCDOA format was compared with the track-wise method called EINV2 [14]. We prepared the original EINV2 architecture [14]. Since the architecture uses MHSA blocks, we also replace GRU layers with MHSA blocks for the multi-ACCDOA format. We called the architecture CNN-MHSA.

Then we compared the proposed method with state-of-the-art SELD methods without model ensembles. Although model ensemble techniques effectively achieve high performance, we omitted this for pure comparison between single models. We used RD3Net [11] for comparison. We used cosIPDs, and sinIPDs [28], instead of the IPDs for this model. The STFT was applied with 40 ms frame length and 20 ms frame hop, and the input length was set to 5.11 seconds.

Table 1 summarizes the performance in the single- and multi-ACCDOA formats. The multi-ACCDOA format with the class-wise ADPIT outperformed the single one for all metrics with only a few increases in parameters. The multi-ACCDOA format with the class-wise ADPIT showed 7.2 points higher ${LR}_{CD}$ than the single one in the cases with overlaps from the same class, while the multi-ACCDOA format with the class-wise ADPIT performed comparably in the other cases. The result shows the threshold value slightly affected the performance, especially ${LR}_{CD}$. The result also shows that conventional PITs made it difficult to train the multi-ACCDOA format since the PITs cannot allow each track of the multi-ACCDOA format to learn with the same target as the single one. It is also shown that the multi-ACCDOA format with the class-wise ADPIT can successfully localize and detect the overlaps from the same class and maintain the performance in the other cases.

Table 2 summarizes the performance in the multi-ACCDOA format trained with the class-wise ADPIT and the track-wise format. The multi-ACCDOA format performed better in $\rm{{ER}_{20^{\circ}}}$, $\rm{{F}_{20^{\circ}}}$, and $\rm{\mathcal{E}_{SELD}}$. The result shows that the multi-ACCDOA format performed comparably to the track-wise format with considerably fewer parameters.

Table 3 shows the performances of the SELD methods without model ensembles. The RD3Net using the multi-ACCDOA format trained with the class-wise ADPIT performed the best in $\rm{{ER}_{20^{\circ}}}$, $\rm{{F}_{20^{\circ}}}$, and $\rm{\mathcal{E}_{SELD}}$ for the DCASE 2021 Task 3 development set.