Audio Difference Captioning
Utilizing Similarity-Discrepancy Disentanglement

The audio feature extractor uses a pre-trained audio embedding model to calculate audio embedding vectors. Two audio clips $\boldsymbol{x}$ and $\boldsymbol{y}$ are the input, and the audio embedding vectors corresponding to the clips $\boldsymbol{X}\in\mathbb{R}^{H\times T_{x}}$ and $\boldsymbol{Y}\in\mathbb{R}^{H\times T_{y}}$ are the output, where $H$ is the size of hidden dimension, $T_{x}$ is the time length of $\boldsymbol{X}$, and $T_{y}$ is the time length of $\boldsymbol{Y}$

The difference encoder extracts information about the differences between the two audio clips from audio embedding vectors $\boldsymbol{X}$ and $\boldsymbol{Y}$. To extract difference information efficiently, we utilize a cross-attention-concentrated (CAC) transformer encoder as the main function of the difference encoder. The CAC transformer encoder utilizes the masked multi-head attention layer, allowing only mutual cross-attention between two audio clips by the attention mask illustrated in the upper right of Fig. 2.

The detailed procedure is as follows. First, special tokens that indicate the order of the audio clips $\mathcal{X}\in\mathbb{R}^{H\times 1}$ and $\mathcal{Y}\in\mathbb{R}^{H\times 1}$ are concatenated at the beginning of $\boldsymbol{X}$ and $\boldsymbol{Y}$, respectively. Next, these two sequences are concatenated to make the input of the difference encoder $\boldsymbol{Z}$ like $\boldsymbol{Z}=[\mathcal{X},\boldsymbol{X},\mathcal{Y},\boldsymbol{Y}]$. Then, positional encoding $\mathscr{P}$ is applied to $\boldsymbol{Z}$. Finally, $\mathscr{P}(\boldsymbol{Z})$ is input to CAC transformer encoder to obtain the output $\boldsymbol{\hat{Z}}=[\hat{\mathcal{X}},\boldsymbol{\hat{X}},\hat{\mathcal{Y}},\boldsymbol{\hat{Y}}]$.

The transformer decoder is utilized as a text decoder like as [5]. The text decoder calculates word probability from the output of the difference encoder $\boldsymbol{\hat{Z}}$.

The similarity-discrepancy disentanglement (SDD) loss function is an auxiliary loss function aimed at obtaining a difference-emphasized audio representation. When there is an explainable difference between two audio clips, these clips consist of similar and discrepant parts. To introduce this hypothesis, we design contrastive learning to bring similar parts closer and keep discrepant parts. We propose two types of implementations that apply SDD to the input of the difference encoder $\boldsymbol{Z}$ or the output of it $\boldsymbol{\hat{Z}}$, as shown in Fig. 2, and call the former and latter implementations early and late disentanglement, respectively.

We explain the procedure in the case of early disentanglement. Note that the case of late disentanglement only replaces $\boldsymbol{Z}$ with $\boldsymbol{\hat{Z}}$. First, $\boldsymbol{Z}$ is split along the hidden dimension and assigned to similar and discrepant parts like in the upper left illustration of Fig. 2. If $\boldsymbol{Z}\in\mathbb{R}^{H\times(T_{x}+T_{y}+2)}$, $\boldsymbol{Z}$ is split into similar part $\boldsymbol{Z}_{\rm S}$ and discrepant part $\boldsymbol{Z}_{\rm D}$ like

	$\displaystyle\boldsymbol{Z}_{\rm S}$	$\displaystyle=[\mathcal{X}_{\rm S},\boldsymbol{X}_{\rm S},\mathcal{Y}_{\rm S},\boldsymbol{Y}_{\rm S}]\in\mathbb{R}^{(H/2)\times(T_{x}+T_{y}+2)},$		(1)
	$\displaystyle\boldsymbol{Z}_{\rm D}$	$\displaystyle=[\mathcal{X}_{\rm D},\boldsymbol{X}_{\rm D},\mathcal{Y}_{\rm D},\boldsymbol{Y}_{\rm D}]\in\mathbb{R}^{(H/2)\times(T_{x}+T_{y}+2)}.$		(2)

Then, the SDD is performed by $\mathcal{L}_{\rm SDD}=\mathcal{L}_{\rm S}+\mathcal{L}_{\rm D}$, where

	$\displaystyle\mathcal{L}_{\rm S}={\rm SymInfoNCE}(\Phi([\mathcal{X}_{\rm S},\boldsymbol{X}_{\rm S}]),\Phi([\mathcal{Y}_{\rm S},\boldsymbol{Y}_{\rm S}])),$		(3)
	$\displaystyle\mathcal{L}_{\rm D}={\rm PairCosSim}(\Psi([\mathcal{X}_{\rm D},\boldsymbol{X}_{\rm D}]),\Psi([\mathcal{Y}_{\rm D},\boldsymbol{Y}_{\rm D}])),$		(4)

${\rm SymInfoNCE}$ is the symmetric version of the InfoNCE loss used in [19], ${\rm PairCosSim}$ is the cosine similarity for each correct data pair, $\Phi$ and $\Psi$ are embedding networks consisting of the bidirectional-LSTM and average pooling, and $\mathcal{L}_{\rm SDD}$ is the final value of the SDD loss function. That is, the SDD loss function views $[\mathcal{X}_{\rm S},\boldsymbol{X}_{\rm S}]$ and $[\mathcal{Y}_{\rm S},\boldsymbol{Y}_{\rm S}]$ as similar parts and brings them closer by using $\mathcal{L}_{\rm S}$ and views $[\mathcal{X}_{\rm D},\boldsymbol{X}_{\rm D}]$ and $[\mathcal{Y}_{\rm D},\boldsymbol{Y}_{\rm D}]$ as discrepant parts and keeps them apart by $\mathcal{L}_{\rm D}$.

The entire loss function $\mathcal{L}$ is the weighted sum of cross-entropy loss for word prediction $\mathcal{L}_{\rm CE}$ and the SDD: $\mathcal{L}=\mathcal{L}_{\rm CE}+\lambda\mathcal{L}_{\rm SDD}$, where $\lambda$ is a weighting parameter.

The constructed AudioDiffCaps dataset consists of (i) pairs of similar but slightly different audio clips and (ii) human-annotated descriptions of their differences.

The pairs of audio clips were artificially synthesized by mixing foreground event sounds with background sounds taken from existing environmental sound datasets (FSD50K [15] and ESC-50 [11]) using the Scaper library for soundscape synthesis and augmentation [20]. We used the same mixing procedure as our previous work [21]. Data labeled rain or car_passing_by in FSD50K was used as background, and six foreground event classes were taken from ESC-50 (i.e., data labeled dog, chirping_bird, thunder, footsteps, car_horn, and church_bells). Each created audio clip was 10 seconds long. The maximum number of events in one audio clip was two, with 0-100% overlap (no overlap-range control applied). Each foreground event class had 32 or 8 instances in the development or evaluation set, respectively. Similar to previous work, we focused on the three types of difference: increase/decrease of background sounds, increase/decrease of sound events, and addition/removal of sound events. The development and evaluation sets contained 5996 and 1720 audio clip pairs, respectively. (That is, development and evaluation sets contained 11992 and 3440 audio clips.)

The human-annotated descriptions were written as instruction forms explaining ”what and how” to change the first audio clip to create the second audio clip. In the preliminary study, we found that declarative sentences, in some cases, tend to use ordinal numbers such as “First sound is louder than second sound”. Since these cases do not express what the actual difference is, the AudioDiffCaps dataset uses instruction forms with a fixed direction of change from the first audio clip to the second one, e.g., ”Make the rain louder” ²²2The dataset is available at https://github.com/nttcslab/audio-diff-caps. . A wider variety of descriptions explaining the same concept, such as declarative sentences, should be included in future works. The presentation order of the pair to the annotator was randomly selected. Annotators were five naïve workers remotely supervised by an experienced annotator. Each pair of audio clips in the development set had between 1 and 5 descriptions (a total of 28,892) while each pair in the evaluation set had exactly five descriptions assigned to it (a total of 8600).

We used 10% of the development set for validation. The optimizer was Adam [22]. The number of epochs was 100. We used the BLEU-1, BLEU-4, METEOR, ROUGE-L, CIDEr [23], SPICE [24], and SPIDEr [25] as evaluation metrics. They were also used for conventional audio captioning[26].

We used BYOL-A [27], a pre-trained audio embedding model, as the audio feature extractor in our ADC implementation, and we fine-tuned the BYOL-A throughout experiments. The transformer encoder and decoder used the official implementation of PyTorch. The number of layers was 1. The hidden size was 768. The number of heads was 4. The activation was RELU. The dimension of the feedforward layer was 512. The dropout rate was 0.1. For the attention mask of the transformer encoder, we compared two types; one with the proposed cross-attention mask and the other without a mask. The text decoder used the teacher forcing algorithm during training and the beam search algorithm [28, 29] during inference. The value of $\lambda$ was empirically set to $0$, $0.5$, $1.0$, or $2.0$.

The results of evaluation metrics are shown in Table 1, where bold font indicates the highest score, “Mask” and “Disent.” indicate the attention mask utilized in the transformer encoder and input of SDD loss function, respectively. When the CAC transformer encoder was evaluated by comparing the two lines above, the proposed method had superior or equivalent scores to the conventional method in all evaluation metrics. There was no significant difference in the evaluation metrics related to the degree of matching with single-word references, such as BLEU-1. One likely reason is that the scores above a certain level can be obtained by outputting words in arbitrary sentences, such as “a” and “the” in these metrics. In contrast, the scores of BLEU-4, ROUGE-L, CIDEr, and SPIDEr, affected by the accuracy of consecutive words, were improved using the proposed cross-attention mask. Therefore, the proposed cross-attention mask was thought to make the feature extraction of differences more efficient and simplify the training of the text decoder. As a result, phrase-level accuracy was improved.

The effect of SDD was verified from the results of the second to eighth lines. The results in (a) and (b) were the conventional transformer without cross attention mask or SDD loss and the CAC transformer without SDD loss ($\lambda=0$) Ones from (c) to (h) were the result when using early/late disentanglement. Since the scores of BLEU-4, ROUGE-L, CIDEr, and SPIDEr improved under all conditions comparing (b) and others, the SDD loss function was effective for the audio difference captioning task. The improvement in the case of late disentanglement (f), (g), and (h) was remarkable, and the results obtained the best scores in all evaluation metrics with late disentanglement. In other words, it was essential to use the information to be compared to decompose the similar part and the different parts in the feature amount space. That corresponds to the difference determined depending on the comparison target.

Fig. 3 shows one of the evaluation data and estimated caption and attention weight of the transformer encoder from each system. The leftmost colomn is the Mel-spectrogram of the two input audio clips and one of the reference captions. The three on the right are the attention weight of the transformer encoder and output caption, where the attention weight shows the average of multiple heads. The audio clips on the left and above the weights correspond to the input and memory of the transformer, respectively. The area colored pink and yellow on the weights corresponds to the dog barking. Since there was a difference in the loudness of the dog barking between the two clips, the attention was expected to focus on areas where pink and yellow overlap to extract the difference.

First, in (a), since the attention weight was not constrained, it was also distributed widely to areas other than the above compared with the other two. On the other hand, the attention weights of (b) and (h) concentrated on areas where pink and yellow overlap since the attention of the same input and memory was unavailable. Comparing (b) and (h), while the attention of the part containing the barking of the dog in the memory was large at any time-frame in (b), more attention was paid to the pink and yellow overlapping areas where both input and the memory contain the barking of the dog in (h). Since the late disentanglement required that similar and discrepant parts be retained in the output of the transformer encoder calculated using these attention weights, it was thought that the late disentanglement induced attention to be paid to the part where there was a difference when comparing the two sounds instead of paying attention to the parts that are likely to exist the difference compared with the distribution of training data, such as a dog barking.