A Mutual learning framework for Few-shot Sound Event Detection

Assume we are given a training set, $X_{\mathit{base}}=\left\{\boldsymbol{x}_{i},\boldsymbol{y}_{i}\right\}^{N_{\mathit{base}}}_{i=1}$, where $\boldsymbol{x}_{i}$ denotes the acoustic feature of example $i$ and $\boldsymbol{y}_{i}$ denotes associated one-hot label. Let $Y_{\mathit{base}}$ denote the label set of this base dataset. The few-shot learning assumes that we are given a test dataset: $X_{\mathit{test}}=\left\{\boldsymbol{x}_{i},\boldsymbol{y}_{i}\right\}^{N_{\mathit{test}}}_{i=1}$, with a completely new label set $Y_{test}$ such that $Y_{\mathit{base}}\cap Y_{\mathit{test}}=\emptyset$, and the test dataset only has a few labelled examples.

Transductive inference (TI) is about reasoning from observed, specific (training) cases to specific (test) cases. In this paper, the core idea of TI is about leveraging the statistics of the unlabeled data. More specifically, we adapt the idea from [14], which maximizes the mutual information (MI) between the query features and their label predictions for a few-shot task at inference. It means that the model has seen these unlabeled data before making final prediction.

The overview of the mutual learning framework is shown in Figure 2. In this section, we first introduce how to use class prototypes to build a classifier and update the classifier with transductive inference. After that, we discuss how to make use of the updated class prototypes to fine-tune the feature extractor. Lastly, we summarize the core idea of mutual learning framework.
Building classifier For a given few-shot task, with a support set $S$ and a query set $Q$, let $X$ denote the random variables associated with the acoustic features within $S\cup Q$ and let $Y=\left\{1,2,...,K\right\}$ be the random variables associated with the labels. Let $f_{\phi}:X\rightarrow Z\subset R^{d}$ denote the encoder (i.e., feature extractor) function of a deep neural network, where $\phi$ denotes the trainable parameters, and $Z$ stands for the set of embedded features. The encoder is firstly trained from the base training set $X_{\mathit{base}}$ using the standard cross-entropy loss. Next, for each specific few-shot task, we define a classifier, parametrized by a weight matrix $\boldsymbol{W}=[\boldsymbol{w}_{1},...,\boldsymbol{w}_{K}]\in R^{K\times d}$. The posterior distribution over labels given features is defined by $p_{ik}=\mathbb{P}(Y=k|X=\boldsymbol{x}_{i};\boldsymbol{W},\phi)$. The marginal distribution over query labels is defined by $\hat{p}_{k}=\mathbb{P}(Y_{Q}=k;\boldsymbol{W},\phi)$. $p_{ik}$ and $\hat{p}_{k}$ are calculated as formula (1).

$$p_{ik}=\frac{\mathit{exp}(\boldsymbol{w}_{k}\cdot\boldsymbol{z}_{i})}{\sum_{c=1}^{K}\mathit{exp}(\boldsymbol{w}_{c}\cdot\boldsymbol{z}_{i})},\hat{p}_{k}=\frac{1}{Q}\sum_{i\in Q}p_{ik}$$

(1)

where $\boldsymbol{z}_{i}=\frac{f_{\phi}(\boldsymbol{x}_{i})}{||f_{\phi}(\boldsymbol{x}_{i})||_{2}}$ denotes L2-normalized embedded features. For each task, weights $\boldsymbol{W}$ are initialized by the class prototypes of the support set, as follows

$$\begin{split}\boldsymbol{w}_{k}=\frac{1}{|S_{k}|}\sum_{\boldsymbol{x}_{i}\in S_{k}}f_{\phi}(\boldsymbol{x}_{i})\end{split}$$

(2)

In this paper, we only need to judge whether the audio frame is a positive sample, so we set $K=2$.
Updating classifier To update the weight matrix $\boldsymbol{W}$, for each single few-shot task, we propose a loss function with two complementary terms: (1) a standard cross-entropy loss on the support set; (2) a mutual-information loss, which includes a conditional entropy loss and a marginal entropy loss.

$$\begin{split}L_{w}=\lambda_{\mathit{CE}}\cdot\mathit{CE}-\hat{I}(X_{Q};Y_{Q})\\ \end{split}$$

(3)

$$\begin{split}\mathit{CE}=-\frac{1}{|S|}\sum_{i\in S}\sum_{i=1}^{K}y_{ik}log(p_{ik})\\ \end{split}$$

(4)

$$\begin{split}\hat{I}(X_{Q};Y_{Q})=-\sum_{k=1}^{K}\hat{p}_{k}log\hat{p}_{k}+\frac{1}{|Q|}\sum_{i\in Q}\sum_{k=1}^{K}p_{ik}log(p_{ik})\end{split}$$

(5)

where $\mathit{CE}$ denotes the cross entropy loss function, $y_{ik}$ denotes the true label of the sample in the support set, $p_{ik}$ denotes the prediction result. In our experiments, $\lambda_{\mathit{CE}}$ is set as 0.1. $\hat{I}(X_{Q};Y_{Q})$ denotes the mutual information between the query samples and their latent labels. It is a combination of two terms, the first term is the empirical label-marginal entropy, denoted as $\hat{H}(Y_{Q})$, while the second term is an empirical estimate of the conditional entropy of labels given the query acoustic features, denoted as $\hat{H}(Y_{Q}|X_{Q})$. $\hat{H}(Y_{Q}|X_{Q})$ aims at minimizing the uncertainty of the posteriors at unlabelled query samples, thereby encouraging the model to output confident predictions. This entropy loss is widely used in the context of semi-supervised learning (SSL) [17, 18], as it models effectively the cluster assumption: the classifier’s boundaries should not occur at dense regions of the unlabelled features. The label-marginal entropy regularizer $\hat{H}(Y_{Q})$ encourages the marginal distribution of labels to be uniform.
Note that we only update the weight matrix $\boldsymbol{W}$ in this step, while the feature extractor is fixed. Our experimental results also show that simultaneously updating feature extractor $f_{\phi}$ and weight matrix $\boldsymbol{W}$ does not offer better performance.
Updating feature extractor Previous works [11, 12] have shown that a task-specific feature extractor works better than a task-agnostic one, so we expect our feature extractor to be task-specific. To achieve this, we propose a novel method to update feature extractor, which uses the updated class prototypes as supervision information to fine-tune the feature extractor. In addition, we plan to make use of the predicted results for unlabelled data. Figure 2 shows the updating process of our method. When we finish step 3 and 4, we make use of $\hat{\boldsymbol{W}}$ and predicted results of high confidence to fine-tune the feature extractor $f_{\phi}$. The loss function has two terms as formula (6) shows, including a cross-entropy (CE) loss according to pseudo label and a contrastive loss.

$$\begin{split}L_{f}=\lambda_{1}*\mathit{CE}+\lambda_{2}*L_{c}\end{split}$$

(6)

where $\lambda_{1}$ and $\lambda_{2}$ are hyper-parameters. In our experiments, $\lambda_{1}=\lambda_{2}=0.5$. The contrastive loss $L_{c}$ is defined as follows.

$$\begin{split}L_{c}=-log(\frac{\mathit{exp}(\mathit{sim}(\hat{\boldsymbol{w}}[1],\boldsymbol{\bar{z}}_{\mathit{pos}}))}{\sum_{k=1}^{N}\mathit{exp}(\mathit{sim}(\hat{\boldsymbol{w}}[1],z_{\mathit{neg}}^{k}))})\end{split}$$

(7)

where $\hat{\boldsymbol{w}}[1]$ denotes the first row vector of $\hat{\boldsymbol{W}}$, and it represents the prototype of positive class in our experiments. $\boldsymbol{\bar{z}}_{\mathit{pos}}$ denotes the mean of the learned representation of the positive samples on the support set, and $z_{\mathit{neg}}$ denotes the learned representation of negative samples. $N$ denotes the number of negative samples. In our experiments, $\mathit{sim}$ stands for cosine similarity. We do not use $\hat{\boldsymbol{w}}[2]$ for the reason that the negative sample is randomly chosen. This loss function also can be viewed as knowledge distillation [19].
Mutual learning According to previous discussion, we can make use of transductive inference to improve the classifier, and we can also improve feature extractor by the updated classifier and pseudo label. After we get a better feature extractor, we can continue running the previous process to update the classifier. It means that feature extractor and classifier can learn from each other, so we name it as mutual learning.

Dataset The dataset is from DCASE2021 task 5 [20], including development and evaluation sets. The development set is pre-splitted into training and validation sets. The training set contains about 14 hours of audio, and the validation set contains 5 hours of audio. The evaluation set consists of 31 audio files acquired from different bioacoustic sources.

Table 1 shows the experimental results. Our method achieves 38.4 $\%$ F-score on evaluation set, which significantly outperforms the baseline [6]. TI denotes we only use transductive learning to update classifier (class prototypes). TI-ML denotes we use mutual learning framework to update classifier and feature extractor. TI-ML performs better than TI, which shows the effectiveness of our mutual learning framework. Anderson et al. [9] also applied ProtoNet, and compared with baseline [6], their model utilized both per-channel energy normalisation (PCEN) on the front end and three data augmentation methods. In contrast, our approach does not need to employ these strategies and gets a better result. In addition, Tang et al. [10] got a very close result with us on the evaluation dataset, but they used a 12-layer ResNet pre-trained on AudioSet [22] as the backbone.

In this part, we discuss the influence of transductive learning and mutual learning. The experiments were carried out on the development set.
Influence of each term on formula (3) We now assess the impact of each term in formula (3). The results are reported in Table 2. We observe that integrating the two terms in our loss consistently outperforms any other configuration. None indicates that we do not update classifier $\boldsymbol{W}$, otherwise we directly use the class prototypes to initialize the classifier, and then use it to predict results. We first analyze their effect on transductive inference (TI). If we do not update the classifier, the F-score is 26.38%. The performance is lower than baseline [6] for the reason that we do not use the meta-learning training strategy. On the contrary, when we use either cross entropy loss or mutual information loss to update the classifier, the performance will be significantly improved. We can find that only using cross-entropy loss brings improvement. It proves that insufficient support set leads to the problem that class prototypes cannot represent the true class center, because class prototypes cannot even properly classify the support set. Secondly, we analyze their effect on the mutual learning (TI-ML). If we use the classifier which is not updated to fine-tune feature extractor, we will get the worst results. If we only use the cross entropy loss to update the classifier, it brings a small improvement. Compared with the cross entropy loss, the mutual information loss has more advantages on mutual learning. The best results can be obtained when both two terms are used.
Influence of each term on formula (6) To fine-tune the feature extractor, we devise a loss function as shown in (6) which includes a cross the entropy loss and a contrastive loss. If we only use the cross entropy loss, the result is 51.64%. If the contrastive loss is used alone, the result is 51.06%. The best result (55.26%) can be obtained when both are used, which shows that these two loss terms are both important.
Influence of iterations Table 3 reports the impact of iterations for mutual learning. Here, the iteration equals to 0 means we do not update the feature extractor. Experimental results indicate updating it only once offers the best F-score. We find that the performance tends to decline when the number of iterations increase, one of the reasons is that the number of negative samples is far more than that of positive samples in this dataset, which makes the model learn more information about negative samples, so the result shows higher False Positives (FP) rate.