After: Active Learning Based Fine-Tuning Framework for Speech Emotion Recognition

Given speech datasets $\mathcal{D}$ = $\{(\mathbf{x}_{i},y_{i})\}_{i=1}^{N}$ where $\mathbf{x}_{i}$ represents the $i$-th speech signal and $y_{i}$ represents its corresponding emotion label. We aim to fine-tune a pre-trained automatic speech recognition model $\mathbf{M}$, such as wav2vec 2.0 [29], on labeled speech datasets $\mathcal{D}$ to obtain accurately predicted emotion labels for all speech signals.

Here we introduce our TAPT component in detail. To better leverage pre-trained prior knowledge for the benefit of downstream tasks, Gururangan et al. [28] continue training the pre-trained model RoBERTa [30] on downstream datasets via the same loss of the pre-training task (reconstruct the masked tokens [17]) and significantly improved the performance on text classification. Inspired by their work, we added an extra step to After by continuing the wav2vec 2.0 training speech recognition model on downstream SER’s training datasets while keeping the same loss function with wav2vec 2.0 unchanged. By conducting this process, we bridge the information gap between the pre-trained ASR task and the target SER task, which we have preliminarily confirmed helps according to our experiments in Section 3.2.

As shown in Figure 1 (a), the wav2vec 2.0 model $\mathbf{M}$($\mathbf{W}_{0}$), with pre-trained weights $\mathbf{W}_{0}$, consists of three sub-modules: the feature encoder module, the transformer module and the quantization module. Specifically, we use a CNN-based encoder to encode the $i$-th input unlabeled speech signals into low-dimensional vectors as $\mathbf{x}_{i}$. Then, we randomly mask 15%pt. features (following BERT [17]) of the speech vectors and decode them with two decoders to obtain quantized and context representations, where the quantization decoder can decode continuous speech vectors $\mathbf{x}_{i}$ into discrete code word from phonemes codebooks ¹¹1A quantized codebook refers to a set of predetermined values or codewords used to represent a continuous signal in a discrete form [18]. as $\mathbf{z}_{i}^{q}$ and wav2vec 2.0 decoder (transformer layers) can use self-attention to decode continuous speech vectors $\mathbf{x}_{i}$ into context-aware representations $\mathbf{z}_{i}^{c}$. Then, we design contrastive loss [18] (cl) to minimize the differences between quantized and context representations as

where the temperature hyperparameter $\kappa$ is set to 0.1, and $\text{sim}(\mathbf{a},\mathbf{b})=\mathbf{a}^{T}\mathbf{b}/\|\mathbf{a}\|_{2}\|\mathbf{b}\|_{2}$ with $T$ representing the transposition of a vector. Eq. (1) can help to obtain better quantized and context representations because two decoders can provide highly heterogeneous contexts for each speech signal [31].

To minimize the information gap between the pre-trained model and downstream SER task, following BERT [17], we first randomly mask 15%pt. tokens of each speech signal and then use reconstruction loss on the corrupted downstream SER dataset to generate tokens for reconstructing the original data, which can be formulated as

where $N_{m}$ is the number of masked tokens, $s^{\text{true}}_{i}$ and $s_{i}^{\text{predicted}}$ are the ground-truth and predicted token probability of the $i$-th masked token.

Finally, we combine contrastive loss and reconstruction loss for the TAPT process as

Although Pseudo-TAPT [29] also adopts TAPT, they spent much time using K-means to extract frame-level emotional pseudo labels and continually pre-train their model in a supervised manner. However, K-means is sensitive to the initial value and outliers [32], making Pseudo-TAPT unstable and computationally expensive.

After the TAPT process, we obtain the model $\mathbf{M}_{\textrm{TAPT}}(\mathbf{W}_{0}^{{}^{\prime}})$ with $\mathbf{W}_{0}^{{}^{\prime}}$ as the weight initialization for the AL process (cf. Line 1 of Algorithm 1). A typical AL setup starts by treating $\mathcal{D}$ as a pool of unlabeled data $\mathcal{D}_{\textrm{pool}}$ and performs $\tau$ iterations of sample selection. Specifically, in the $i$-th iteration, $k$ samples $Q_{i}$ = $\{\mathbf{x}_{1},\cdots,\mathbf{x}_{k}\}$ are selected using a given acquisition function $\mbox{ac}()$. For example, we adopt Entropy [33] as $\mbox{ac}()$ to measure the uncertainty of the samples and select the most uncertain $k$ samples. These selected samples are then labeled and added to the $i$-th training dataset $\mathcal{D}^{i}_{\textrm{train}}$, with which a model is fine-tuned for SER.

One primary goal of After is to explore whether AL strategies can reduce the number of annotation samples, as labeling large-scale datasets is the most laborious part of SER. Thus, for simplicity, we adopt five of the most well-known and influential AL strategies for evaluation, including Entropy [33], Least Confidence [34], Margin Confidence [34], ALPs [35] and BatchBald [36]. These methods use different criteria to help select the most uncertain and informative samples from $\mathcal{D}_{\textrm{pool}}$. For example, we can adopt entropy to measure the uncertainty of $\mathbf{x}_{i}$ as

where $c$ is the number of emotional classes and $P(y_{j}|\mathbf{x}_{i})$ represents the predicted probability of $\mathbf{x}_{i}$ for the $j$-th emotion.

And we select the number of $k$ most uncertainty samples for annotation and add them to the training dataset $\mathcal{D}_{\textrm{train}}$. Traditional AL methods use random initialization, while we found that the AL methods are sensitive to initialized samples and easily select redundant samples or outliers in each AL iteration with a bad initialization. Thus, instead of directly using AL methods, we propose a clustering-based initialization for all AL methods (we use K-means in this study) and obtain better performance (details about K-means are given in Sec 3.2). Please note that, as shown in Algorithm 1, clustering-based initialization is only applied in the initialization process once, and the subsequent AL loop does not need a K-means process.

As shown in Figure. 1 (b), we add a task-specific classification layer with additional parameters $\mathbf{W}_{c}$ for emotion recognition on the top of wav2vec 2.0. We fine-tune the classification model $\mathbf{M}_{i}([\mathbf{W}_{0}^{{}^{\prime}},\mathbf{W}_{c}])$ in each AL iteration with all labeled samples in $\mathcal{D}_{\textrm{train}}$ (cf. Lines 6-10 of Algorithm 1). We use the cross-entropy loss for the emotion recognition classifier:

where $c$ is the number of emotion classes, $k$ is the number of slected samples at $t$-th iteration, $\hat{y}_{i}^{j}$ is the $i$-th predicted label, and $y_{i}^{j}$ is the $i$-th ground-truth of $j$-th class.

As shown in Figure 1 (c), After incorporates an AL strategy for sample selection. To identify the most suitable AL method for After, we combined it with multiple well-known AL methods and evaluated their performance. Furthermore, we found that AL methods are sensitive to initialization, with most AL methods randomly selecting 1%pt. samples for initialization [48]. Unlike them, we propose a novel clustering-based (K-means) initialization method to improve the performance of SER. Specifically, we first extract sample representations of training data from the wav2vec 2.0 CNN-based encoder. Then, we employed K-means on the training data and selected 1%pt. samples closest to the cluster centres as our initialized samples. Please note that we used elbow method [49] to determine the number of clusters for K-means automatically, and we used the Euclidean distance to measure the distance between sample representations.

Figure 2 demonstrates that the clustering-based initialization outperforms the random initialization for all AL methods. The initial set of samples can well influence the selection order of samples in each iteration of AL, and an effective initialization can significantly enhance the performance and stability of AL methods. Figure 2 illustrates that Entropy+Clustering is the most effective AL strategy for After on the Merge dataset. Although we only show the diagram for UA due to space constraints, the diagram for WA is similar to UA. Therefore, Entropy+Clustering is selected as the primary AL method for After. And we recommend using Entropy+Clustering, the simplest yet most efficient strategy for real-world applications.

We analyzed the relationship between the ratio of labeled samples, performance, and time consumption of After. Results in Table 2 show that both performance and time consumption of After increased as the ratio of labeled samples increased. Our findings indicate that using 20%pt. labeled samples yielded a significant improvement in performance while reducing the time consumption by 79%pt. compared to fine-tuning on 100%pt. samples. Thus, we selected 20%pt. labeled samples as a trade-off between performance and time consumption for subsequent experiments.

Table 3 displays the primary results of After and the baselines on two datasets regarding unweighted and weighted accuracy. After outperforms all baselines with only 20%pt. labeled samples for fine-tuning. Specifically, After improves UA and WA by 2.38%pt. and 0.36%pt. respectively, compared to the SOTA baseline (UA of Pesdo-TAPT and WA of GLAM), on the Merged dataset. And After improves UA and WA by 8.45%pt. and 4.12%pt. respectively, compared to SOTA baseline (UA of GLAM and WA of Light), on the Merged dataset.

LSSED [43] and RH-emo [44] achieved good results with the IEMOCAP but performed poorly with the Merged dataset. This discrepancy may be attributed to their limited denoising and domain transfer capabilities, preventing them from effectively handling the noise in the Merged dataset. On the other hand, GLAM [42] and Light [15] employed multi-scale feature representations and deep convolution blocks to capture high-level global data features. They are beneficial for filtering out noisy low-level features and enhancing performance on both datasets. Psedp-TAPT [29] improved model robustness by using K-means to capture higher-level frame emotion labels as pseudo labels for supervised TAPT. Although baselines can eliminate the noise of the dataset to a certain extent, they exhibited high time complexity during fine-tuning with large-scale datasets. They did not effectively bridge the gap between pre-training and the downstream SER task. In contrast, After uses unsupervised TAPT to mitigate the information gap between the source domain (ASR) and the target (SER) domain. Additionally, After selects a subset of the most informative and diverse samples for iterative fine-tuning, which has three advantages: Firstly, it reduces labour consumption for manually labelling large-scale SER samples; Secondly, by utilizing a smaller labeled dataset, After significantly reduces the overall time consumption (Figure 3), making it practical and feasible for real-world applications; Finally, the iterative fine-tuning process employed by After improves performance and stability by eliminating noise and outliers present in the selected samples, leading to enhanced overall model performance in SER tasks.

We performed an additional ablation study to evaluate the efficacy of After, as shown in Table 4. Specifically, we conducted fine-tuning (FT) and TAPT+FT on random sample selection and AL-based (Entropy) sample selection with varying ratios of labeled samples, ranging from 10%pt. to 100%pt..

From Table 4, we have four interesting observations: (1) Fine-tuning with active learning significantly improves performance compared with random sampling (FT+Entropy vs FT+Random), regardless of the number of labeled samples. This result demonstrates that the AL-based fine-tuning strategy efficiently eliminates noise and outliers and selects the most informative and diverse samples for fine-tuning; (2) TAPT+FT outperforms FT on both random sampling and Entropy sampling, indicating that TAPT can effectively minimize the domain difference and significantly enhance the performance of the downstream SER task; (3) With the same number of labeled samples, After can obtain better results than TAPT+FT+Random on the Merged dataset. However, After with 20%pt. labeled samples performs worse than TAPT+FT+Random with 80%pt.$\sim$100%pt. labeled samples. The reason is that TAPT+FT uses more labeled data for fine-tuning to prevent the model from overfitting and improve its robustness. In a fair comparison with the same size of the training data for fine-tuning, TAPT+FT+Random with 20%pt. labeled samples performs worse than After(20%pt.), demonstrating the effectiveness of After; (4) when 100%pt. sample is used, AL-based method significantly outperforms random sampling method (FT+Random vs FT+Entropy). The main reason is that Random-sampling is affected by noise data, and the model constantly corrects the classification boundary, making it difficult to improve the results. Entropy sampling avoids the effect of noisy data by selecting the most informative and diverse samples for FT in advance to fix the classification boundary properly.

Figure 3 (A) demonstrates that FT+AL with 20%pt. labeled samples significantly reduces the time consumption of FT (fine-tuning on all labeled samples). Compared to TAPT+FT, TAPT+FT+AL significantly decreases the time consumption with the main cost incurred by TAPT. Additionally, the relationship between time consumption and the ratio of labeled samples is shown in Figure 3 (B). AL-based fine-tuning exhibits a linear increase in time consumption with sample size from 1%pt.$\sim$20%pt. (exponential growth from 30%pt.$\sim$100%pt. in Table 2), indicating the efficiency of After and its potential to be applied in large-scale unlabeled real-world scenes.