Modality-Transferable Emotion Embeddings for Low-Resource Multimodal Emotion Recognition

Since the early 2010s, multi-modal emotion recognition has drawn more and more attention with the rise of deep learning and its advances in computer vision and natural language processing (Baltrušaitis et al., 2018). Schuller et al. (2011) proposed the first Audio-Visual Emotion Challenge and Workshop (AVEC), which focused on multi-modal emotion analysis for health. In recent years, most achievements in this area aimed to find a better modality fusion method. Zadeh et al. (2017) introduced a tensor fusion network that combined data representation from each modality to a tensor by performing the Cartesian product. In addition, the attention mechanism (Bahdanau et al., 2015) has been widely applied to do modality fusion and emphasis (Zadeh et al., 2018a; Pham et al., 2018; Tsai et al., 2019a). Furthermore, Liu et al. (2018) proposed a low-rank architecture to decrease the problem complexity, and Tsai et al. (2019b) introduced a modality re-construction method to generate occasional missing data in a modality.

Although prior works have made progress on this task, the relationship between emotion categories has not been well modelled in previous works, except by Xu et al. (2020), who captured emotion correlations using graph networks for emotion recognition. However, the model is only based on a single textual modality. Additionally, the previous studies have not put much effort toward unseen and low-resource emotion categories, which is a problem of multi-modal emotion data by nature.

Zero-shot and few-shot learning methods, which address the data scarcity scenario, have been applied to many popular machine learning tasks where zero or only a few training samples are available for the target tasks or domains, such as machine translation (Johnson et al., 2017; Gu et al., 2018), dialogue generation (Zhao and Eskenazi, 2018; Madotto et al., 2019), dialogue state tracking (Liu et al., 2019c; Wu et al., 2019), slot filling (Bapna et al., 2017; Liu et al., 2019b, 2020), and accented speech recognition (Winata et al., 2020). They have also been adopted in multiple cross-lingual tasks, such as named entity recognition (Xie et al., 2018; Ni et al., 2017), part-of-speech tagging (Wisniewski et al., 2014; Huck et al., 2019), and question answering (Liu et al., 2019a; Lewis et al., 2019). Recently, several methods have been proposed for continual learning (Rusu et al., 2016; Kirkpatrick et al., 2017; Lopez-Paz and Ranzato, 2017; Fernando et al., 2017; Lee et al., 2017), and these were applied to some NLP tasks, such as opinion mining (Shu et al., 2016), document classification (Shu et al., 2017), and dialogue state tracking (Wu et al., 2019).

We define the input multi-modal data samples as $X=\{(t_{i},a_{i},v_{i})\}_{i=1}^{I}$, in which $I$ denotes the total number of samples, and $t$, $a$, $v$ denote the textual, acoustic, and visual modalities, respectively. For each modality, there is a set of emotion embeddings that represent the semantic meanings for the emotion categories to be recognized. In the textual modality, we have $E_{t}=\{e^{t}_{k}\}_{k=1}^{K}$, which is from the pre-trained GloVe embeddings. In acoustic and visual modalities, we have $E_{a}=\{e^{a}_{k}\}_{k=1}^{K}$ and $E_{v}=\{e^{v}_{k}\}_{k=1}^{K}$, which are mapped from $E_{t}$ by the mapping function $f_{t\rightarrow v}$ and $f_{t\rightarrow a}$. $K$ denotes the number of emotion categories, and it can be changed to fit different tasks and zero-shot learning. $Y=\{y_{i}\}_{i=1}^{I}$ denotes the annotations for multi-label emotion recognition, where $y_{i}$ is a vector of length $K$ with binary values.

As shown in Figure 2, for each data sample, there are three sequences of length $T$ from the three modalities. For each modality, we use a bi-directional Long-Short Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) network as the encoder to process the sequence and get a vector representation. In other words, for the $i^{th}$ data sample, we will have three vectors, $r_{t}^{(i)}\in\mathbb{R}^{d_{t}}$, $r_{a}^{(i)}\in\mathbb{R}^{d_{a}}$, and $r_{v}^{(i)}\in\mathbb{R}^{d_{v}}$, that represent the textual, acoustic, and visual modalities. Here, $d_{t}$, $d_{a}$, and $d_{v}$ are the dimensions of the emotion embeddings of the textual, acoustic, and visual modalities, respectively.

As mentioned in Section 1, previous works do not consider the connections in different emotion categories, and the only information about emotions is in the annotations. In our model, we use emotion word embeddings to inject the semantic information of emotions into the model. Additionally, emotion embeddings also contain the relationships between emotion categories. For the textual modality, we use pre-trained GloVe (Pennington et al., 2014) embeddings of $K$ emotion words, denoted as $E_{t}\in\mathbb{R}^{K\times d_{t}}$. For the other two modalities, because there are no off-the-shelf pre-trained emotion embeddings, our model learns two mapping functions which project the vectors from the textual space into the acoustic and visual spaces:

	$\displaystyle E_{a}=f_{t\rightarrow a}(E_{t})\in\mathbb{R}^{K\times d_{a}}$		(1)
	$\displaystyle E_{v}=f_{t\rightarrow v}(E_{t})\in\mathbb{R}^{K\times d_{v}}\text{.}$		(2)

To predict the emotions for input sentences, we calculate the similarity scores between the sequence representation and the emotion embeddings for each modality. As shown in Eq.3, for a data sample $i$, every modality will have a vector of similarity scores of length $K$ by dot product attention. We further add a modality fusion module to weighted sum all the vectors, in which the weights are also optimized end-to-end (Eq.4). Finally, as the datasets are multi-labelled, the sigmoid activation function is applied to each score in the fused vector $s^{(i)}$, and a threshold is used to decide whether an emotion exists or not.

		$\displaystyle s_{t}^{(i)}=E_{t}r_{t}^{(i)}\text{, }s_{a}^{(i)}=E_{a}r_{a}^{(i)}\text{, }s_{v}^{(i)}=E_{v}r_{v}^{(i)}$		(3)
		$\displaystyle s^{(i)}=\text{{Sigmoid} }(w_{t}s_{t}^{(i)}+w_{a}s_{a}^{(i)}+w_{v}s_{v}^{(i)})$		(4)

Ideally, our model is able to directly adapt to a new emotion based on its embedding. Given a new text emotion embedding $e^{t}_{k+1}$, we can generate the visual and acoustic emotion embeddings $e^{v}_{k+1}$ and $e^{a}_{k+1}$, respectively, using the already learned mapping functions $f_{t\rightarrow v}$ and $f_{t\rightarrow a}$. After that, the similarity scores between the input sentence and the new emotion can be computed for each modality.

In this section, we assume 1% of the positive training samples in a new emotion are available, and to balance the training samples, we take the same amount of negative training samples for the new emotion. However, the model could lose its ability to predict the original emotions when we simply fine-tune it on the training samples of a new emotion. To cope with this issue, we propose two fine-tuning settings. First, after we obtain the trained model in the source emotions, we jointly train it with the training samples of the source emotions and the new target emotion. Second, we utilize a continual learning method, gradient episodic memory (GEM) (Lopez-Paz and Ranzato, 2017), to prevent the catastrophic forgetting of previously learned knowledge. The purpose of using continual learning is that we do not need to retrain with all the data from previously learned emotions since the data might not be available. We describe the training process of GEM as follows:

We define $\Theta_{S}$ as the model’s parameters trained in the source emotions, and $\Theta$ denotes the current optimized parameters based on the target emotion data. GEM keeps a small number of samples N from the source emotions, and a constraint is applied on the gradient to prevent the loss on the stored samples from increasing when the model learns the new target emotion. The training process can be formulated as

	$\displaystyle\text{Minimize}_{\Theta}~{}~{}L(\Theta)$
	$\displaystyle\text{Subject to}~{}~{}L(\Theta,N)\leq L(\Theta_{S},N),$

where $L(\Theta,N)$ is the loss value of the N stored samples.

We use CMU-Multimodal SDK (Zadeh et al., 2018c) for downloading and pre-processing the datasets. It helps to do data alignment and early-stage feature extraction for each modality. The textual data is tokenized in word level and represented using GloVe (Pennington et al., 2014) embeddings. Facial action units are extracted by the Facet (iMotions, 2017) to indicate muscle movements and expressions (Ekman et al., 1980). These are a commonly used type of feature for facial expression recognition (Fan et al., 2020). For acoustic data, COVAREP (Degottex et al., 2014) is used to extract fundamental features, such as mel-frequency cepstral coefficients (MFCCs), pitch tracking, glottal source parameters, etc.

For both the CMU-MOSEI and IEMOCAP datasets, we use Early Fusion LSTM (EF-LSTM) and Late Fusion LSTM (LF-LSTM) as two baseline models. Additionally, for CMU-MOSEI, the Graph Memory Fusion Network (Graph-MFN) (Zadeh et al., 2018b) and a multi-task learning (MTL) model (Akhtar et al., 2019) are included for comparison with previous state-of-the-art models. For IEMOCAP, the Recurrent Multistage Fusion Network (RMFN) (Liang et al., 2018), Recurrent Attended Variation Embedding Network (RAVEN) (Wang et al., 2018), and the Multimodal Transformer (MulT) (Tsai et al., 2019a) are included. To compare the AUC scores and zero-shot performance with baselines, we re-run the MTL and MulT models based on their reported best hyper-parameters, and we also carry out hyper-parameter search for a fair comparison.

The model is trained end-to-end with the Adam optimizer (Kingma and Ba, 2015) and a scheduler that will reduce the learning rate by a factor of 0.1 when the optimization stays on a plateau for more than 5 epochs. The best hyper-parameters in our training for both datasets are shown in Table 3. Also, we use the largest GloVe word embeddings (glove.840B.300d ²²2https://nlp.stanford.edu/projects/glove/) for both the input text data and the emotion embeddings in the textual modality. The weights of the textual embeddings are frozen during training to keep the pre-trained relations, which is also essential for doing zero-shot learning.

Table 1 shows our model’s performance on the CMU-MOSEI dataset. Compared to existing baselines, our model surpasses them by a large margin. The weighted accuracy (W-Acc) and AUC score are used for evaluation, with a threshold set to 0.5 to calculate the W-Acc. As discussed in Section 5.3, we do not follow the previous papers in using the weighted F1-score (W-F1) because it does not provide an effective evaluation when the dataset is very imbalanced. For example, the weakest baseline, EF-LSTM, can even achieve  90% W-F1 by predicting almost all samples as negative. More plots and analysis of this defect of W-F1 are included in Appendix B.

We further test our model on a second dataset called IEMOCAP, and the results are shown in Table 2. Similarly, our model achieves better results on most emotion categories, except happy. For a fair comparison on IEMOCAP, we use accuracy instead of W-Acc, following the previous works compared in the table.

Quantitatively, our model makes a large improvement in the multi-modal emotion recognition task. We think it benefits greatly from the emotion embeddings, which can model the relationships (or distances) between emotion categories. This is especially important for emotion recognition, which is a multi-label task by nature, as people can have multiple emotions at the same time. For example, if a person is surprised, it is more likely that this person is also happy and excited and is less likely to be disgusted or sad. This kind of information is expected to be modelled and captured by emotion embeddings. Intuitively, in the textual space, related emotions (e.g., angry and disgusted) tend to have closer word vectors than unrelated emotions (angry and happy). To ensure the effectiveness of word embeddings, for each emotion word, we investigated multiple forms of it. For example, for surprised, we also tried with Surprised, (S/s)urprising, (S/s)urprise. Generally, they all show a similar trend, and in most cases, the word form that is used to describe human shows the best results. In our final setting, we iterate and pick the best performing form for each emotion category.

Moreover, our model can transfer the relationship of emotion categories from the textual space to the acoustic and visual spaces using end-to-end optimized mapping functions. In Figure 3, we show the Euclidean distances of emotion embeddings between categories. The relative positions are preserved very well after being transferred from the textual space to the visual and acoustic spaces. This indicates that the learned mapping functions ($f_{t\rightarrow v}$ and $f_{t\rightarrow a}$) are effective. Although it is not the main focus of this paper, we think improving the pre-trained textual emotion embeddings is an essential direction for future work. It can benefit all modalities and further enhance the overall performance. For example, incorporate semantic emotion information (Xu et al., 2018) to the original word embeddings.

Benefiting from the pre-trained textual emotion embeddings and learned mapping functions, our model can recognize unseen emotion categories to a certain extent. We evaluate our model’s zero-shot learning ability on the low-resource categories in CMU-MOSEI (shown in Table 4) and IEMOCAP (shown in Table 5). For a fair comparison, we use the same training setting that is used in Table 1. This can ensure that no downgrade happens on the seen emotions, and the model is not selected to overfit a single unseen category. As we can see, the zero-shot results of the baselines are similar to random guesses, because the weights related to that unseen emotion in the model are randomly initialized and have never been optimized. For our model, the zero-shot performance is much better than that of the baselines in almost all emotions. This is because our model learns to classify emotion categories based on the similarity between the sentence representation and emotion embeddings, which enables our model better generalization ability to other unseen emotions since emotion embeddings contain semantic information in the vector space.

Furthermore, we perform few-shot learning using only 1% of data of these low-resource categories. As we can see from Table 4, using very few training samples, our model can adapt to unseen emotions without losing the performance in the source emotions. In addition, we observe that simply fine-tuning (FT) our trained model sometimes obtains inferior performance. This is because our model will gradually lose the ability to classify the source emotions, and we have to early stop the fine-tuning process, which leads to inferior performance. We can see that CL and JT prevent our model from catastrophic forgetting and improve the few-shot performance in the unseen emotion. Moreover, JT achieves slightly better performance than CL. This can be attributed to the fact that CL might still result in performance drops in source emotions since our model only observes partial samples from them. At the same time, JT directly optimizes the model on the data samples of such emotions.

To further investigate how each individual modality influences the model, we perform comprehensive ablation studies on supervised multi-modal emotion recognition and also zero-shot prediction.

In Table 6, we enumerate different subsets of the (textual, acoustic, visual) modalities to evaluate the effect of each single modality. Generally, the performance will increase if more modalities are available. Compared to single-modal data, multi-modal data can provide supplementary information, which leads to more accurate emotion recognition. In terms of a single modality, we find that textual and acoustic are more effective than visual.

Similarly, in Table 5, we show the zero-shot performance with different combinations of modalities during the inference time (all modalities exist in the training phase). As there are many more negative samples than positive ones in the ZSL setting, we also evaluate the models with the unweighted F1 score. Because if a model has high accuracy but a low F1, it is heavily biased to the negative samples so it cannot do classification effectively. Empirical results indicate that zero-shot on only one modality is possible. Moreover, if the data of an emotion category has strong characteristics in one modality and is ambiguous in other modalities, single-modality can even surpass multi-modality on zero-shot prediction. For example, the performance of single-modality zero-shot prediction using the acoustic modality on the surprised category is better than using all modalities.