Multimodal Fusion via Hypergraph Autoencoder and Contrastive Learning for Emotion Recognition in Conversation

Regarding the non-graph learning methods, BC-LSTM captures contextual information from surrounding utterances in three different modalities by using three independent bidirectional LSTM networks, and the output utterance representations are used for label classification (Poria et al., 2017). However, this method lacks the usage of speaker information and thus is not applicable to multi-person conversation scenarios. DialogueRNN utilizes three gate recurrent units (GRUs) to track the global context, speaker state, and emotion state throughout the entire dialogue, which effectively integrates speaker modeling, contextual modeling, and emotion modeling (Majumder et al., 2019). To mimic the human reasoning process, DialogueCRN introduces reasoning modules to integrate the factors that make emotions happen (Hu et al., 2021b).

The standard graph-based methods, such as DialogueGCN (Ghosal et al., 2019), represent textual information in utterances as nodes and capture contextual relationships through different types of edges connecting nodes within a given window size. The multimodal graph convolutional network (MMGCN) develops DialogueGCN by further incorporating audio and video modalities into the model (Hu et al., 2021a). To address the challenge of cross-modal interaction in information fusion within ERC, MIMMN introduces a multi-view network that leverages complementary information from all modalities. It dynamically balances the relationships between all modalities during the fusion process (Wen et al., 2023). MM-DFN (Hu et al., 2022) uses a dynamic fusion mechanism to fully understand the context relationship between multiple modalities and reduce the redundancy between modalities. COGMEN (Joshi et al., 2022) utilizes graph neural network (GNN) to leverage both local and global information in a conversation. GraphMFT (Li et al., 2023) not only designs a multimodal fusion method based on graphs but also utilizes multiple graph attention networks (GATs) to capture the intra-modal contextual details and inter-modal complementary information. M3NET (Chen et al., 2023) introduces the hypergraph into the field of MERC. Through simple fully connected structures and randomly initialized edge weights, significant improvement in prediction accuracy and time efficiency has been achieved by multiple hypergraph convolutions.

A hypergraph acts as an extended version of the standard graph learning, specifically designed to extract high-order correlations within the data (Antelmi et al., 2023; Zhang et al., 2023). The examples of hypergraphs are shown on the left side of Figure 1, with the corresponding standard graphs shown on the right side. The circular dots represent five nodes, i.e., from $V_{1}$ to $V_{5}$. Curves with the same color form a hyperedge, and there are three hyperedges $e_{1},e_{2},e_{3}$ in total. In a hypergraph, connections are not limited to pairwise relationships as in a standard graph. Hyperedges can link multiple nodes together, and a single node can be linked by multiple hyperedges simultaneously. Meanwhile a hypergraph can include multiple types of hyperedges, representing multiple meanings. In this paper, we create a hypergraph in which all utterances linked to the same speaker are grouped together on a hyperedge, while also connecting similar modality into another hyperedge. This structure closely resembles the physical structure of certain models, capturing higher-order correlations and minimizing information loss during the modeling process. The effectiveness of hypergraph learning in solving the association problem of multimodal data has already been verified in various applications, such as including recommendation system (Xia et al., 2022), video segmentation (Yan et al., 2020), sleep stage classification (Liu et al., 2024; Zhang et al., 2024), and drug-target interaction prediction (Ruan et al., 2021).

Regarding hypergraph convolutions, the learning process involves aggregating node information onto connected hyperedges with varying weights, followed by sending messages from the hyperedges back to the connected nodes. This process is not constrained by distance, thereby mitigating the limitations of message transmission during the process (Gao et al., 2022). These benefits are particularly pronounced in long-distance transmissions (Gao et al., 2020). Therefore, the hypergraph learning process is anticipated to be effective in the MERC task, as speakers frequently discuss topics that are distant from the current conversation, utilizing long-distance cues.

This module involves extracting essential information from raw visual, textual, and acoustic modalities data. Following the approach outlined in M3NET (Chen et al., 2023), features from visual modalities are extracted using DenseNet (Huang et al., 2016) or 3D-CNN (Yang et al., 2019), depending on the adopted dataset. Features from acoustic and textual modalities are extracted using the OpenSmile toolkit (Eyben et al., 2010) and the RoBERTa large model (Liu et al., 2019) respectively.

As mentioned above, incorporating contextual information is crucial for emotion category prediction in conversations. To enhance discourse feature representation, we employ various encoding methods tailored to the characteristics of different modalities. Specifically, we utilize GRU network (Dey and Salem, 2017) to encode context information for the textual modality, while acoustic and visual information is encoded using two fully connected multilayer perceptrons (MLPs). To facilitate the information fusion across modalities, we normalize the encoded dimension to a unified $d$ dimension as below:

where $u_{i}$ is the input of unimodal encoding. $U_{i}$ is the output of the model with the dimension $d$. $a,v,t$ stands for visual, acoustic, and textual modalities respectively. $W$ and $b$ are trainable parameters.

Speaker information is a critical factor that affects the performance of the MERC task. We firstly encode the speaker information into vectors $s_{i}$ in one-hot form as:

Next, we integrate them into the modality information by:

The output of this module is $V_{i}$, which is the feature embeddings with modality-independent context awareness and speaker information.

This module is composed of three stages: structure initialization, VHGAE, and hypergraph reconstruction.

With the new hypergraph $\mathcal{G}_{0}=(\mathcal{V},\mathcal{E}_{0})$, we first perform node convolution by aggregating node features to update the hyperedge embeddings. The aggregation stage facilitates the integration of information from neighboring nodes into the hyperedge representation. Following the update of the hyperedge embeddings, we proceed to the hyperedge convolution stage, where hyperedge messages are disseminated to the nodes. This operation enables the information propagation from hyperedges to their incident nodes. For each hyperedge $\epsilon\in\mathcal{E}_{0}$, we aggregate the embeddings of its incident nodes $v$ according to a predefined aggregation function by:

where $n_{\epsilon}$ represents the updated embedding for the hyperedge $\epsilon$ and $n_{v}$ denotes the embedding of the node $v$. The aggregation function Agg combines the embeddings of the incident nodes to generate the new hyperedge embedding. For each node $v\in\mathcal{V}$, we aggregate the messages from its incident hyperedges $\epsilon$ using a predefined aggregation function similar to Equation 11.

By performing the node and hyperedge convolutions, we can effectively propagate information and update the embeddings in the hypergraph $\mathcal{G}_{1}=(\mathcal{V}_{0},\mathcal{E}_{0})$. This reformulated solution enables the capture of the relationships and interactions between nodes and hyperedges, facilitating a more comprehensive understanding of the hypergraph structure.

In order to mitigate the instability inherent in the sampling and decoding processes, we devise a dual-path scheme within our model. The primary objective is to minimize the dissimilarity between corresponding points in two hypergraphs $\mathcal{G}_{1}^{(1)}=(\mathcal{V}_{0}^{(1)},\mathcal{E}_{0})$ and $\mathcal{G}_{1}^{(2)}=(\mathcal{V}_{0}^{(2)},\mathcal{E}_{0})$, which are obtained through the progression of VHGAE and convolution. Concurrently, we aim to maximize the distance between each point and other points within the embedding space.

Within the context of the two hypergraph views, pairs of vertices that correspond to one another are regarded as positive pairs, whereas the remaining vertex pairs are considered negative pairs. The embedding of the $i$-th vertex in the two views is denoted as $v_{i}^{(1)}\in\mathcal{V}_{0}^{(1)}$ and $v_{i}^{(2)}\in\mathcal{V}_{0}^{(2)}$. The contrastive loss $\mathcal{L}_{cl}$ between $\mathcal{V}_{0}^{(1)}$ and $\mathcal{V}_{0}^{(2)}$ is:

Here, $\mathcal{V}_{0}$ denotes the set of vertices and $|\mathcal{V}_{0}|$ signifies the cardinality of $\mathcal{V}_{0}$. The term $f(v_{i}^{(1)},v_{i}^{(2)})$ is calculated as:

where

Here, $\tau$ is a temperature parameter and $g(,)$ denotes the cosine similarity function. Considering that the function $g(,)$ is not symmetric, we average the positive and negative aspects. Specifically, $\sum_{i\neq j}q(v_{i}^{(1)},v_{j}^{(2)})$ and $\sum_{i\neq j}q(v_{i}^{(1)},v_{j}^{(1)})$ denote the negative pairs in the other graph and the same graph, respectively. Meanwhile, $q(v_{i}^{(1)},v_{i}^{(2)})$ represents a positive pair in the other graph.

By minimizing the combined loss function $\mathcal{L}_{cl}$, the similarity between corresponding points is expected to increase while enhancing the distance between each point and other points within the embedding space. This approach promotes alignment and discrimination of the embeddings, thereby yielding more stable and meaningful representations of the hypergraph structure.

After acquiring contextual knowledge, we perform a fusion process on the node embeddings of the two hypergraphs $\mathcal{G}_{1}^{(1)}$ and $\mathcal{G}_{1}^{(2)}$ to obtain $\mathcal{G}_{2}=(\mathcal{V}_{2},\mathcal{E}_{2})$. This process aims to integrate the information from the two hypergraphs into a unified representation.

Following that, we concatenate the node embeddings of the three modalities that belong to the same utterance, resulting in a comprehensive representation. Specifically, let {$v^{t}_{i},v^{a}_{i},v^{v}_{i}\}\in\mathcal{V}_{2}$ denote the node embeddings of the hypergraphs corresponding to the textual, acoustic, and visual modalities, respectively. We concatenate these embeddings to obtain a fused representation by:

The Concatenate function combines the embeddings of the three modalities into a unified vector, allowing for the integration of multiple sources of information. The fused representation $v_{i}$ encompasses a broader range of information, enhancing subsequent analysis and prediction tasks by providing a more comprehensive input.

Given the fused representation $v_{i}$ for an utterance, the formulas for predicting the emotion label are as follows:

In these formulas, $W_{2}$ and $W_{3}$ are weight matrices, $b_{2}$ and $b_{3}$ are bias vectors, $\widehat{v}_{i}$ is the processed output of $v_{i}$ using the ReLU activation function, $P_{i}$ is the probability distribution over the emotion labels, and $\widehat{y}_{i}$ is the predicted emotion label. $\tau$ represents the dimension corresponding to the emotion labels. By applying these formulas, we can predict the emotion label for each utterance based on the fused representation $v_{i}$ and the learned parameters $W_{2}$, $W_{3}$, $b_{2}$, and $b_{3}$.

We use categorical cross-entropy loss with $L2$ regularization term to define the error loss between the predicted emotion category and the true label during the training process as below:

where $N$ represents the number of dialogues in a dataset. $c(s)$ represents the number of utterances in dialogue $s$. It is worth noting that each dialogue can have a different number of utterances. $P_{i,j}$ denotes the predicted probability distribution of emotion labels for utterance $j$ in dialogue $i$, while $y_{i,j}$ represents the expected class label. The regularization weight $\lambda$ controls the importance of the regularization term relative to the cross-entropy loss.

By combining Equations 17, 10 and 12, we define the final loss function as:

where the hyperparameter weights $\lambda_{g}$ and $\lambda_{cl}$ control the importance of the generalized adversarial loss and the contrastive loss, respectively.

In this paper, we conduct experiments on two popular multimodal datasets in the field of MERC: the interactive emotional dyadic motion capture database (IEMOCAP) (Busso et al., 2008) and multimodal emotionlines dataset (MELD) (Poria et al., 2019).

• •

  IEMOCAP: It contains videos of two-way conversations with 10 actors (5 male and 5 female). IEMOCAP records the tone and power of speech, facial expressions, torso posture, head position, gestures, transcripts, and gaze in a duo session. In this paper, we use facial expressions, the tone and power of speech and transcripts. The emotions in this dataset are artificially classified into six categories: happy, sad, neutral, angry, excited, and frustrated. We use 120 dialogues containing 5,810 utterances for training and validation, while the remaining 31 dialogues with 1623 utterances for testing.

• •

  MELD: It is a multimodal dataset for emotion recognition in multi-party conversations, containing textual, acoustic and visual modalities for ERC, selected from Friends TV series. This dataset includes seven emotions: neutral, surprise, fear, sadness, happiness, disgust, and anger. We use 1,153 dialogues with 11,098 utterances for training and validation, while the rest 280 dialogues with 2610 utterances for testing.

It is worth noting that IEMOCAP dataset features a fixed set of two speakers engaging in multiple rounds of conversation, whereas MELD dataset may involve multiple speakers but with fewer utterances per conversation. Meanwhile, the emotion distribution within MELD dataset is imbalanced, with a significantly higher proportion of ”neutral” emotions compared to other emotional categories, comprising nearly half of the dataset. These characteristics pose significant challenges to the model’s stability.

We perform all experiments on an NVIDIA GTX 1050Ti with Win11 operating system. The versions of Pytorch and cuda are 2.1.2 and 11.8, respectively. Adam optimizer is used for training. We set the batch size as 12 and the learning rate as 0.0001 on both datasets. The hyperparameter $\tau$ of Gumbel-softmax in Equation 9 is 0.1 . The number of hypergraph convolutions is 1. More details regarding the main parameters can be found in Table 1

In order to validate the performance of the proposed method HAUCL in the MERC task, we introduce the ten state-of-the-art methods for comparison: (1) non-graph learning: LSTM (Poria et al., 2017), DialogueRNN (Majumder et al., 2019), and DialogueCRN (Hu et al., 2021b); (2) standard graph learning: DialogueGCN (Ghosal et al., 2019), MMGCN (Hu et al., 2021a), DIMMN (Wen et al., 2023), MMDFN (Hu et al., 2022), COGMEN (Joshi et al., 2022) and GraphMFT (Li et al., 2023); and (3) hypergraph learning: M3NET (Chen et al., 2023). More details regarding the baseline methods can be found in Section 2.

For validation, we adopt the most mainstream evaluation metrics in this field: accuracy (Acc.) and weighted F1 score (WF1).

Table 2 summaizes the performance of different methods tested on IEMOCAP and MELD datasets. The results show that our proposed HAUCL achieves superior performance in terms of the overall accuracy and weighted F1 score. In detail, compared with M3NET, which achieves the second-best performance, HAUCL enhances the accuracy and WF1 by 0.43% and 0.57% respectively on MELD dataset and by 1.29% and 1.15% on IEMOCAP dataset.

Our model also has advantages in training time and model size.The average training time of each epoch of HAUCL is 38 secs and the model size is 173,945 KB. In comparison, M3NET[3], which obtains the second-best performance, has a training time of 56 secs with the size of 608,867 KB. This superior performance can be attributed to the fact that HAUCL utilizes only one hypergraph convolution layer, whereas M3NET incorporates three convolutions and three high-frequency information convolutions.

The ability of HAUCL to dynamically modify the connection structure of the hypergraph helps in reducing information redundancy, particularly in IEMOCAP dataset with a high average utterance per conversation. Additionally, the use of hypergraphs helps prevent excessive smoothing, reducing the risk of excessive smoothing occurring in standard graph-based methods. Compared with non-graph learning methods, our proposed HAUCL can demonstrate significant enhancement in long-distance information transmission and multimodal information fusion, resulting in satisfactory accuracy and weighted F1 score performance.

We select the following four main parameters in HAUCL for sensitivity analysis tested on MELD dataset. Figures 4 and 4 show the ratio of hypergraph reconstruction loss and contrastive learning loss to the total loss, respectively. Figure 4 represents the number of convolution layers passed by the new hypergraph obtained by reconstruction to learn the contextual information. Figure 4 shows the effect of batch size. Similar trends are also observed on IEMOCAP data.

The weight of the hypergraph reconstruction loss $\lambda_{g}$: It reflects the deviation from the original graph over the total loss (See Equation 18). Higher values of $\lambda_{g}$ indicate that the reconstructed hypergraph closely resembles the original graph. As shown in Figure 4, when $\lambda_{g}$ is set to 0.5, our method demonstrates optimal performance in accuracy. Meanwhile, deviating from this optimal value, either towards larger or smaller values, results in a decline in the overall performance.

The weight of contrastive learning loss $\lambda_{cl}$: Similar with $\lambda_{g}$, as the value of $\lambda_{cl}$ increases, the method will increasingly focus on the differences between the two hypergraphs derived from the two paths. Conversely, when the dissimilarity between the two hypergraphs diminishes, our proposal’s capability to withstand interference strengthens. However, when this value is excessively large, it will impact the loss of emotion recognition, i.e., when $\lambda_{cl}$ exceeds 1.1, there is a degradation in accuracy performance as plotted in Figure 4.

Hypergraph layer $L$: Figure 4 demonstrates that increasing the number of covolutional layers in a hypergraph does not necessarily lead to enhanced accuracy performance. A large value of $L$ not only amplifies the model’s complexity and runtime, but also risks oversmoothing, potentially complicating the differentiation of emotions with similar characteristics. When $L$ is 5, there is a sharp decrease in accuracy performance, indicating an over-smoothing phenomenon.

Batch size: The selection of batch size is a crucial factor that impacts the performance of recognition (Kandel and Castelli, 2020; He et al., 2019). Given the non-uniform distribution of MELD dataset, employing a batch size that is too small can render the model susceptible to the interference of small samples, leading to significant gradient fluctuations and convergence challenges. Conversely, an excessively large batch size may prompt the model to overly generalize, potentially compromising accuracy. As depicted in Figure 4, the best performance is attained with a batch size of 12.

For a more comprehensive analysis of the effectiveness of our proposed method HAUCL, we conduct ablation experiments from three different aspects: the impact of (1) speaker embedding, (2) VHGAE and contrastive learning, and (3) contrastive learning in terms of accuracy performance. The results are summarized in Table 3.

Impact of Speaker Embedding: Speaker embedding can distinguish the input features from different speakers. Existing research has shown that incorporating speaker information can enhance the accuracy of emotion recognition tasks (Hu et al., 2021a). The exclusion of speaker embedding (”w/o SE” in Table 3) results in a decrease in accuracy, with a degradation of 1.11% and 0.81% observed on IEMOCAP and MELD datasets, respectively. These findings indicate that incorporating person modeling can enhance the model’s performance in the MERC domain.

Impact of VHGAE and Contrastive Learning: We utilize VHGAE for dynamic hyperedge selection to minimize redundancy and employ contrastive learning to mitigate random errors. In Table 3, ”w/o GCL” indicates the direct fusion of two hypergraph convolutions without the inclusion of VHGAE and contrastive learning module. The results demonstrate the effectiveness of our proposed HAUCL: It can improve the accuracy performance by 0.78% and 0.43% on IEMOCAP and MELD datasets respectively.

Impact of Contrastive Learning: ”w/o CL” in Table 3 refers to the model that incorporates hypergraph and VHGAE without the integration of contrastive learning. The experimental results indicate a 0.98% enhancement in accuracy on IEMOCAP dataset and a 0.58% improvement on MELD dataset. These results verify that the contrastive learning module effectively controls the random fluctuations of VHGAE, enhancing model accuracy while reducing information redundancy.

In order to demonstrate the discriminability of nodes, we present the node representations acquired through our proposed method HAUCL and the M3Net (the second-best method in Table 2) on MELD dataset. To visualize these representations in a more comprehensive manner, we employ t-SNE (Van der Maaten and Hinton, 2008) method for dimensionality reduction, transforming the obtained nodes into three dimensions. Furthermore, we assign distinct colors to indicate the true labels of the nodes. By comparing the two figures in Figure 5, it is evident that the data points depicted in Figure 5 (our method) exhibit greater separation, resulting in a more discriminative segmentation. As aforementioned, the representations derived from the proposed HAUCL exhibit reduced redundancy and enhanced discriminability, thereby enabling the attainment of superior outcomes.

This subsection summarizes the time complexity of hypergraph reconstruction. The hypergraph construction in HAUCL includes variantional encoding and decoding part. Starting with encoder, a hypergraph convolution layer is involved with time complexity $O(Nd^{2}+NdM)$, where $N$ is the number of utterances. $M$ is the number of hyperedges in the initial trivial input graph, and $d$ is the feature dimension. To generate the latent embedding of mean and variance in Equations 6a and 6b, the time complexity is $O(Nd^{2})$. The sampling process can be calculated as $O(N+M)$. In the decoding process, the complexity to compute the incidence matrix is $O(NMd^{2})$. In all, the total computational cost of our framework is $O(NMd^{2})$$\rightarrow$ $O(N^{2}d^{2})$.