MMA-MRNNet: Harnessing Multiple Models of Affect and Dynamic Masked RNN for Precise Facial Expression Intensity Estimation

The Multiple Models of Affect (MMA) extractor component processes an input video X by extracting affective representations from each frame using three distinct models of affect. Specifically, the MMA is a Multi-Task Learning (MTL) CNN model that concurrently performs: (i) continuous affect estimation in terms of valence and arousal (VA); (ii) recognition of 7 basic facial expressions; and (iii) detection of 17 action units (AUs). The architecture of the MMA, illustrated in Fig. 2, is structured around residual units, with ’bn’ indicating batch normalization layers. The model integrates the valence-arousal estimation, 7 basic expression recognition, and 17 AU detection tasks within the same embedding space derived from a shared feed-forward layer. Consequently, the output of the MMA when processing X is a multiple affect-level representation matrix $\textbf{{Z}}=\{\textbf{z}_{1},...,\textbf{z}_{t}\}\in\Re^{26\times t}$.

For training the MMA, we utilize multiple in-the-wild datasets, including Aff-Wild2 [38, 40, 20, 36, 22, 37, 34, 28, 57, 18, 33, 41, 32, 42], AffectNet [49], and EmotioNet [5], which are annotated for valence-arousal, 7 basic expressions, and 17 action units (these action units are an aggregate in all datasets).

Our recent studies [15, 14] have highlighted challenges in the current evaluation of affect analysis methods, noting inconsistencies in database partitioning and evaluation practices that lead to biased and unfair comparisons. To address these issues, a unified protocol for database partitioning was proposed, ensuring fairness and comparability, while also accounting for subjects’ demographic information. It was demonstrated that methods previously considered state-of-the-art on original partitions may not retain their performance under this new protocol. Consequently, in this current paper, we adopt this updated partitioning protocol.

A key challenge in utilizing these datasets is the non-overlapping nature of their task-specific annotations. For instance, EmotioNet only includes AU annotations, lacking valence-arousal and 7 basic expression labels. Training the MMA directly with these datasets using a combined loss function for all tasks would result in noisy gradients and poor convergence, as not all loss terms would be consistently contributing to the overall objective function. This can lead to issues typical of Multi-Task Learning (MTL), such as task imbalance, where one task may dominate training, or negative transfer, where the MTL model underperforms compared to single-task models [34, 33, 32].

To address this issue, we generate AU pseudo-representations ($r_{AU}^{\prime}$) from the 7 basic expression representations ($r_{expr}$) produced by the MMA for each frame. This is achieved by leveraging the relationship between expressions and AUs, as defined in Table 1 of [8]. The study by [8] conducted a cognitive and psychological analysis of the associations between facial expressions and AU activations, summarizing the findings in a table that details the relatedness between expressions and their corresponding AUs. This table is presented in Table 1 for reference. Prototypical AUs are those consistently identified as activated by all annotators, while observational AUs are those marked as activated by only a subset of annotators.

The AU pseudo-representations are modeled as a mixture over the basic expression categories:

Then we match MMA’s AU representations ($r_{AU}$) with the AU pseudo-representations by minimizing the binary cross entropy with soft targets loss:

With this loss we aim to infuse prior knowledge on task’s relationship (according to Table 1) into the network, so as to guide the generation of better and consistent expression and AU representations. For instance, if the network predicts happiness with probability 1 and also predicts that AUs 4, 15 and 1 are activated, this is a mistake as these AUs are associated with the expression sadness. In this case, the AU and expression representations are in conflict. Therefore the overall objective function ($\mathcal{L}_{MMA}$) minimized during MMA’s training is:

As described in the previous section, the MMA component processes an input video X by extracting affective representations from each frame ($\textbf{{x}}_{i}$) using three distinct affective models. This results in an affect-level representation matrix $\textbf{{Z}}=\{\textbf{z}_{1},...,\textbf{z}_{t}\}\in\Re^{26\times t}$. This matrix is then fed into an RNN positioned atop the MMA component, which captures temporal dependencies and sequential information across consecutive frames of the video. The RNN sequentially processes the extracted vector representations from frame $0$ to frame $t$, mapping these representations $\{\textbf{z}_{k}\}^{K}_{k=1}$ to embeddings $\{\textbf{z}^{\prime}_{k}\}^{K}_{k=1}$, where each $\textbf{z}^{\prime}_{k}\in\Re^{d^{\prime}}$.

These embeddings (corresponding to all video frames) are concatenated into a single vector embedding $\textbf{{z}}^{\prime}\in\Re^{d^{\prime}\cdot t}$, aligning with the goal of estimating the intensity of various facial expressions across the entire video sequence, consistent with the provided annotations. This embedding $\textbf{{z}}^{\prime}$ is then passed through a Mask layer, producing new embedding $\textbf{{z}}^{\prime\prime}\in\Re^{d^{\prime}\cdot t}$. The original (pre-padding) length $l$ of the input video is propagated to the Mask layer to guide the routing mechanism. During training, the routing mechanism dynamically selects elements from various positions within $\textbf{{z}}^{\prime}$ based on the video’s length $l$, preserving the values of these selected elements and setting the remaining elements to zero. This process effectively routes only the relevant elements into the subsequent layer, thereby enhancing the model’s focus on key temporal features.

The embedding $\textbf{{z}}^{\prime\prime}\in\Re^{d^{\prime}\cdot t}$ is then transformed into another embedding $\textbf{{z}}^{\prime\prime\prime}\in\Re^{d^{\prime\prime}}$ through a feed forward layer, which is trained to extract high-level information from the ’masked’ embedding $\textbf{{z}}^{\prime\prime}$. During training, only the weights connecting the feed-forward layer neurons to the elements within $\textbf{{z}}^{\prime}$ routed by the Mask layer are updated. The remaining weights are updated when their corresponding feed-forward layer neurons are connected to elements within $\textbf{{z}}^{\prime}$ that are selected by the Mask layer in a different video input. The loss function minimization is conducted similarly to networks with dynamic routing, where the weights not involved in the routing process remain constant, and links corresponding to non-routed elements within $\textbf{{z}}^{\prime}$ are ignored. Finally, the embedding $\textbf{{z}}^{\prime\prime\prime}$ is mappes to the logits u via a feed forward layer parameterized by W, resulting in the video-level classification, expressed as: $\textbf{{u}}=\textbf{{W}}^{T}\textbf{{z}}^{\prime\prime\prime}$.

The loss function that we utilized for training MMA-MRNNet: was not the typical Mean Squared Error (MSE) but a loss based on the pearson correlation since that correlation metric was the evaluation criterion for the utilized database:

The Hume-Reaction dataset was used as part of both the Emotional Reactions Sub-Challenge of MuSe 2022 [6] and the Emotional Reaction Intensity Estimation Challenge of the 5th ABAW Competition in 2023 [40, 18, 41, 42, 33, 27, 29, 30, 19, 20, 35, 25, 21, 36, 31, 22, 37, 34, 26, 1, 23]. The participants of this subchallenge explore a multi-output regression task, utilizing seven, self-annotated, nuanced classes of emotion: ‘Adoration,’ ‘Amusement,’ ‘Anxiety,’ ‘Disgust,’ ‘Empathic-Pain,’ ‘Fear,’ and ‘Surprise.’ The dataset consists of 25,067 videos taken from 2,222 subjects of which 15,806 constitute the training set, 4,657 the validation set and 4,604 the test set.

The Aff-Wild2 database [38, 40, 20, 36, 22, 37, 34, 28, 57, 18, 33, 41, 32, 42, 43] is the largest in-the-wild database and the only one to be annotated in a per-frame basis for the seven basic expressions (i.e., happiness, surprise, anger, disgust, fear, sadness and the neutral state), twelve action units (AUs 1,2,4,6,7,10,12,15,23,24,25, 26) and valence and arousal. In total, it consists of 564 videos of around 2.8M frames with 554 subjects. Aff-Wild2 displayes a big diversity in terms of subjects’ ages, ethnicities and nationalities; it has also great variations and diversities of environments.

The AffectNet dataset [49] contains around 1M facial images, 300K of which were manually annotated in terms of 7 discrete expressions (plus contempt) and valence-arousal. The original training set of this database consists of around 290K images and the original validation of 4K. We evaluate our method on the updated partitioning protocol of this database according to our previous work [15, 14] (as we mentioned in Section 3.1). This new partitioning consists of a training set of around 160K images, a validation set of around 45K images and a test set of around 90K images.

The EmotioNet database [5] contains around 1M images and was released for the EmotioNet Challenge in 2017. 950K images were automatically annotated and the remaining 50K images were manually annotated with 11 AUs (1,2,4,5,6,9,12,17,20,25,26); around half of the latter constituted the validation and the other half the test set of the Challenge. We evaluate our method on the updated partitioning protocol of this database according to our previous work [15, 14] (as we mentioned in Section 3.1). This new partitioning consists of a training set of around 25K images, a validation set of around 7K images and a test set of around 14K images.

We used the RetinaFace detector [7] to extract, from all images, face bounding boxes and 5 facial landmarks; the latter were used for face alignment. All cropped and aligned images were resized to $112\times 112\times 3$ pixel resolution and their intensity values were normalized to $[-1,1]$.

We chose batch size equal to 4, length $t$ equal to 480, Adam optimizer with learning rate $10^{-4}$ when training from scratch and $10^{-5}$ when training in an end-to-end manner, after having initialised each subnetwork. For RNN we utilize 1-layer GRU with 128 units; feed forward layer consists of 32 units. Training was performed on a Tesla V100 32GB GPU; training time was 3 days. The TensorFlow platform has been used.

At first we compare the performance of MMA-MRNNet to that of various baseline [6] and state-of-the-art methods: ViPER and Netease Fuxi Virtual Human methods (which are multi-modal methods exploiting audio, visual and text information); the best performing HFUT-CVers method (presented in the related work section; it is an ensemble multi-modal method exploiting both audio and visual information); USTC-IAT-United method (which was presented in the related work section and is a multi-modal method exploiting both audio and visual information); USTC-AC and NISL-2023 methods (both presented in the related work section; they are ensemble multi-modal methods exploiting both audio and visual information). Table 2 shows that our uni-modal non-ensemble learning MMA-MRNNet (that exploits only the visual information and does not employ any ensemble learning) outperforms all other methods by large margins (although some methods are multimodal ones or even ensembles). Let us also note that all baseline and state-of-the-art methods utilized the ad-hoc strategy of selecting fixed input length by removing or duplicating images within each video sequence.

We conducted a series of ablation experiments to evaluate the impact of different elements and components on our model’s performance.

Initially, we used only single-task affective representations (extracted from MMA) as input to the RNN. We then tested combinations of two tasks (e.g., VA & AUs), and finally, we utilized the affective representations from all three tasks concurrently. The results are summarized in Table 3, where we present only the best performance for each experiment to avoid cluttering of the results. Notably, even when using only valence and arousal representations, our network outperformed all other methods except HFUT-CVers. The model’s performance improved substantially when incorporating additional per-frame features, such as the 7 basic expressions or 17 AUs. In the two-task experiments, we observed a further increase in the Pearson’s correlation coefficient ranging from 1% to 1.5%. Ultimately, when all three tasks were used together, our method achieved the highest performance.

To identify the optimal architecture for our network, we conducted experiments with various configurations, including different CNNs (e.g., ResNet50 instead of MMA) and RNNs (e.g., LSTM instead of GRU), as well as varying the number of layers and units, as detailed in Table 4. After evaluating a wide range of combinations, we determined that the most effective configuration consists of a single GRU layer with 128 units, followed by a feed forward layer with 32 units. Additionally, we evaluated the impact of incorporating the Mask layer, dynamic routing, and our proposed loss function (as an alternative to the conventional MSE). The results presented in Table 3 demonstrate that these components significantly enhance the performance of MMA-MRNNet.

Here we provide an extensive experimental study in which we utilise the top-performing methods from the ABAW Competitions (FUXI [58], SITU [45], CTC [61]) and the state-of-the-art methods (DACL [11], DAN [55], POSTER++ [47]; ME-GraphAU [46] & AUNets [52]) for 7 basic expression recognition, AU detection and valence-arousal estimation, and compared their performance to our proposed MMA component.

As can be seen on Table 5, our proposed MMA component outperformed all these methods on all tasks (7 basic expression recognition, AU detection and valence-arousal estimation) and on all utilized databases (Aff-Wild2, AffectNet and EmotioNet) by large margins.

The performance of the proposed MMA method was evaluated against several state-of-the-art approaches across multiple datasets (Aff-Wild2, AffectNet and EmotioNet), as detailed in Table 5. The MMA component consistently outperformed by large margins all methods on all tasks (7 basic expression recognition, AU detection and valence-arousal estimation) and on all utilized databases, across all evaluation metrics. Specifically, on the Aff-Wild2 dataset, MMA surpassed the closest competitor, SITU, for valence-arousal estimation, by 3.24%. It also outperformed the closest competitor, FUXI, for 7 basic expression recognition by 4%, as well as for AU detection by 3.38%. On the AffectNet dataset, MMA again demonstrated superior performance, outperforming the sota methods by at least 4.2% for valence-arousal estimation and by at least 2.2% for 7 basic expression recognition. On the EmotioNet dataset, MMA outperformed the sota methods by at least 2.6%. These results underscore the robustness and superiority of MMA in delivering precise and reliable affect representations.