Feature refinement: An expression-specific feature learning and fusion method for micro-expression recognition

The success of existing traditional approaches in MER is attributed in good part to the quality of the handcrafted visual features representation. Normally, handcrafted features are fed into a supervised classifier e.g. Support Vector Machines [9] to train a recognizer for the target expressions. The features are generally categorized into appearance-based and geometric-based features.

Recently, deep learning has been considered as an efficient way to learn feature representations. According to the different evaluation mechanisms, the existing methods were evaluated on the sole database, on Composite Database Evaluation protocol, and on Cross-database Micro-expression Recognition protocol, respectively.

As shown in Fig. 1, the expression-shared feature learning module consists of three critical components: apex frame chosen, optical flow extraction and two-stream Inception network.

Note that, as SMIC database [48] doesn’t supply the human-annotated apex frame, the apex frame spotting becomes very necessary. Several apex frame spotting algorithms have been proposed in recent years [49, 3, 50, 51, 52, 38]. For example, the mid-position frame is straightforwardly chosen as the apex frame [52, 38]. Moreover, Liu et al. [39] used motion difference to locate the apex frame. Quang et al. [40] divided the face image into ten regions, and then computed the absolute pixel differences to find the apex frame. Considering a trade-off between efficiency and effectiveness, the inter-frame difference method (interframe-Diff) is presented to locate apex frame on SMIC database. The interframe-Diff defines the index of apex frames $t_{apex}$ as follows:

where $I(x,y,t)$ denotes the $t$-th frame of each sample, $mean()$ computes the mean absolute value of the pixel value difference between the onset and the $t$-th frames at $(x,y)$.

As a motion information feature, the optical flow is extensively used by [37, 36, 53] for micro-expression recognition. More specifically, the optical flow of each sample is extracted from the onset and apex frames, where onset and apex frames mean the frames with neutral-expression and the highest expression intensity, respectively. For FR, TV-L1 optical flow method [54] is utilized to obtain motion feature from the onset and apex frames of each micro-expression video. As shown in Fig. 1, for preserving more motion information, two optical flow components are extracted to represent the facial change along horizontal and vertical directions.

Furthermore, Considering horizontal and vertical components of optical flow, FR designs two-stream Inception network based on the Inception V1 block [55], which complements with two-layer depths. Specifically, inception block is designed to capture both the global and local information of the optical component for feature learning. Distinguishing from traditional convolution with a fixed size of filters, Inception blocks parallelize filters of multiple sizes at the same level. Additionally, motivated by LeNet [56], the number of filters in the first and second layers is set at 6 and 16, respectively. Finally, the flattened two sets of feature maps are concentrated into an expression-shared feature.

As illustrated in Fig. 1, based on the expression-shared feature, expression proposal module with the attention mechanism and proposal loss is introduced to learn the expression-specific features. Given a micro-expression sample $x$ and an expression-shared feature $z$, a proposal module with $K$ sub-branches is designed, where each of sub-branch learns a set of specific-feature for each micro-expression category (total $K$ categories). The proposal module becomes the core component of FR to obtain salient and discriminative features. The $Softmax$ attention and proposal loss are described as follows.

Softmax attention: Attention mechanism provides the flexibility to our model to learn $K$ expression-specific features from the same set of expression-shared feature. $K$ attention units are shown in the part (2) of Fig. 1, and the detail of each attention unit is depicted in Fig. 2. Specifically, the expression-shared feature $z$ is connected with $K$ fully connected layers separately. After activated by $Softmax$, we gain the attention weight of $z$ for each specific expression,

where $a_{k}$ is a vector that has the same dimension as $z$, and $k\leq K$. Then, the representation $z_{k}^{*}$ for each specific expression is given by:

Proposal loss: Besides the inducing of attention strategy, expression-specific feature learning is also constrained with a prososal loss which is averaged from $K$ expression-specific detection losses in the proposal module. Each expression-specific detector (‘Detector$\_k$’ in Fig. 1) is aligned with a feature vector $z_{k}^{*}$, which contains two fully connected layers with a $Sigmoid$ layer as the output. Thus, each sub-proposal branch is trained by optimizing the following detection loss,

where $y_{p}^{k(i)}$ and $p({y_{p}}^{k(i)})$ denote the ground truth (either 1 or 0) and the probability of the $i$-th sample as the $k$-th category expression, respectively. $N$ is the total number of training samples. $\mathcal{L}^{(k)}_{detc}$ allows the network to generate expression-specific features for every expression. Based on the loss occurring in each sub-proposals, the loss of the proposal module consisted of $K$ sub-branches is defined as the average of $K$ detection losses:

By the restraint of the proposal loss of $\mathcal{L}_{prop}$ and the attention mechanism, the proposal module obtains salient and discriminative expression-specific feature for micro-expression recognition.

Generally, a simple way is to concatenate the expression-specific features into one feature vector, which is directly fed into Fully Connected Layers. However, this method will cause the high dimension and contain more trainable parameters. Motivated by the feature fusion method in [57], this work utilizes a simple but efficient method, namely, the element-wise sum as a fusion function. The efficiency evaluation about element-wise sum for fusion can be referred to Section 4.3. Consequently, the aggregated feature representation $z^{\prime}$ is defined as follows:

Then, the expression-refined feature $z^{\prime}$ is fed into the final classification module which consists of two fully connected layers. To avoid over-fitting, the first fully connected layer was followed by a Dropout layer (with dropout probability being 0.5). Lastly, the output of the last fully connected layer $F_{i}$ is activated by a $softmax$ unit. Hence, the classification loss $L_{cls}$ is expressed as:

where $y_{c}^{i}$ is the class label for the $i$-th training instance.

By joining the proposal loss of $\mathcal{L}_{prop}$ in Eq. (5) and classification loss of $L_{cls}$ in Eq. (7), a novel FR loss is proposed as follows:

where the hyper-parameter $\lambda$ balances the contribution of expression proposal and category classification.

Experiments are conducted on three commonly used spontaneous micro-expression databases: SMIC [48], CASME II [26], and SAMM [58].

SMIC [48]: The SMIC database contains SMIC-HS (recorded by a high-speed camera of 100 fps), SMIC-VIS (by a normal visual camera of 25 fps), and SMIC-NIR (by a near-infrared camera). SMIC-HS has 164 micro-expression clips from 16 subjects, while SMIC-VIS/SMIC-NIR consists of 71 samples from 8 participants. Additionally, these samples in three sub-databases are annotated as Negative, Positive, and Surprise. The sample resolution is $640\times 480$ pixels and the facial area is around $190\times 230$ pixels.

CASME II [26] : The CASME II contains two versions: the first one includes 247 samples of 5 micro-expression classes (Happiness, Surprise, Disgust, Repression, and Others), and the second has 256 samples of 7 classes (Happiness, Surprise, Disgust, Sadness, Fear, Repression, and Others). All the samples are gathered from 26 subjects. It was recorded by a camera with 200 fps. The resolution of the samples are $640\times 480$ pixels and the resolution of facial area is around $280\times 340$ pixels.

SAMM [58]: The SAMM database contains 159 micro-expression instances from 32 participants at 200 fps. The resolution of the samples are $2040\times 1088$ pixels and the resolution of facial area is around $400\times 400$ pixels. Samples in SAMM are categorized into Happiness, Surprise, Disgust, Repression, Angry, Fear, Contempt, and Others.

Since the apex frame annotation in SMIC database is not available, interframe-Diff method as described in Section III is applied to spot the apex frame. For the CASME II and SAMM databases, the ground truth of the apex frames is directly used. The Libfacedetection [59] is utilized to crop the facial area out of onset and apex frames. TV-L1 optical flow is extracted from the onset and apex frames. Two components of optical flow images are resized to $28\times 28$ pixels before feeding to the Inception network. All the experiments are conducted with Ubuntu 16.04, Python 3.6.2 with Keras 2.2.4 and Tensorflow 1.11.0 on 1 NVIDIA GTX Titan X GPU (12 GB).

Setup: To evaluate the effect of each module of FR, an ablation study is first designed to investigate the backbone selection, strategy selection, and fusion module selection. Second, three groups of experiments, namely, CDE experiment, CDMER experiment, and the single database experiment, are designed to validate the effectiveness of the proposed method. For fair comparison, all the experiments on CDE and the single database evaluation are conducted with Leave-One-Subject-Out (LOSO) cross-validation, where samples from one subject are held out as the testing set while all remaining samples for training. For CDMER, the model is trained on the source dataset and tested on the target dataset.

The detail settings for model ablation experiment, CDE experiment, CDMER experiment, and the single database experiment can be referred to Section 6.

Performance metric: Here, this work applies different performance metric for the three groups.

• 1.

  For CDE protocol, according to the MEGC 2019, Unweighted F1-score (UF1) and Unweighted Average Recall (UAR) are used to measure the performance of various methods on composite and individual databases.

• 2.

  For CDMER protocol, according to [41], Accuracy (Acc) and UF1 are reported for evaluating the performance.

• 3.

  For the evaluation on the single database, Acc is used.

All results of each type of experiments are the average of at least ten rounds. The evaluation metrics can be also referred to Section 6.

The ablation study is performed on the composite database of CDE protocol. Table 1 reports the results in terms of UAR and UF1 on different models.

(1) Backbone selection for the expression-shared feature learning

To better capture the subtle motion of micro-expression for our proposed model, we make a backbone selection for the expression-shared feature learning.

a. Dual-Inception [38]: The model straightforwardly uses the mid-position frame of each sample as the apex frame for MER.

b. Basic Inception: Different from [38], Basic Inception uses interframe difference algorithm described in Section III to approximately locate the apex frames for SMIC-HS database, and uses ground truth of apex frames in CASME II and SAMM.

Table 2(a) compares basic Inception to Dual-Inception on the composite database. Basic Inception outperforms Dual-Inception in terms of UAR and UF1. Thus, the Basic Inception is chosen as the backbone of our framework to learn the expression-shared feature.

(2) Strategy selection for the expression-specific feature learning

In the proposal module, two followed strategies are presented to learn expression-specific features:

a. FR-fc: FR-fc only uses fully-connected layers in the proposal module for the expression-specific feature learning, and then uses the element-wise sum mode to aggregate expression-specific features of each expression category for classification.

b. FR: FR differs from FR-fc using attention strategy to learn expression-specific features.

From Table 2(b), FR with attention mechanism boosts the performance from 0.7377 to 0.7838 in terms of UF1, from 0.7443 to 0.7832 in terms of UAR. This suggests that attention factors in the proposal module have the capability to highlight specific characteristics and generate salient features.

(3) Fusion mode selection to obtain the expression-refined feature

To validate the fusion mode for fusing the expression-specific features, this part compares element-wise sum mode used in this paper to concatenated mode.

a. FR-concatenated: FR-concatenated model directly concatenates the expression-specific features.

b. FR: FR model uses element-wise sum mode to obtain expression-refined features.

According to Table 2(c), FR outperforms the FR-concatenated. It may attribute to that element-wise sum mode alleviates the over-fitting problem as element-wise sum model more suppress the dimension of expression-refined feature than concatenated method. Consequently, the element-wise sum mode is chosen for expression-specific features fusion to obtain expression-refined features.

Table 2 compares our method to several state-of-the-art methods on CDE protocol. These methods contain two handcrafted features (LBP-TOP [10] and Bi-WOOF [22]), six deep learning features without data augmentation technology (AlexNet [60], GoogLeNet [55], VGG16 [61], OFF-ApexNet [36], Dual-Inception [38], and STSTNet [37]), and two deep learning features with data augmentation (CapsuleNet [40] and EMR [39]). Specifically, AlexNet, GoogLeNet, and VGG16 were reproduced by Liong et al. [37] instead by the inputs of optical flow features.

In this experiment, the proposed FR is further evaluated by using CDMER protocol [41]. CDMER protocol contains 12 sub-experiments from the TYPE-I and TYPE-II tasks. The detail setting of CDMER is referred to Section 6 and Table 10. Table 3 and Table 4 compare FR to the state-of-the-art algorithms referred by  [43]. Their parameter settings are described as follows:

• 1.

  For LBP-TOP [10], the uniform pattern is used. For the neighboring radius $R$ and the number of the neighboring points $P$, experiments consider three cases: $R=1,P=4$ (R1P4), $R=1,P=8$ (R1P8), $R=3,P=4$ (R3P4), and $R=3,P=8$ (R3P8).

• 2.

  For LBP-SIP [14], the neighboring radius $R$ is set at 1 and 3, respectively.

• 3.

  For LPQ-TOP [63], the size of the local window in each dimension is set as $[5,5,5]$, and the factor for correlation model $decorr$ is set as [0.1, 0.1] and [0, 0], respectively.

• 4.

  The number of bins $p$ is set as 4 and 8 for HOG-TOP and HIGH-TOP [62], respectively.

• 5.

  For three dimensional convolutional neural network (C3D) [64], the micro-expression features are extracted from the last two fully connected layers from the pre-trained Sport-1M [65] and UCF101 [66] model.

Table 5 compares our proposed FR to the state-of-the-art algorithms on four single micro-expression database.

As previously described in Section 3, expression-specific feature learning and fusion aim to learn the salient and discriminative feature. Here, to better reveal the effect of expression-specific feature learning and fusion module to FR, the comparison is conducted to compare the expression-refined features to the expression-shared features obtained by the Basic Inception on the composite database. The CDE protocol is used.

Table 6 reports comparison results in terms of UF1 and UAR. It is seen that with expression-specific feature learning and fusion module FR obtains the significant improvement which improves the Basic Inception from 73.60% to 78.38% in terms of UF1, from 73.91% to 78.32% in terms of UAR. This suggests that expression-specific feature learning and fusion module is the most contributed module for our FR. Furthermore, Fig. 3 shows the confusion matrices of Basic Inception and FR on each micro-expression category. As seen from Fig. 3, FR obtains the accuracy of 83.60%, 70.64%, and 80.72% for negative, positive, and surprise, respectively. When adding expression-specific feature learning and fusion module to FR, FR improves Basic Inception consistently with gains of 3.2%, 6.42%, and 3.61% for negative, positive, and surprise, respectively. This indicates that the expression-refined features can improve the salience in each class and highlight the specific characteristics of each type of expression, and thus perform better than the expression-shared features in individual categories.

In order to get more reasonable and stable feature distribution visualization, 34 subjects are randomly chosen, where 282 samples are used for training and the rest for testing. Fig. 4 shows the feature distribution of Basic Inception and FR from the testing samples, where all features are mapped to 2D using t-SNE [81]. For the Basic Inception model, the features are from the concentrated layer before the classifier, while for FR, the expression-refined features are obtained before the finally classification module. It is observed that the feature representations learned by FR are better separated, making the intra-class distribution more compact and the inter-class distribution more dispersed. These visualization results indicate that expression-refined features learned by FR are more discriminative than the expression-shared features learned by Basic Inception.

As previously described in Section 3, FR with the backbone of Basic Inception learns expression-shared feature, while FR with the expression-specific feature learning and fusion module learns and aggregates expression-specific feature. To analyze the complexity of the expression-specific feature learning and fusion of FR, Table 7 compares FR to Basic Inception in terms of accuracy (Acc), learnable parameters (Parameters), and execution time (ET) on three micro-expression databases.

According to Table 7, compared with the number of parameters in Basic Inception, the learnable parameters size indeed increases significantly in FR which is caused by the expression-specific feature learning and fusion. In other words, as SMIC-HS contains three categories, FR includes three expression-specific feature learning sub-branches. It leads to more 3.7615 $million$ parameters to learn, but only more 0.4949 $s$ to execute for FR when compared with Basic Inception. It reveals that although the number of parameters increases by adding expression-specific feature learning and fusion module, the additional execution time is still acceptable. The comparison results and analysis validate expression-specific feature learning and fusion is still efficient and effective for MER.

This previous parts extensively discuss on results under CDE, CDMER, and the single database evaluation protocols. According to the previously discussed results, several observations can be concluded in the following. First, algorithm in the single database of SMIC-HS obtained worse results than by using CDMER protocol. Second, algorithm failed to achieve same or similar performance under CDE protocol and in the single database experiment. The following will discuss these two observations.

(1) Why does algorithm under single database evaluation not always outperform under CDMER protocol?

According to Table 8, the performance of SMIC-HS database under the single database protocol is only better than that of CDMER protocol in the $Exp.4:N\rightarrow H$ and $Exp.7:C\rightarrow H$. In contrast, it works worse than CDMER in the $Exp.2:V\rightarrow H$. In $Exp.4:N\rightarrow H$ and $Exp.7:C\rightarrow H$, the data between SMIC-NIR/CASME II and SMIC-HS is heterogeneous. In other words, the data were collected under different conditions. In contrast, in $Exp.2:V\rightarrow H$, samples in SMIC-VIS recorded by a normal visual camera are more similarly to the sample in SMIC-HS database. As well, both databases contains the same participants.

The recent works [39, 45, 30] have indicated that the model leveraging on macro-expression can boost the deep neural network. The way may benefit to micro-expression recognition: Compared with collecting more micro-expressions of a person to reveal one’s true feeling, it is much easier to collect the person’s macro-expressions. It motivates us to leverage macro-expressions transferring learning method to boost our proposed FR in future. Additionally, leveraging the subject information and transfer learning strategy allows us to obtain better recognition performance.

(2) Why does algorithm under CDE protocol outperform that under the single database evaluation?

This is explained by that increasing number of samples from other micro-expression databases and the optical flow feature contribute to FR. As we know, more samples can partly avoid from the overfitting problem. On the other hand, in our framework, optical flow is fed into FR. Optical flow mainly focuses on extracting motion feature of samples, which mostly suppresses the facial identity. Consequently, besides transfer learning and other domain adaption mechanisms, adding samples from different micro-expression databases and also utilizing proper motion feature to train the model is a considerable approach to obtain better performance of MER.

Firstly, to easily understand the settings of the four types of experiments, we give the detailed description of the experiment settings as follows.

The Acc, UAR, and UF1 metrics are defined as follows:

and

where,

and

where $K$ is the number of classes, $M$ is the number of folds of LOSO, $n_{k}$ is the total number of samples in the ground truth of the $k$-th class, and $Acc_{k}$ is the per-class accuracy scores. For the $m$-th fold of LOSO by the $k$-th class, $TP_{k}^{(m)}$, $FP_{k}^{(m)}$, and $FN_{k}^{(m)}$ are true positives, false positives, and false negatives, respectively.