Data Augmentation for Human Behavior Analysis in Multi-Person Conversations

Bodily behavior recognition aims to recognize the category of behavior (e.g., shrug, leg crossed, and fold arms in Figure 1) through the given videos from different views, so it is formulated as a multi-label classification task. As shown in Figure 2, given the raw video sequence $V\in\mathbb{R}^{T\times 1000\times 1000\times 3}$, we first remove the irrelevant background to construct the cropped video $V^{\prime}\in\mathbb{R}^{T\times 1000\times 600\times 3}$. Subsequently, following the official baseline (Müller et al., 2023), we utilize the video swin transformer (Liu et al., 2022b) as the baseline to build a multi-label classification model. Data augmentation plays a crucial role in deep learning. By expanding and diversifying the training dataset, it enhances the model’s ability to generalize and learn robust representations. As shown in Figure 2, we can see that the participants’ clothing and background colors in the video are very similar, which can easily cause confusion between the foreground and background, affecting the accuracy of model recognition. Based on this characteristic, we utilize the “ColorJitter” data augmentation strategy to enhance the sensitivity of the model to the background. For detailed analysis and experimental results, please refer to Section 3.2.

Eye contact detection aims to estimate the gaze direction of each participant. In this sub-challenge, eye contact is defined as a discrete indication of whether a participant is looking at another participant’s face. Thus, this sub-challenge can be formulated as a multi-class (i.e., left, frontal, right, and no eye contact) classification task. Following previous work (Müller et al., 2021; Ma et al., 2022), as shown in Figure 2 (b), we first use the OpenFace 2.0 (Baltrusaitis et al., 2018) to detect and align faces from the original video and crop them to 224$\times$224 images. Then, these images are input to swin transformer (Liu et al., 2021) for multi-class classification. Similar to the bodily behavior recognition task, we utilize the “ColorJitter” and “Lighting” data augmentation strategies into the training process to improve the robustness of the model.

Next speaker prediction aims to predict the speaking status of each participant one second after the end of the 10s context window. Thus, this sub-challenge can be formulated as a multi-label classification task. Unlike previous work (Ma et al., 2022), we attempt to predict the next speaker directly from the original video frame sequence without using cropped facial features. As shown in Figure 2 (c), we first extract the video frames, then concatenate these frames according to the order in the annotation file. Subsequently, the concatenated frames are input to the video swin transformer (Liu et al., 2022b) to perform multi-label classification to predict the next speaker. Due to the limited time available for the challenge, we did not have time to attempt to extract facial features or apply data augmentation to improve the accuracy of predictions.

As shown in Table 1, we report the performance comparison between our approach and other methods. Considering the large amount of data for behavior recognition, we first attempt to use the officially provided skeleton data. Specifically, we trained the PoseC3D model (Duan et al., 2022) using frontal view skeleton data as input and achieved a mAP of 0.3810 on the validation dataset. We believe that directly using the PoseC3D model cannot achieve good results in action recognition, and further adjustments are needed to the input features. Considering the limited number of online evaluations, we did not make in-depth improvements on the skeleton-based action recognition models. Compared with the baseline with the frontal view as input, our approach achieves the best performance of 0.6262 on the test set with 9.47% improvements. Due to limitations in the training duration of the model, we did not use videos from three views to train the model. Observing the third row of Table 1, we can see that model using videos from all views can significantly improve performance (i.e., 0.5628 vs. 0.5315 on the test set). We believe our method will further improve if we use videos from three views to train the model. As shown in Table 2, we give the ablation study results of our method. The RandomErasing (Zhong et al., 2020) strategy randomly erases a rectangle region with random values, which is widely used in image and video classification tasks. Specifically, we increase the probability of “RandomErasing” from the initial 0.25 to 0.5. Observing the first and third rows of Table 2, we can see that the cropped videos can improve the mAP, i.e., (0.4757 vs. 0.4679 on the val set). Observing the third and fourth rows, we can conclude that the “CorlorJitter” and “RandomErasing” data augmentation strategies can further enhance the mAP of the model (i.e., 0.6262 vs. 0.6143 on the test set).

The validation and test set results of eye contact detection are shown in Table 3. Compared with SVM-RBF (Müller et al., 2021), our method achieves a relative improvement of 49.44% in terms of accuracy on the test set. MH (Fu and Ngai, 2021) and Baseline (Müller et al., 2022) employ different feature sampling strategies. For improved accuracy, the baseline trains four eye contact detection models for each seating position and produces an accuracy of 0.64 and 0.576 on the validation and test sets, respectively. We use the face image extracted by OpenFace 2.0 (Baltrusaitis et al., 2018) as input. Compared with the latest methods (i.e., TA-CNN (Ma et al., 2022) and TA-CNN (Ma et al., 2022)^†), our method achieves the best accuracy of 0.8122 on validation set and 0.7771 on the test set. The significant improvement means that our method has superior generalization capabilities.

The ablation study results of eye contact detection are shown in Table 4. Firstly, we use the last frame face feature of each video as input and use Swin V1-B (base model) and Swin V1-L (large model) as baseline methods. On the validation set, Swin V1-B achieves 0.8175, i.e., 0.71% higher than Swin V1-L. We attribute this to the fact that the Swin V1-L model may be difficult to fully converge on this small dataset. Secondly, we expand the dataset using face images from the last 5 frames of each video and add two data augmentation methods. “ColorJitter” can randomly change the brightness, contrast, and saturation of an image. “Lighting” can adjust image lighting using AlexNet-style PCA jitter. These two data augmentation strategies are widely image transforms. Both data augmentation strategies contribute enhancement to the results on the validation set. Finally, we experiment on the latest Swin V2 model (Liu et al., 2022a) as well and utilize the data augmentation methods mentioned above, reaching 0.8307 on the validation set. Due to limitations on the number of tests, we are unable to provide results for the test set.

As shown in Table 5, we report the experimental results compared to previous methods. The baseline methods utilized static and dynamic features extracted over the complete input video (frame-by-frame) via OpenFace 2.0. GLFVA (Birmingham et al., 2021) used the combination of group-level focus on visual attention features and publicly available audio-video synchronizer models, which achieved the best score of 0.632 on the test set in this sub-challenge. TA-CNN (Ma et al., 2022) utilized group-level features, achieving 0.6538 and 0.5766 on the validation and test sets, respectively. Similar to the sampling strategy of TA-CNN, we extracted the last frame of the video from different viewpoints for each sample. Then, the frame sequence organized according to the order of viewpoints is used as input to the network. Considering the limited computing resource and evaluation chance, we have not yet had time to experiment with the methodology of previous work (Birmingham et al., 2021; Ma et al., 2022) extracting facial features as input to the model. Unlike the above methods with individually cropped face images, we attempted to use the complete original frames to capture information about changes in the physical behavior of different participants and predict the next speaker based on it. Although our results on the test set reached only 0.5281, a new attempt was made to explore the task of predicting the next speaker.