Exploring Ordinal Bias in Action Recognition for Instructional Videos

Instructional video analysis has emerged as a prominent area of research in the field of video comprehension. In particular, multiple datasets of instructional videos (Fathi et al., 2011; Stein & McKenna, 2013; Kuehne et al., 2014) have been introduced, offering extensive contextual information on human activities. Although various methods have been proposed (Farha & Gall, 2019; Li et al., 2020; Ishikawa et al., 2021; Yi et al., 2021; Liu et al., 2023), and Li et al. (2022) explores the understanding of ordinal action by explicitly modeling inter-action context, the ordinal bias issue can overestimate their performance. This comes from their exploitation of the action sequence observed during training stage.

Recent work has advanced reliable video systems and evaluations across various video understanding tasks. Otani et al. (2020); Yuan et al. (2021); Jung et al. (2024) have analyzed the reliable evaluation of temporal understanding, and Li et al. (2018); Hara et al. (2021) have raised the bias problem in action recognition. Consequently, a line of works (Zhai et al., 2023; Li et al., 2024) have explored the dual challenges of background and foreground biases, demonstrating that action recognition models can be inadvertently biased by static and dynamic cues. Although the previous works (Nam et al., 2020; Duan et al., 2022) have addressed the bias by retraining the model, these approaches are computationally expensive and overlook the underlying imbalance of the data set. In contrast, we directly address the issue by manipulating the video data itself and provide insights into how current models perform if they are trained with these video variants.

We utilize three action recognition datasets: Georgia Tech Egocentric Activities (GTEA) (Fathi et al., 2011), 50Salads (Stein & McKenna, 2013), and Breakfast (Kuehne et al., 2014). GTEA includes 28 videos depicting daily kitchen activities, featuring 11 action categories. Each video has an average of 20 action units and a duration of approximately 30 seconds. 50Salads contains 50 videos of actors preparing salads in various kitchen environments, with more than 20 actors participating. The videos in 50Salads are more than six minutes long and cover 17 action categories. Breakfast consists of over 1700 videos that contain breakfast preparation scenes and has 48 action categories. This dataset has the most complex labeling scheme among the three datasets.

We consider five distinct models: MS-TCN (Farha & Gall, 2019), MS-TCN++ (Li et al., 2020), ASRF (Ishikawa et al., 2021), and DiffAct (Liu et al., 2023). MS-TCN employs a multi‑stage architecture with dilated temporal convolutions and a smoothing loss to iteratively refine frame‑level predictions. MS-TCN++ extends this approach by integrating dual dilated layers that capture both local and global contexts while decoupling prediction generation from refinement. ASRF improves segmentation quality by adding an auxiliary branch that explicitly regresses action boundaries to mitigate over‑segmentation errors. ASFormer leverages a Transformer‑based framework augmented with temporal convolutions and a hierarchical representation pattern for iterative prediction refinement. Lastly, DiffAct formulates action segmentation as a conditional sequence generation task that iteratively denoises a noisy action sequence by leveraging priors such as positional, boundary, and relational cues. For our experiments, we utilized I3D (Carreira & Zisserman, 2017) video features, pre-trained on the Kinetics dataset (Kay et al., 2017), and a single NVIDIA RTX 3090.

To evaluate the performance of the model, we adopt frame-wise accuracy, a primary metric that gauges the percentage of accurately classified actions within a unit frame of a test dataset. To produce a result, we use 5-fold cross-validation to evaluate the proposed approach’s performance on the 50Salads dataset. For the remaining datasets, 4-fold cross-validation is performed to estimate the average performance measure.

We begin by analyzing the distribution of action pairs across the datasets. As shown in Figure 2, each dataset exhibits a pronounced long-tailed pattern. In detail, in the Breakfast dataset, 16 of 228 action pairs represent 30% of all action pair occurrences. Similarly, in the 50Salads dataset, 8 of 120 action pairs contribute to 30% of the total occurrences of action pairs”, and in the GTEA dataset, only 3 of 32 action pairs comprise 30% of the total action pair occurrences. This skewed distribution may lead to biased predictions, as models can become overly influenced by the few frequently occurring action pairs, potentially misrepresenting the diversity of real-world instructional videos. To address this issue and enable more reliable evaluations, we introduce video manipulation methods designed to counteract the effects of this long-tailed distribution.

We propose two video manipulation techniques, Action Masking and Sequence Shuffling, as shown in Figure 3. For action masking, we mask the video frames of a specific action unit, and the corresponding action label is replaced with ‘no action.’ By doing so, we verify whether the model predicts ‘no action’ accounting for visual variants or if it makes biased predictions. The second method, sequence shuffling, randomly rearranges the order of action segments without altering the frame order within each action unit. This allows us to construct datasets with distinctive label distributions, mitigating the influence of skewed distributions, and enabling reliable evaluations. Importantly, sequence shuffling preserves the internal semantic coherence of each action unit since the visual continuity of each segment remains intact. Although altering global action sequences may challenge models that implicitly rely on positional cues, this manipulation explicitly targets the assessment of a model’s robustness against ordinal bias. Consequently, sequence shuffling facilitates rigorous evaluation of model performance in scenarios involving unfamiliar action orders, ensuring meaningful insights into their generalization capability. Further study can be found in Appendix C.

Figure 4 shows the result of our methods, demonstrating that our manipulation methods successfully change the distributions of action pairs. Specifically, for action masking, all the subsequent labels of ‘close’ have been switched to ‘background’, whereas the sequence shuffling reduced the maximum value of high occurrences and introduced a new pair of actions. Also, in the sequence shuffling, the number of existing biased pairs is decreased while previously absent action pairs are created. More visual examples can be found in Appendix D.1

Now, we apply the action masking technique to the original dataset and conduct an experiment to see how the model trained with the original dataset behaves when it encounters a masked section. We first select the action pair that is frequently seen in the original dataset. We selected action pairs according to these criteria: 1) Considering the initial action, we observe the frequency of subsequent actions to determine if a particular combination is significantly more common compared to the others. 2) Subsequent actions should not equate to ‘no action.’ As a result, in the GTEA dataset, ‘close’ is used as a prior action, making up about 7.5% of the entire dataset, with the ‘put’ combination comprising approximately 95.5% of these actions. Similarly, in the Breakfast dataset, ‘pour_dough2pan’ serves as the initial action, accounting for around 1.6% of the total dataset, while ‘fry pancake’ constitutes nearly 91.1% of the subsequent actions. Then, we mask frames that correspond to the latter action unit and replace its action label with ‘no action.’ Lastly, we make the model predict the masked parts and inspect the accuracy.

Figure 5 shows the results of our experiment, demonstrating that the model finds it difficult to accurately predict the manipulated test videos. This result indicates that the model misclassifies masked regions as having an action label from the original dataset instead of identifying them as ‘no action,’ suffering from the ordinal bias problem. The result also reveals that the model does not utilize visual information, but exploits spurious correlation for prediction. We will discuss the ordinal bias problem in detail in the next section.

We investigate the extent to which a model is responsible for the problem by training it on a manipulated dataset using the sequence shuffling method described in Section 4.2. Here, we skip the action masking method as it only introduces no-action labels; therefore, specific actions are only followed by ‘no-action.’ In contrast, sequence shuffling provides more diverse action pairs, allowing us to assess how models handle varied action patterns. We then evaluate the performance of the model on the original dataset and sequence shuffling. A model with satisfactory generalization should exhibit good performance in the original dataset despite being trained on a manipulated one. Table 1 shows a significant performance discrepancy between the model trained on the modified dataset (S/O) and the superior performance of the model trained on the unchanged dataset (O/O).

Furthermore, we have investigated whether these results come from ordinal bias by comparing the label distribution among the original dataset, the manipulated dataset, and the model’s predictions. If the model is robust, its prediction distribution (green) should resemble the manipulated dataset distribution (blue), rather than the original dataset distribution (red). Figure 6 shows results, which implies that the model tends to make predictions by following the trend of the training dataset, not by utilizing given visual information. This outcome implies that the model exploits spurious correlations during inference to achieve higher scores, resulting in an overestimation. Therefore, a model must have an improved generalization capability to reduce the ordinal bias.

In many bias-related problems, the incorporation of additional data helps alleviate bias. Therefore, we investigate whether training models with an additional augmented dataset can mitigate ordinal bias in action recognition. To this end, we designed a curriculum learning-like strategy by sequentially training the model on three variants of the dataset: the original, the masked, and the shuffled versions. This progression is intended to gradually expose the model to increasing difficulty levels and reduce its reliance on spurious correlations. However, as highlighted in Section 5.1, the use of action masking is not suitable for model training. To overcome this, we used a strategy for action masking inspired by Masked Language Modeling (Devlin, 2018), where actions are randomly hidden with a likelihood of 15%, instead of our original approach. In contrast, for sequence shuffling, we utilized the method we had proposed. Consequently, every augmented dataset retains its original size, allowing the model to be trained on a dataset that is triple the size of the initial one, which we refer to as the ‘Combined’ dataset.

However, as shown in Table 2, the model trained with an additional dataset did not exceed the performance of the model trained solely on the original dataset. These results indicate that simply augmenting the training data, even through a curriculum-learning-like approach, does not effectively mitigate ordinal bias. This suggests that the bias is deeply ingrained in the training dynamics, and additional intervention, such as architectural modifications or specialized loss functions, may be required to address the issue. The ablation study can be found in the Appendix B.