Advancing Human Action Recognition with Foundation Models trained on Unlabeled Public Videos

The evolution of action recognition research has been significantly shaped by the development and availability of benchmark datasets, which guide and validate advancements in computer vision methodologies. The UCF101[3] dataset, known for its diverse array of human actions, has been instrumental in pioneering research within this domain. Likewise, the HMDB-51[4] dataset, despite its smaller scale, offers a rich variety of human actions captured under realistic conditions, providing essential challenges for model evaluation.

Introduced in 2017, the Kinetics Human Action Video dataset [1] became a cornerstone in this field with over 400 distinct human action classes derived from YouTube videos. It set a new benchmark, catalyzing further research and development in action recognition technologies. Similarly, AVA [6] provides densely labeled video segments that allow for action localization in addition to recognition, offering a deeper understanding of context and action dynamics.

Emerging datasets such as Ego4D [7], which focuses on first-person video analysis, and HowTo100M [8], which leverages instructional videos for action recognition, reflect the diversifying sources and modalities being explored. EPIC-Kitchens [9], ActivityNet [10], and THUMOS [11] offer further variations in complexity and scenarios, pushing the envelope on what models can understand and predict.

Recent datasets have addressed limitations of earlier collections. The Something-Something dataset [5] focuses on human-object interactions within everyday scenarios, offering a unique perspective that highlights the complexity of routine human activities. Moments in Time [12] and Multi-Moment Dataset [13] extend this by providing large-scale resources for temporal action understanding, capturing short videos that encompass a wide range of activities. The recent Tencent-MVSE dataset [14], another substantial contribution, is designed for multi-modal video similarity evaluation, adding another layer of complexity to action recognition tasks.

Building upon the diversity and scope of existing datasets, the TikTokActions dataset is meticulously designed to capture the rapidly evolving landscape of social media-driven human interactions. Unlike traditional datasets, which often focus on pre-defined, standard action categories, TikTokActions delves into a broader spectrum of spontaneous and culturally varied human behaviors, which are inherently more aligned with contemporary social media trends. This alignment is critical as platforms like TikTok foster unique user-generated content that is not only highly dynamic but also reflective of current societal norms and interactions.

In direct comparison to the Kinetics dataset, which largely aggregates content from YouTube, TikTokActions offers an array of actions driven by the unique constraints and creativity of TikTok’s platform, such as shorter video lengths and a more diverse global user base. This distinction is significant as it introduces new challenges in action recognition, particularly in recognizing quick, context-driven actions that are less prevalent in the typically longer-form content of YouTube.

To substantiate the dataset’s novelty and utility, we conducted a thorough analysis of the action categories present in TikTokActions compared to those in Kinetics and other major benchmarks. Our findings reveal that TikTokActions includes a higher proportion of non-standard actions, such as specific dance moves, challenge-based activities, and regional cultural expressions, which are underrepresented in other datasets. This inclusion not only enhances the dataset’s coverage of real-world scenarios but also provides a more granular understanding of human actions within the digitally connected world.

By providing detailed statistics on the distribution of these categories, along with examples in our supplementary materials, we demonstrate the unique value of TikTokActions in filling the existing gaps within the field of action recognition. This detailed comparison and analysis affirm that TikTokActions is not just an addition to the plethora of action recognition datasets but a necessary evolution to keep pace with the changing dynamics of human activity in the digital age.

Foundation models in computer vision have revolutionized the way we approach diverse tasks across modalities, offering scalable solutions that extend from image to video understanding. The Florence model [15], Specifically, encapsulates the progression of foundation models, adeptly offering transitions from broad scene outlines to detailed object-centric views and from static imagery to dynamic video sequences. Additionally, its effectiveness spans various modalities, from RGB imagery to textual data, establishing new standards in tasks like classification, object detection, retrieval, and particularly in action recognition.

Significant advancements have also been seen with the introduction of VideoMAE V2 [2]. This self-supervised pre-training methodology stands out for its ability to train on datasets exceeding 1 million videos. With its innovative dual masking strategy and progressive training approach, VideoMAE V2 sets new standards in action recognition. Empirical evaluations demonstrate its superior performance across multiple benchmarks, emphasizing its effectiveness in real-world scenarios.

Further developments include ViViT [16] and TimeSformer [17], which adapt transformer architectures specifically for video, offering improvements in handling the temporal dynamics of action sequences. These models demonstrate the potential of transformers in capturing complex, time-related patterns in video data.

The Quo Vadis model [18] introduced a new approach by integrating the Kinetics dataset to set a performance benchmark, inspiring subsequent models to focus on improving temporal understanding and action recognition capabilities. Similarly, Temporal Relational Reasoning in videos [19] provided insights into the importance of temporal dynamics, influencing how subsequent models like Non-local Neural Networks [20] handle long-range dependencies within video sequences.

A Closer Look at Spatiotemporal Convolutions [21] further explored the convolutional network architectures, focusing on how variations in spatial and temporal modules affect performance on action recognition tasks. This study paved the way for more specialized models like the Video Action Transformer Network [22], which introduced an approach to directly model relationships between actions and corresponding contextual features in videos.

Furthermore, 2D CNNs such as TSN [23], TSM [24], TANet [25], and TDN [26] have been influential by effectively incorporating temporal information, a critical aspect of video understanding.

The use of 3D CNNs such as I3D [18], R(2+1)D [21], ARTNet [27], and Slow-Fast [28] has further pushed the envelope in spatial-temporal feature integration, enhancing model performance on complex video tasks.

In the domain of transformers, advancements have continued with models such as MViT [29] and Video Swin Transformer [30] which have introduced ways to integrate spatial and temporal features more effectively, pushing the boundaries of action recognition performance. Additionally, UniFormer [31] integrates local and global interactions more efficiently, which is crucial for detailed action understanding in videos.

The diversity of foundation models extends to multi-modal approaches, where models like Perceiver IO [32] and Data2Vec [33] have been developed to handle various inputs ranging from text to video, showing flexibility and robustness in learning from heterogeneous data sources.

In the context of enhancing action recognition, SlowFast Networks [28] and X3D [34] introduce novel architectural choices that optimize the processing of temporal variations and the physical intensity of actions. These developments underscore the trend towards more specialized and efficient models that are not only capable of high performance but also adaptability across different action recognition scenarios.

Our TikTokActions dataset, with its focus on spontaneous and diverse user-generated content, provides a rich testing ground for these advanced models. By evaluating these models on our dataset, we aim to explore how well they can adapt to and interpret the nuanced, quick-transition actions typical of content found on modern social media platforms. The insights gained can help tailor these foundation models to better handle real-world variability and complexity in human actions.

In our efforts to push the frontiers of action recognition further, we also draw upon the extensive research documented in the Multi-dataset Training of Transformers [35], which explores the robustness gained from training across varied video datasets. Such approaches are crucial as they contribute to the generalization capabilities of models, preparing them for deployment in diverse, unstructured environments.

To support our experimentation exploring the utility of unlabeled public data for pre-training foundation models for human action recognition, we curated an extensive collection of TikTok videos. This collection, consisting of over 280,000 video clips from 386 hashtags, captures a wide spectrum of human actions across various scenarios. The videos span diverse categories, from dance movements and fitness exercises to hand gestures and daily activities. Unlike traditional datasets like Kinetics, our collection uniquely incorporates elements of pop culture, memes, and rapid trend evolution, providing dynamic content that significantly boosts the model’s ability to understand and generalize from real-world, culturally relevant scenarios.

To curate a diverse and relevant set of videos from TikTok, we initially examined several well-known action recognition datasets, including UCF101[3], HMDB51[4], and Google DeepMind Kinetics[1]. These datasets helped us to understand the broad categories of human actions typically represented in research. We identified four primary categories: dance, sports, fitness, and kinetics.

Building upon these categories, we explored TikTok to identify relevant hashtags that capture a wide array of human actions within these domains. Our selection was strategically guided by six principles designed to ensure a comprehensive and culturally relevant collection: generalization, specification, noun substitution, noun-verb order, hashtag consolidation, and social/cultural relevance. These principles helped ensure that the videos not only cover a broad spectrum of actions but also include dynamic and culturally significant content such as memes, pop culture trends, and unique challenges—elements that are distinctly prevalent on TikTok.

The social/cultural relevance was particularly important, allowing us to incorporate an additional category that includes a wide variety of actions originating from distinct social and cultural contexts, such as viral dance challenges and fitness trends. Popularity and engagement levels also played a crucial role in our selection process; we set a minimum threshold of 5 million views per hashtag to ensure that the content is not only relevant but also resonates widely with users, reflecting substantial public engagement and visibility. This approach allows us to harness rich, culturally nuanced content that can aid in training models to better understand and interpret human actions in real-world contexts.

We curated a comprehensive video collection using the official TikTok research API, focusing on approximately 900 videos per hashtag from a set of hashtags identified through our selection process. Each video and its metadata were processed to facilitate model training, using tools such as PySceneDetect and YOLO v8[36]. This processing resulted in a JSON file for each video, which included metadata capturing attributes like title, duration, and scene distribution, as analyzed by PySceneDetect.

VideoMAE V2[2] is a state-of-the-art masked autoencoding framework, notable for its flexibility in backbone selection, ranging from base to billion-level configurations. We opted for both the ViT-base and ViT-giant backbones. The ViT-base was used to provide a baseline comparison with other models using similarly sized backbones, while the ViT-giant was employed to explore the maximum potential of the dataset, aiming to achieve the highest possible accuracy. The generalization capability of VideoMAE V2 pre-trained ViTs[37] as video foundation models has been demonstrated across multiple downstream tasks, highlighting its robustness and adaptability.

Our training was conducted on 8 NVIDIA A100 40GB GPUs, utilizing the full set of 283,582 unlabeled TikTok videos. The pre-training on the ViT-base backbone, which ran for 800 epochs and lasted for 20 days, provided a comprehensive base for developing a robust model capable of understanding a wide array of human actions. The ViT-giant backbone was pre-trained under similar conditions, running for 1200 epochs over 30 days, to push the performance boundaries further.

After initial pre-training, the model underwent two distinct paths of fine-tuning. The first path involved direct fine-tuning on the Kinetics-400 and Something-Something V2 datasets to assess immediate applicability to broad action recognition tasks. The results of this direct fine-tuning are detailed in Table 2 and Table 3. The second path involved using the model fine-tuned on Kinetics-400 as an intermediary for transfer learning on UCF-101 and HMDB51, leveraging robust pre-training on TikTok videos. The outcomes of this transfer learning process are presented in Table 4.

Observations on data scaling. We studied the influence of pre-training data size on the performance of VideoMAE V2. In this experiment, we pre-trained the video models with ViT-B and ViT-g backbones on our TikTok dataset, which contains approximately 0.28M videos. The fine-tuning accuracy is detailed in Table 2 for Kinetics-400 and Table 3 for Something-Something V2. Despite using approximately five times fewer pre-training samples compared to the UnlabeledHybrid dataset (1.35M videos), our model achieved comparable results. Specifically, the ViT-B backbone pre-trained on TikTok data achieved 81.1% on Kinetics-400, which is only 0.4% lower than the 81.5% achieved using the much larger UnlabeledHybrid dataset. Similarly, for Something-Something V2, the ViT-B backbone pre-trained on TikTok data achieved 69.2%, compared to 71.2% with the UnlabeledHybrid dataset, showing a difference of 2%. These small differences highlight the efficacy of our foundation model on TikTok videos, which did not include any information from the original fine-tuned dataset.

Observations on model scaling. We also explored how different model capacities affect performance. The fine-tuning results for models pre-trained with ViT-B and ViT-g backbones are presented in Table 2 and Table 3. The ViT-g model, representing a billion-level parameter architecture, consistently demonstrated improved performance over the ViT-B model. For example, the ViT-g backbone pre-trained on TikTok data achieved 85.51% on Kinetics-400, compared to 81.1% for the ViT-B backbone, showing an improvement of 4.41%. Similarly, for Something-Something V2, compared to the ViT-B backbone, there is an improvement of 5.07%. These results suggest that increasing model capacity can further boost performance, validating the potential of large-scale pre-training on diverse video datasets like those from TikTok.

Observations on transfer learning. We further evaluated the performance of our pre-trained models by using Kinetics-400 fine-tuned models for transfer learning on HMDB-51 and UCF-101 datasets. The results are summarized in Table 4. The ViT-B model achieved 83.9% on HMDB-51 and 97.5% on UCF-101, while the ViT-g model achieved 86.08% on HMDB-51 and 99.05% on UCF-101.

Overall, our findings highlight the significant potential of using large, unlabeled video collections from platforms like TikTok for advancing human action recognition tasks. This approach capitalizes on the rich, varied nature of content available on social media, which, when used for pre-training, enhances the model’s ability to generalize across different datasets and action recognition scenarios. The robust performance observed across both direct fine-tuning and transfer learning paths underscores the value of TikTok videos in training models for complex action recognition challenges. The TikTok dataset, which is unlabeled and lacks ground truth annotations, still produced promising results, demonstrating the model’s capacity to learn effectively from unstructured video content.