A Human-ML Collaboration Framework for Improving Video Content Reviews

The importance of content moderation on online video platforms such as TikTok, YouTube or Instagram is growing (Gogarty, 2022; Mellor, 2022). These platforms strive to accurately detect the presence of policy violations within the video, which drive enforcement actions, e.g., the video can be taken down. Given the complexity of this problem, content moderation relies heavily on human judgement and employs large teams of content moderators to perform reviews. Since human annotations directly lead to high stakes decisions, such as content take downs, the quality of the annotations is critical.

For content moderation decisions there is a growing need for transparency in detected policy violations to provide feedback to content creators (Jhaver et al., 2019). This motivates a in-video taxonomic annotation task, where the goal is to provide localized and fine-grained policy-specific annotations, i.e., both the time regions (video segments) and the exact policies violated, which inform downstream video content moderation decisions. To cover the spectrum of potential violations, policy playbooks typically contain hundreds of fine-grained policies. This large space of policy violations can be organized as a taxonomy of broad categories such as Profanity, Violence, Nudity, etc., each of which contains several granular violations. For instance, Violence could include a range of granular classes such as animal abuse or graphic violence in video games. The class definitions are complex, ambiguous and often require nuanced judgment to apply, e.g., graphic violence. New policy classes may be added over time as well, e.g., Covid anti-vaccination. Moreover, there is a class imbalance issue - some egregious violations may be very rare.

Our goal is to maximize the quality and efficiency of the complex, granular, localized policy annotations task, hence leading to the correct video level enforcement decision. We achieve this by tackling the key issue of ”information overload” faced by raters in providing high quality annotations, where 1) the sheer volume of videos on large online video platforms means raters only have limited review time per video; and 2) the large taxonomy of policies makes it hard for raters to recall the complete set of all granular violations for every video region they watch in the limited review duration.

We propose a human-ML collaboration framework to maximize human ratings quality and efficiency by addressing ”information overload”. We train models on granular rater annotations to predict policy violations, which are then combined with innovative front-end elements in the rating tool to provide ”hints” to assist raters. We borrow from information retrieval literature and use ranking mechanisms for identifying the most useful and succinct set of hints. In experiments, we show that this enables raters to efficiently label policy violations more correctly and comprehensively. The human interactions with model hints pave the way for leveraging human feedback to improve the underlying ML models.

The crowd sourcing literature is very rich in the application of human annotations to perform a variety of tasks such as text processing (Kittur et al., 2008; Bernstein et al., 2010; Hu et al., 2011), audio transcription (Lasecki et al., 2012), taxonomy creation (Chilton et al., 2013), and social media analysis (Zubiaga et al., 2015; Founta et al., 2018). Although there is existing literature on video annotation, it is primarily focused on identifying actions or labeling entities easily distinguishable by humans using visual information only, e.g., high jump, thunderstorms. The primary goal of these tasks is to create large datasets for facilitating Machine Learning/Perception applications (Zhao et al., 2017; Trujillo et al., 2022), e.g., the YouTube-8m dataset (Abu-El-Haija et al., 2016). This is very different from our set up, where raters annotate granular, ambiguously defined policies using multi-modal signals - video, audio and text, from the transcript, and the video title and description. The recent emergence of crowd sourcing literature on content moderation primarily covers textual content such as user comments (Lai et al., 2022; Chandrasekharan et al., 2019), however there has been little focus on video moderation tasks.

Human-ML collaboration is an emerging area of research with two main categories of work:

ML-Assisted Human Labeling: ML-assistance through predictions and explanations has been used to improve the quality of human decisions in several domains (Beede et al., 2020; Lai and Tan, 2019; Park et al., 2019), including content moderation (Chandrasekharan et al., 2019; Lai et al., 2022). Crossmod (Chandrasekharan et al., 2019), for instance, uses a model trained on historic cross-community moderation decisions to enable Reddit human moderators to find more violations. Interactive ML-assistance, which we leverage in Section 3.1.2, is used by Bartolo et al. (Bartolo et al., 2021) to assist human annotators to develop adversarial examples for improving a natural language question answering model. ML-assistance has been shown to also improve the efficiency of the human labeling task (Ashktorab et al., 2021; Chandrasekharan et al., 2019; Anjum et al., 2021). Our work aims to exploit both the human labeling quality and efficiency benefits.

Improving ML-models Through Human Annotations: Human annotations are useful in constructing hybrid human-ML systems that leverage the complementary strengths of both to improve the performance of ML models (Arous et al., 2020; Vaughan, 2018). Existing work on active learning (Settles, 2009; Yang et al., 2015) shows that strategically sampling data points can reduce human workload, but the purpose is to improve machine learning models instead of assisting raters. Recent work on explainable active learning (XAL) (Ghai et al., 2020) has called for better designing for the human experience in the human-AI interface.

Our work shows that it is possible to achieve model improvements and assist human raters, bridging the gap between ML-assisted human labeling and active learning. The novelty of our approach is that the models are re-trained continuously on the output of the human annotation task, which they provide assistance for, constructing a positive feedback loop between humans and models. In this collaborative framework, we have the opportunity to improve both modeling and human rater performance.

Information overload, which we encounter in our content moderation setting, is a well studied problem that reduces the effectiveness of a human’s decision making ability (Eppler and Mengis, 2008; Chen et al., 2009). To address this, we build on the intuition that humans find it easier to verify or correct suggestions rather than produce new annotations from scratch (Hu et al., 2020; Bernstein et al., 2010). Our ML-assistance proposal strives to select the most informative but succinct ML-based ”hints” to surface to raters by drawing on the information retrieval and ranking literature. ML-based ranking has been shown to reduce information overload effectively in electronic messaging (Losee Jr, 1989) and social media (Koroleva and Bolufé Röhler, 2012; Chen et al., 2010). We draw inspiration from the learning to rank idea (Karatzoglou et al., 2013) to reduce information overload for raters.

The main contribution of this paper is the human-ML collaboration framework visualized in Figure 1. We use the predictions of ML models to provide assistance to human raters and evaluate the effectiveness of different user interfaces for the ML-assistance. Since the policy violation prediction task is hard for ML models, the feedback from human raters is useful to improve the models. ML-assistance enables raters to provide segment level annotations more efficiently leading to more ground truth to train/update the ML models. Additionally, we can enable raters to interact with the ML hints (accept/reject), providing direct feedback to refine the model, establishing a positive human-ML feedback loop.

Our proposed methodology is evaluated using raters from the author’s organization that regularly perform video content moderation reviews on live queues. Raters are separated into two pools: experts and generalists, with 150 and 400 raters respectively. The expert pool is a group of more experienced quality assurance (QA) raters with demonstrably better decision quality over a long period of time. Since their focus is QA, they don’t have fixed productivity targets as generalists do. They can hence spend more time reviewing each video comprehensively, leading to higher decision quality. In our experiment setup, each video in an evaluation dataset is independently reviewed by 1 expert and 2 generalist raters. We use the labels from expert raters on the dataset as the ground truth to evaluate the 2 sets of generalist rater decisions.

One of the potential risks of our proposed human-ML collaboration framework is automation bias (Skitka et al., 1999), where a rater’s over-reliance on ML-assistance can result in (i) blind-spots due to humans missing violations and (ii) raters accepting model hints without verification. Our video-level ratings quality evaluation metrics are robust to this since the ground truth comes from expert (QA) raters who review videos comprehensively, looking beyond ML-hints. In practice, we observe little evidence of both (i) and (ii). 56% of the violation segments submitted by raters in the V2 setup are organically created, i.e., don’t overlap with pre-populated hint segments. The segment acceptance rate is 35%, aligned with our segmentation model precision tuning point of 40%, indicating that raters are verifying and rejecting false positive hints at the expected rate. We could mitigate the risk of (ii) further by enforcing that at least some percentage of hint segments/video is actually watched or by surfacing the model’s confidence in the predicted hint to raters. To ensure robust evaluation of model quality, the AUC improvements in Section 4.2.4 are evaluated on a set of labels collected without model generated segments.

For content moderation to scale to the size of online platforms, it is necessary to take model-based enforcement action. We would like to explore the relation between improved ground truth and improvement of automated, model based enforcement. Leveraging active learning strategies in combination with utilizing rater feedback on model generated segments to show further quality improvements in the models is another open area of research. Finally, we will explore multi-armed bandits to balance active learning based exploration for model improvement with model exploitation for providing high quality ML-assistance (McInerney et al., 2018).

This paper used content moderation as the test bed for our human-ML collaboration proposal. However, it is a more generalized framework that applies to the problem of granular, localized video annotation encountered in various other industry applications such as identifying products/brands in videos to inform the placement of relevant ads, which we would like to explore further.