Distilling Aggregated Knowledge for Weakly-Supervised Video Anomaly Detection

Sultani et al. [27] proposed a deep Multiple-Instance Learning (MIL) framework, incorporating sparsity and temporal smoothness constraints and knowledge transfer for enhancing anomaly localization. Zhong et al. [41] used a graph convolutional network to mitigate label noise, but had higher computational costs. Feng et al. [6] introduced a two-stage approach to fine-tune a backbone network for domain-specific knowledge. Tian et al. [29] used top-k instances and a multi-scale temporal network for feature magnitude learning. Li et al. [13] employed a scale-aware approach for capturing anomalous patterns using patch spatial relations. Zaheer et al. [37] minimized anomaly scores in normal regions with a Normalcy Suppression mechanism and introduced a clustering distance-based loss to improve discrimination. Despite current approaches, limited training data and weakly-supervised constraints restrict model learning. Knowledge transfer plays a crucial role in anomaly detection performance. Building upon the deep MIL framework [27], we propose architectural changes to enhance performance on unseen data.

Knowledge distillation is a technique to transfer knowledge from a complex Teacher model to a simpler Student model. Hinton et al. [11] introduced the concept of aligning Teacher and Student model probabilities. FitNets[26] extended distillation to intermediate-level hints, focusing on matching intermediate representations. Zagoruyko and Komodakis [12] introduced attention-based distillation to transfer attention maps. Papernot et al. [22] emphasized matching intermediate representations for effective knowledge transfer. Zhang et al. [40] introduced self-distillation, leveraging the Student model as a Teacher to improve generalization. Heo et al. [10] improved knowledge transfer by distilling activation boundaries formed by hidden neurons.

To alleviate the limited size of the training set and the highly imbalanced distribution of classes in weakly-supervised VAD, we adopt intensive knowledge transfer by employing multiple pre-trained video backbones for feature extraction. Different feature backbones encode different types of information, which can aid in anomaly detection and help circumvent the challenges arising from limited supervision. Additionally, different backbones can help increase the diversity of the extracted features, which can lead to a more comprehensive aggregated representation of the input videos.

Consider that the video dataset consists of $n_{v}$ pairs $\{(V_{i},y_{i})\}_{i=1}^{n_{v}}$, where the $i^{th}$ video, $V_{i}$, is a sequence of clip instances $\mathbf{v}_{i,j}$ and $y_{i}\in\{0,1\}$ is the corresponding video-level label. Let $\psi_{1},\dots$, $\psi_{T}$ denote the set of pre-trained backbones for extracting representations from the input videos. For each input video clip $\mathbf{v}_{i,j}$, we extract features using the $t^{th}$ backbone as $\mathbf{z}_{i,j,t}=\psi_{t}(\mathbf{v}_{i,j})$, where $\mathbf{z}_{i,j,t}\in\mathbb{R}^{d_{t}}$ and $d_{t}$ denotes the cardinality of the output of the $t^{th}$ backbone. After extracting representations using multiple backbones, we aggregate them using a novel Temporal Aggregation Module described in the next section.

In weakly-supervised VAD, incorporating relative positional information is vital due to the low-pass temporal frequency characteristics of natural events, where anomalous frames cluster together rather than appearing sporadically. Leveraging this information enhances anomaly detection performance, which we achieve by utilizing a disentangled attention mechanism that inherently accounts for relative positional information during attention computation. This mechanism [9] employs a relative positional bias, with the maximum relative distance parameterized by $k$. The relative distance between positions $i$ and $j$ is encoded by the function $\gamma(i,j)$, constrained within $[0,2k)$, thereby reducing attention model complexity and making it suitable for low-data and weak-supervision scenarios. Formally, $\gamma(i,j)$ is defined as follows:

$$\gamma(i,j)=\begin{cases}0&\text{for $i-j\leq-k$ },\\ 2k-1&\text{for $i-j\geq k$},\\ i-j+k&\text{others}.\end{cases}$$

(1)

The disentangled attention has three attention components: i) content-to-content attention: This component attends to the content of a token at position i by interacting with the content of the token at position j within the same sequence, ii) content-to-position attention: This attention component considers the content of token i and its relative position to token j, capturing how the content of token i influences its attention weight concerning token j. iii) position-to-content attention: Similarly, this component assesses the content of token j and its relative position to token i, elucidating how the content at position j influences its attention weight with respect to token i.

We further disentangle this attention mechanism to adopt it for multi-input scenarios. To this aim, we add a cross-attention module to fuse information from multiple input sequences. Table 5 evaluates the contribution of each of these attention components. Given $T$ input sequences $Z_{t}$, $t\in\{1,\dots,T\}$, we define the query, key, and value for each of the representations as:

$$\begin{split}Q^{c_{t}}=Z_{t}W_{q,c_{t}},\quad K^{c_{t}}=Z_{t}W_{k,c_{t}},\quad V^{c_{t}}=Z_{t}W_{v,c_{t}}.\end{split}$$

(2)

The shared relative position key and query are also computed as :

$$Q^{r}=\tilde{Z}W_{q,r},\quad K^{r}=\tilde{Z}W_{k,r},$$

(3)

where $\tilde{Z}\in\mathbb{R}^{2k\times d}$ represents the relative position embedding vectors shared across all layers and backbones (i.e., staying fixed during forward propagation).

Our aggregation attention mechanism is then formulated as:

	$\displaystyle\tilde{A}_{i,j,t}=$	$\displaystyle\underbrace{Q_{i}^{c_{t}}{K_{j}^{c_{t}}}^{\top}}_{\text{(self content-to-content)}}+\underbrace{\sum_{h,h\neq t}Q_{i}^{c_{t}}{K_{j}^{c_{h}}}^{\top}}_{\text{(cross content-to-content)}}$		(4)
		$\displaystyle+\underbrace{Q_{i}^{c_{t}}{K^{r}_{\gamma(i,j)}}^{\top}}_{\text{(content-to-position)}}+\underbrace{K_{j}^{c_{t}}{Q^{r}_{\gamma(j,i)}}^{\top}}_{\text{(position-to-content)}},$

and the output is computed as:

$$H_{t}=softmax\big{(}\dfrac{\tilde{A}_{t}}{\sqrt{(T+2)d}}\big{)}V^{c_{t}},$$

(5)

where $T$ is the number of backbones, and the final aggregated output is $H=\tfrac{1}{T}\sum_{t}H_{t}$. $Q^{c_{t}}$, $K^{c_{t}}$, and $V^{c_{t}}$ are content vectors derived through projection matrices $W_{q,c_{t}}$, $W_{k,c_{t}}$, $W_{v,c_{t}}$ $\in\mathbb{R}^{d\times d}$ and $t\in\{1,\dots,T\}$ is the index of the backbone. $Q_{r}$ and $K_{r}$ correspond to the projected relative position vectors, facilitated by projection matrices $W_{q,r}$ and $W_{k,r}\in\mathbb{R}^{d\times d}$, respectively. The architecture of the Temporal Aggregation Module integrates the aforementioned disentangled attention mechanism to improve the relative encoding and fusing of representations from multiple backbones.

Our approach leverages the knowledge from multiple pre-trained backbones, allowing it to benefit from their collective expertise. We employ the MIL ranking approach, proposed by Sultani et al. [27] for training the first stage aggregated model. This intensive knowledge transfer aims to address the scarcity of supervision, which often hinders learning in the current weakly-supervised learning setup. However, using multiple backbones drastically increases computational overhead, thereby making the model less suitable for real-world applications. To overcome these issues, we develop a knowledge distillation approach, as shown in Figure 2, that distills the knowledge of the aggregated Teacher at prediction and representation levels into the Student model.

We refine the anomaly scores into $Y^{a}=\{\hat{y}_{i}^{a}\}_{i=1}^{n_{s}}$ and use these as soft pseudo-labels. Since we are certain about the segment-level annotation in the normal videos, we can combine the soft anomaly labels with normal videos. Given the nature of the VAD task, which typically involves predictions for two classes, the conventional posterior matching approach encounters limitations due to the limited support of the distribution. To mitigate this issue, we extend the distillation process to operate at the feature level, employing the InfoNCE loss [21, 28].

Our feature distillation loss using InfoNCE is defined as:

$$\begin{split}\mathcal{L}_{nce}=-\log\dfrac{e^{\text{sim}(\mathbf{z}_{a_{i}}^{T},\mathbf{z}_{a_{i}}^{S})/\tau}}{\sum_{k=1}^{N}e^{\text{sim}(\mathbf{z}_{a_{i}}^{T},\mathbf{z}_{n_{k}}^{S})/\tau}+e^{\text{sim}(\mathbf{z}_{a_{i}}^{T},\mathbf{z}_{a_{i}}^{S})/\tau}}\\ -\log\dfrac{e^{\text{sim}(\mathbf{z}_{n_{i}}^{T},\mathbf{z}_{n_{i}}^{S})/\tau}}{\sum_{k=1}^{N}e^{\text{sim}(\mathbf{z}_{n_{i}}^{T},\mathbf{z}_{a_{k}}^{S})/\tau}+e^{\text{sim}(\mathbf{z}_{n_{i}}^{T},\mathbf{z}_{n_{i}}^{S})/\tau}},\end{split}$$

(7)

where $\tau$ represents the temperature parameter. The complete loss function for the distillation is as follows:

$$\mathcal{L}_{d}=\mathcal{L}_{bce}(y^{T},y^{S})+\alpha\tau^{2}\mathcal{L}_{nce}(T,S),$$

(8)

where $\mathcal{L}_{nce}$ represents the feature-level distillation loss using InfoNCE, $\mathcal{L}_{bce}$ represents the BCE loss, $\mathcal{L}_{d}$ represents the combined loss for distillation, and $\alpha$ represents the scaling coefficient to control the contribution of the two loss terms.

Our model is evaluated on three benchmark datasets for weakly-supervised VAD: UCF-Crime, ShanghaiTech, and XD-Violence. The UCF-Crime dataset [27] contains 1900 untrimmed videos totaling 128 hours, captured by surveillance cameras in diverse real-world settings, with 13 types of anomalies. The ShanghaiTech dataset [16] comprises 437 videos from fixed-angle street cameras, featuring 13 background scenes. We follow Zhong et al.’s [41] approach to adapt it for weakly-supervised learning. The XD-Violence dataset [34] is a comprehensive multiscene collection sourced from various media, containing 4754 untrimmed videos spanning over 217 hours. All three datasets provide video-level labels for training and frame-level labels for testing, allowing for robust evaluation of weakly-supervised VAD models across diverse scenarios and anomaly types.

Our proposed method is implemented using PyTorch [24]. We divide each video into 32 non-overlapping segments to pass through the feature extractors.

In our Temporal Aggregation Module, we utilize disentangled attention along with our proposed cross-attention mechanism to combine multiple inputs, as explained in Section 3.2. The disentangled attention module has one hidden layer with a hidden embedding dimension of 1024 and 8 attention heads. Notably, the TAM shares positional embeddings among all backbones. We experimented with values of maximum relative distance $k$ from 1 to 32, where 32 is the maximum segment index, to determine the optimal value that constrains the maximum distance between two positions $(i,j)$ in disentangled attention. The aggregated features are then passed through a feedforward network with hidden dimensions of 512 and 32, and with a sigmoid activation in the final layer to obtain the segment-level prediction. The conventional MIL Loss [27] serves as the loss function during training.

The Teacher and Student models are trained for 100 epochs, using the Adagrad optimizer [5] with a weight decay of 0.001 and a learning rate of 0.0001 for the temporal models and 0.001 otherwise. The training batch size is 60, and each batch consists of 30 normal and 30 anomalous video clips.

Table 2 presents our main results on the UCF-Crime dataset, while Table 3 shows the results for the ShanghaiTech and XD-Violence datasets. DAKD_T and DAKD_S outperform all the existing weakly-supervised methods by a significant margin on all the datasets. Remarkably, on the UCF-Crime dataset, DAKD_S outperforms current SOTA methods, MGFN [4] by 1.36%, SSRL [13] by 1.55%, S3R [32] by 2.35%, and RTFM [29] by 4.04%. DAKD_S is also extremely efficient compared to SSRL, which uses multi-scale video crops for training, making it suitable for real-world use. Additionally, DAKD also achieves an AUC score of 98.10% on the ShanghaiTech dataset, providing superior performance although the performance of previous methods seems to be saturated on this dataset. Moreover, DAKD achieves an AP score of 85.61% on the XD-Violence dataset and outperforms existing methods by a significant margin.

We further analyze the performance of the models on each anomaly class in UCF-Crime to highlight the effectiveness of DAKD. Figure 6 presents the class-wise AUC scores for the Anomaly classes in UCF-Crime of our method compared with that of Sultani et al. [27] and RTFM [29]. Our approach outperforms existing methods by a significant margin in classes such as Assault, Arrest, Burglary, Explosion, and Vandalism.

From Figure 1, it is clear that while individual backbones struggle at corresponding to the ground truth frame-level annotations, our approach of using aggregated features is able to correctly localize the anomaly. The combined features leverage the power of individual backbones and show effective performance where dataset size is limited. Our approach also shows a comparatively smoother transition between normal and anomalous regions, demonstrating that it is able to consider the temporal localization of an event.