Unsupervised Video Anomaly Detection
via Normalizing Flows with Implicit Latent Features

As a basic network for video data, 3D convolution-based networks have been proposed and used for feature extraction in video anomaly detection. Recently, a transformer-based model for video representation learning has been studied [3, 2, 11, 30]. TimeSformer [3] and Video Vision Transformer (ViViT) [2] divide space and time information to perform self-attention using a sequence of spatio-temporal tokens; Multiscale Vision Transformers (MViT) [11] contains several channel-resolution scale stages to create a multiscale pyramid of feature activation; Swin-T [30] extends the Swin Transformer to the spatio-temporal domain for spatio-temporal locality of videos. These methods usually utilize a Resnet backbone because of their low throughput. For video recognition, two-stream networks have also been proposed to model motion features explicitly. However, these require the explicit extraction of temporal information, such as temporal differences or optical flows. A Siamese network structure with contrastive learning [18] that maximizes the similarity of positive pairs while minimizing those of the negative ones has been studied as an unsupervised learning paradigm for representation learning. Furthermore, Multi-view Contrastive Learning with Noise-robust loss (MvCLN) [55] has been proposed to learn consistent representations from multi-view/modal data. Recently, for video recognition, Feichtenhofer et al. [12] proposed slow and fast pathway networks that operate at different temporal and spatial resolutions to extract stationary and motion information. The slow-pathway is composed of a narrow temporal window, while the fast-pathway uses a higher temporal rate. In this paper, to learn normal appearance and motion patterns through reconstructing frames, we propose ITAE, which has two encoders and a single decoder composed of shallow layers, excluding residual blocks.

Many anomaly detection algorithms based on frame reconstruction that harness the powerful representation ability of deep convolutional networks have been proposed. These algorithms exploit the structure of convolutional AEs [20], recurrent neural networks [34], or 3D convolutions [56]. Other algorithms have been proposed by learning reconstruction with other objectives [39], usingmemory modules [15, 42], reconstructing optical flows from frames [48], or encoding normal patterns with sparse dictionary learning [57, 35]. Zhou et al. [57] suggested a sparse long short-term memory (SLSTM) unit with adaptive ISTA $l_{1}$-solvers to encapsulate the historical information and achieve sparse codes in an unsupervised manner for anomaly detection; Luo et al. [35] proposed a Temporally-coherent Sparse Coding (TSC) framework, which has a special type of sRNN that preserves the similarities between neighboring frames. Frame prediction-based approaches have been proposed to increase the probability of unpredictability for abnormal samples [29, 38, 41]. However, prediction-based approaches generally require heavier structures or optical flows, and bi-directional models rely heavily on future frames.

As another approach, anomaly detection methods that learn the compactness of normal clusters has been proposed. Clusters are configured by removing small clusters of normal samples, features from the reconstruction objective [54], or extracting from a pre-trained object detector [22] or pose estimator [36].

Compared with discriminative models, generative models do not require either normal annotations or proxy tasks such as frame generation. Some approaches based on a Gaussian model or non-parametric density estimation have been proposed based on either latent features or extracted features. Deep generative models can be categorized into implicit and explicit density estimation models [16]. However, implicit density models, such as GAN-based anomaly detectors [49], do not define the data likelihood and cannot use in-distribution estimation without modifying the discriminator for the likelihood estimator, whereas explicit models first define likelihoods and try to maximize them. An approximation-based model (e.g., VAE) has been proposed to calculate likelihood, which requires approximation through a lower bound of the likelihood (ELBO), and has been used for anomaly detection. Utilizing Expectation Maximization (EM) to estimate density for anomaly detection has also been proposed [40], which clusters features and applies E-step to retrieve the posterior likelihoods and M-step to update parameters iteratively. Auto-regressive models have shown promising results in estimating the density more precisely and have been adapted for anomaly detection [1]. However, these autoregressive models are not efficient for high-dimensional data, cannot be implemented with parallel processing, and are sensitive to the choice of sequence order.

Tractable density estimators using NFs have been proposed to alleviate these issues [8, 23]. Dinh et al. [8] proposed invertible networks using an affine coupling layer and calculated the tractable likelihood using the change of variable theorem. Kingma and Dhariwal proposed Glow [23] for further improvements using activation normalization and invertible $1\times 1$ convolution. In this paper, we suggest distribution learning with NF models using static and dynamic features obtained from ITAE, which helps to deal with the complex distribution of normal scenes. Although there have been studies using likelihood for anomaly detection, the proposed method differs from them in the following aspects: (1) we compute exact tractable likelihood through normalizing flow, and (2) we perform density estimation of features learned with two path encoders and show effectiveness through experiment analysis, especially on the complex and multi-scene ShanghaiTech dataset, without using any pre-trained external models. To the best of our knowledge, this is the first attempt at learning appearance and motion normality through NFs in video anomaly detection, and it demonstrates effectiveness.

In surveillance anomaly detection, normal and anomalous scenes have differences in appearance (e.g., a car driving down a sidewalk), motion (e.g., jumping), or both (e.g., chasing a person with an abnormal object). Therefore, to learn the appearance and motion normal patterns of both scene types, we propose an ITAE that implicitly focuses on static and dynamic features. With these learned features from ITAE, we suggest distribution modeling through NF models to learn complex and diverse normality.

The framework is trained in two steps using only normal training videos. In the first step, ITAE learns to reconstruct the normal frames. As depicted in Fig. 2, the sequence of frames is embedded through the ITAE with the static encoder ($E_{static}$) and dynamic encoder ($E_{dynamic}$). The embedding features of the two encoders are combined and reconstructed into original frames through a single decoder. In the second step, to estimate the density of the normal appearance and motion pattern, NF models ($F_{static}$ and $F_{dynamic}$) learn the distribution modeling of the two embedding features from the ITAE. In this step, ITAE is frozen to prevent the NF model from becoming difficult to be optimized when the weight of ITAE is updated and the feature distribution is changed.

During testing, the abnormality score is calculated using the reconstruction error of the ITAE and the estimated likelihood of the NF models. When an abnormal scene is input, the ITAE learned using normal frames outputs a large error because of its poor reconstruction. Furthermore, with NF models, the static and dynamic embedding features obtained from ITAE differ from the features of the normal training set, which entails a low likelihood value.

As the AE gets deeper and the parameter increases, the capacity becomes too powerful to reconstruct the anomalies accurately, so we compose each path into four shallow layers. In Table 1, each encoder composed with different spatial and temporal sizes of kernels and output features is presented. Furthermore, instead of a framework consisting of two encoder-decoder networks, ITAE fuses the features of two encoders into one latent feature, which the decoder uses to generate the original frame. These two encoders generate a higher reconstruction error than a one-path encoder for scenes with abnormal motion or appearance and perform better anomaly detection (visualized error maps are depicted in Fig. 4 in Section 5.1).

We can learn normality from unlabeled normal training data by the unsupervised density estimation method. By using explicit likelihood generative models, it is possible to compute the likelihood of input data. An NF-based generative model can calculate the tractable likelihood via changes of variables toward a simple distribution (e.g., multivariate Gaussian). The likelihood is calculated by passing an invertible parametric function composed of multiple layers that maps the complex data distribution to the simple distribution, which can be any distribution. In this study, we set the prior $\boldsymbol{p_{z}}$ to be an isotropic unit norm Gaussian. With an input variable $\boldsymbol{x}\in X$, the distribution of which is unknown, a simple distribution $\boldsymbol{z}\sim p_{z}$, and a parametric function $\boldsymbol{f}_{\boldsymbol{\theta}}:X\rightarrow Z$, the integral of the probability density function is

where $\textrm{det}\left(\frac{\partial\boldsymbol{f}_{\boldsymbol{\theta}}}{\partial\boldsymbol{x}}\right)$ is the Jacobian determinant of function $\boldsymbol{f}_{\boldsymbol{\theta}}$ under change of variable theorem. When the generative model $\boldsymbol{f}$ with parameter $\boldsymbol{\theta}$ is $\boldsymbol{f}_{\boldsymbol{\theta}}=\boldsymbol{f}_{\boldsymbol{\theta_{M}}}\circ\boldsymbol{f}_{\boldsymbol{\theta_{M-1}}}\circ\cdots\circ\boldsymbol{f}_{\boldsymbol{\theta_{1}}}$ and $\boldsymbol{h}_{i}=\boldsymbol{f}_{\theta_{i}}(\boldsymbol{h}_{i-1})$ with $\boldsymbol{h}_{i-1}\sim p_{i-1}(\boldsymbol{h}_{i-1})$ where $\boldsymbol{h}_{0}=\boldsymbol{x}$ and $\boldsymbol{h}_{M}=\boldsymbol{z}$, the probability density function $p_{i}$ is as follows (for brevity, we omit $\boldsymbol{\theta}$ from $\boldsymbol{f_{\boldsymbol{\theta}}}$):

Equation 2 is written as Eq. 3 according to the inverse function theorem: if $y=f(x)$ and $x=f^{-1}(y)$ then $\frac{\textsl{d}f^{-1}(y)}{\textsl{d}y}=\frac{\textsl{d}x}{\textsl{d}y}=(\frac{\textsl{d}y}{\textsl{d}x})^{-1}=(\frac{\textsl{d}f(x)}{\textsl{d}x})^{-1}$. Equation 4 is from Jacobians of invertible function of $\textrm{det}(M^{-1})=(\textrm{det}(M))^{-1}$. Then,

By repeatedly applying the rule for change of variables, the expanded equations are as follows:

With these steps, the probability density function of $\boldsymbol{x}$ is as given in Eq. 7.

If the density function of the latent variable $\boldsymbol{z}$ is tractable like a Gaussian distribution and the Jacobian matrix $\frac{\partial\boldsymbol{f}_{\boldsymbol{\theta}}}{\partial\boldsymbol{x}}$ is triangular, the likelihood of input variable $\boldsymbol{x}$ can be obtained simply. We use the Glow model with $L$ level multi-scale architecture and $K$ series of Actnorm, invertible convolution, and Affine layer for density estimation [23].

By maximizing the likelihood in Eq. 7, an NF model estimates the density of high-dimensional data through multiple layers of convolutional networks. As the likelihood of a generative model heavily depends on the image complexity [44], unlike Glow, which begins from the image space, the intermediate latent feature of ITAE is used for complex density modeling of normal videos. After training the ITAE with frame reconstruction, we estimate the density of the static and dynamic embedding features obtained from each encoder. The max-pooling and average-pooling along the channel axis of the feature from each path are applied to reduce the dimensions and are concatenated and input into each NF model, i.e., $F_{static}$ and $F_{dynamic}$ (Fig. 2). We also concatenate the resized intensity of the input frame to the $F_{static}$ input, which provides additional sparse appearance information of the feature map. For abnormal scenes, static and dynamic NF models trained with latent features that embed each appearance and motion pattern of normal scenes show low likelihood results.

The total anomaly score $S_{t}$ is computed by summing the reconstruction error and nll with scaling factor $\lambda_{L}$ in Eq. 14. $\lambda_{L}$ is experimentally obtained value from the set [0.1, 0.3, 0.5, 0.7, 0.9, 1.0] (in Section 4.3.4).

We consider three databases in our experiments: UCSD, CUHK, and ShanghaiTech (ST). For training, we resize the input frame to $256\times 256$ and set $T$ to 16. For UCSD, which has a small foreground scale, we use the original frame size ($240\times 360$). We use the Adam optimizer and Cosine annealing scheduler. In the first training step, the batch sizes are 2, 2, and 8, and the learning rates are $1e^{-3},1e^{-2}$, and $1e^{-2}$ for the UCSD, CUHK, and ST databases, respectively. For the second step, the batch sizes are 8, 5, and 8, and the learning rates are $5e^{-4},5e^{-4}$, and $1e^{-4}$. $\lambda_{L}$ is 0.3, 0.1, and 0.7, respectively. Glow is used for the NF models with $K=32$ and $L=3$ ($L=1$ for UCSD). For the ST dataset, which contains multi-scene videos, each anomaly score is normalized to match the scale of the score. For real-world scenario datasets, UCF-Crimes, LV, and UBI-Fights, all settings are the same as ST. Please refer to the supplementary material for detailed information on experiments.

A comparison with other state-of-the-art methods on three benchmarks is presented in Table 5. The proposed approach achieves superior or competitive performance with the state-of-the-art methods on three datasets, and ITAE significantly improves performance over the well-designed AE (AE-Conv3D [56], TSC [34], StackRNN [34], and DissociateAE [5]). In particular, ITAE and ITAE+NFs both show superior performance in UCSD Ped2 with 98.7% and 99.2% AUC, respectively, without any external models or datasets such as optical flow model (e.g., SelFlow, OpticalFlow, Flownet, Flownet2 pre-trained on FlyingThing3D, and ChairsSDHom datasets); object detector (e.g., FRCN, Yolov3 pre-trained on MS-COCO dataset); and feature extractor (e.g., Resnet-50 pre-trained on the ImagNet dataset). In addition, compared to the latest methods using an object detector [14, 22, 13], some results show slightly higher performance than the proposed method, but the proposed method’s performance is still 0.5%, 4.9%, and 1.7% higher on Ped2. Furthermore, these methods have a crucial issue: detecting anomalies is impossible for the object classes that are not pre-trained on external datasets as noted in BG-Agnostic [14]. Utilizing an object detector for knowledge distillation by considering unseen object classes (e.g., bike, car) during training as anomalies may be difficult to generalize in real-world scenarios.

Without an external model, the memory-based approach [15, 42] that stores and updates normal query features by memory module shows good performance, but this approach demonstrates low performance on the ST database with complex and multiple scenes, which indicates that storing fixed numbers of memory items may not be suitable for the general problem. In contrast, ITAE demonstrates better performance, i.e., 74.8%, on the ST database, and the performance with NFs is 76.3%, which shows the highest improvement in multi-scene anomaly detection among the results on the three benchmarks, which is very important from the perspective of generality. For all three databases, the results are superior to those of Auto-reg [1], which performs auto-regressive generative modeling that requires a causal network and data ordering to perform sequential operations. It is noteworthy that the ITAE+NFs model outperforms on three databases without using any external dataset and model, and NFs also show effectiveness on generalization through high performance improvement on multi-scene anomaly detection.

Figure 4 illustrates a comparison of AE (consisting of only a static encoder) and ITAE. As in a previous study [42], the error maps are visualized by marking the pixel that is larger than the average error value within the frame. The first row of the figure is a jumping child whose appearance is normal, which leads to low reconstruction error with one-path AE. In contrast, the ITAE, which focuses on motion as well as appearance, generates a substantial error owing to the poor reconstruction for inputs that differ from the normal learned motion. For the second row, a person throwing a paper, the abnormality of the paper’s appearance, and flying motion produce a large error in ITAE.

Figure 5 is a histogram of likelihood within a video clip from static and dynamic NF models. The first row (a) is a car and biker on the walkway, and the second row (b) is a person riding a bike on the sidewalk. As depicted in the histogram, the likelihoods of both NF models are low in abnormal frames, with abnormal appearance and motion on the sidewalk.

With the two-path encoder and likelihoods, we compute the anomaly score of each frame in Fig. 6. The two scores complement each other and achieve satisfactory results, even when the pedestrian density is high or low and the foreground scale is small or large. (Please refer to the supplementary material for various qualitative results.)

When considering real-world scenarios, generality is a crucial issue in surveillance anomaly detection. In order to detect undefined abnormal events, it is necessary to learn a general normal pattern through normal scenes of training data, and to prevent the occurrence of many false alarms by focusing on specific normality due to overfitting the data. As the unsupervised anomaly detection approach trains with unlabeled data, we investigate the generalization ability by testing the model that trained the normal pattern on another dataset. As the Ped2 dataset is in grayscale , and the CUHK and ST datasets are composed of RGB scale frames, the color domain of the source and target datasets might differ. Instead of training a model with the same color domain as the target dataset, we use the model trained on the source dataset as it is. Therefore, when CUHK $\rightarrow$ Ped2 or ST $\rightarrow$ Ped2, given that the trained model on CUHK or ST takes 3-channel input, we duplicate the grayscale value to 3-channel for evaluation following Fewshot [33].

Table 6 presents the result of testing the model, which was trained on the source dataset, on the target dataset. ITAE and ITAE+NFs both show a slight decrease in performance, but this decrease is much smaller than the latest methods, i.e., BG-Augnostic [14] and Fewshot [33]. When the target dataset is Ped2 and the source dataset is CUHK or ST, the performance degradation of the proposed method is 4.9% and 2.4%, respectively, that of BG-Augnostic is 11.7% and 8.1%, respectively, and that of Fewshot is 14.2% for CUHK. ITAE and NFs also show the smallest decrease in other target datasets, and the highest performance in Ped2 and CUHK datasets. The degradation is the most in CUHK $\rightarrow$ Ped2, because Ped2 is grayscale unlike CUHK and the characteristics of the dataset such as camera angle and object size differ the most. For this reason, the distribution modeling of NFs is difficult owing to the large difference between the target and source datasets and the performance of ITAE+NFs is lower than that of ITAE when Ped2 is the target dataset. ITAE+NFs, which focuses on static and dynamic features and performs normality learning, shows high performance in cross-domain testing and effectiveness in generalization ability.

We compose subsets with 1, 6, and 13 scenes in the ST dataset and compare the performance as the scenes of the data become diverse and extensive. The sizes of the test sets are proportional, and the train-to-test set ratio in each subset is similar. In Fig. 7, through various scenes and a large number of videos, the NF model learns the general normal pattern and shows higher performance on a multiple scene subset than on a single scene subset. In contrast, ITAE illustrates higher performance as there are fewer scenes because of overfitting, but exhibits limitations as scenes become more diverse. NFs complement the drawback of AE and produce an excellent performance improvement (1.5%) in the most diverse 13-scene subset. From Fig. 7 and Table 3, it can be observed that the distribution learning of the NF models on ITAE features is effective in coping with the complex normality distribution.

We also consider more realistic and dynamic databases: UCF-Crimes, LV, and UBI-Fights. As our method is an unsupervised-learning-based approach, the framework is trained using only normal videos from the training set of each database. In Table 7 (a) and (b), the performance NFs is higher than that of ITAE in UCF-Crimes as well as LV datasets. In Section 5.3, the experimental results show that as the number of scenes increases, the performance of the autoencoder deteriorates, while the performance of NF model remains similar by learning the distribution of general normal patterns. In the case of UBI-Fights, in Table7 (c), ITAE and NFs achieve 57.7% and 53.3% AUC score, respectively. ITAE shows better performance than NFs, which seems to indicate that detecting anomalies with reconstruction error is more effective than distinguishing abnormality by distribution of latent features when the abnormal event is fighting; this is because a fighting event involves a large change in appearance and motion at the frame level, which results in a large reconstruction error that leads to anomaly detection. In Table 8(c), the performance of ITAE with NFs on UCF-Crimes, LV, and UBI-Fights dataset is 70.9%, 77.5%, and 57.8% respectively, and achieves the best results compared with unsupervised anomaly detection methods.

In Fig. 8, the first row is an example of the UCF-Crimes dataset ‘Robbery048’ clip, which is a scene of one person assaulting another; the second row is the LV dataset ‘murder’ clip, which is a scene showing a person being shot. Qualitative results show that anomalies are well distinguished even in clips of complex and diverse anomalies in real-world scenarios.

Figure 9 shows false positive (FP) and false negative (FN) samples. In (a), unlike ITAE which considers appearance and motion factor concurrently, the one-path AE shows a high reconstruction error in the pedestrian carrying the red bag. (b) is an FP sample of ITAE, a scene in which a pedestrian carries a long stick, showing a high error on the strange stick that causes a high anomaly score, but the false alarm is prevented by adding the score of the NF model. (c) is an FN sample of the proposed method, a scene where a pedestrian is loitering. In this scene, the appearance of a pedestrian and the motion of walking are both similar to the normal pattern, so the ITAE and NFs are unable to distinguish between normal and abnormal scenes.

We measure the computational time of the proposed method on the UCSD Ped2 dataset (resolution: $240\times 360$) following [13]. ITAE infers the anomaly score of a frame in 9.1 milliseconds (ms), while static and dynamic NF models take 22.2 ms and 20.7 ms, respectively. The total framework takes 52 ms per frame, while calculating the anomaly score takes 0.5 ms. With all components together, the proposed method runs at 19 FPS on an NVIDIA GeForce RTX 3090, 24GB memory. The parameter number of ITAE, static NFs, and dynamic NFs is 7.74, 10.16, and 11.05M, respectively; compared to one-path AE of 5.76M, ITAE with motion path only increases 1.98M parameters as the dynamic encoder focuses on motion with a high frame-rate and low channel size (1/8 of static).