Bayesian Nonparametric Submodular Video Partition for Robust
Anomaly Detection

Let ${\bf x}_{i}^{+}$ be the $i^{th}$ segment in the positive bag $\mathcal{B}_{pos}$ and ${\bf x}_{j}^{-}$ indicates the $j^{th}$ segment in the negative bag $\mathcal{B}_{neg}$. Also consider $n$ as the total number of instances per bag. The maximum score based MIL (MMIL) model tries to maximize the gap between the maximum prediction score from positive bag and that from the negative bag [19]:

$$L(\mathcal{B}_{pos},\mathcal{B}_{neg})=\Bigl{[}1-\max_{i\in\mathcal{B}_{pos}}(f({\bf x}_{i}^{+}))+\max_{j\in\mathcal{B}_{neg}}(f({\bf x}_{j}^{-}))\Bigr{]}_{+}$$

(1)

where $f({\bf x}_{i}^{+})$ (or $f({\bf x}_{j}^{-})$) is the prediction score of ${\bf x}_{i}^{+}$ (or ${\bf x}_{j}^{-}$) and $[a]_{+}=\max\{0,a\}$. As mentioned earlier, MMIL is less effective to handle outlier and multimodal scenarios. The top-$k$ ranking loss partially addresses the limitation of MMIL by maximizing the gap between an average of $k$ highest segment predictions from the positive bag and maximum segment prediction score from a negative bag:

$\displaystyle L(\mathcal{B}_{pos},\mathcal{B}_{neg})=\Bigl{[}1-\frac{1}{k}\sum_{i=1}^{k}f({\bf x}_{[i]}^{+})+\max_{j\in\mathcal{B}_{neg}}f({\bf x}_{j}^{-})\Bigl{]}_{+}$

(2)

where the positive bag segment predictions are sorted in a non-decreasing order, i.e., $f({\bf x}_{[1]}^{+})\geq...\geq f({\bf x}_{[k]}^{+})$.

There are two major issues associated with the top-$k$ ranking loss. First, choosing an optimal $k$ value is a key challenge as the number of abnormal instances may vary significantly from one video to another implying different $k$ value for each video. Second, all the selected top-$k$ instances may come from same sub-sequence of the video. Including all those visually similar instances does not contribute much in the model training process. Furthermore, concentrating only on a specific sub-sequence may make the approach less effective to handle multimodal and outlier scenario. Figure 3 presents the output of the top-$k$ based model in the Avenue dataset. It can be seen that the top-$k$ based approach picks the consecutive video segments while missing quite a few other abnormal frames.

The proposed Bayesian non-parametric submodular video partition (BN-SVP) approach offers a novel integrated solution to address the above two fundamental challenges simultaneously. In particular, since submodular set functions provide a natural measure for diversity, we design a special submodular set function that enables discovery of a representative set of segments from a video. This avoids only choosing visually similar consecutive video segments like in the top-$k$ approach, which enhances the model’s exposure to potential abnormal instances during model training. As a result, the model’s capability to handle multimodal and outlier scenarios can be effectively improved.

However, maximizing a submodular set function still requires to specify the size of the set. As mentioned above, choosing a set with an optimal size in video anomaly detection is highly challenging. To this end, we propose a novel Bayesian non-parametric construction of the submodular set function. The non-parametric construction leverages both visual features of the video segments and their temporal closeness to derive a similarity measure that allows us to define a submodular set function $F(\mathcal{C}^{+})$, where $\mathcal{C}^{+}$ represents a subset of segments in a video. The size of $\mathcal{C}^{+}$ is automatically determined through Bayesian non-parametric analysis of the video. Intuitively, most videos, especially those with anomalies, usually consist of multiple scenes, where each scene is comprised of a consecutive set of visually similar segments. Figure 4 shows the example frames from three different scenes in a video that records an explosion event. Ideally, if a video could be partitioned based on these scenes, we can choose representative (and potentially positive) segments from each scene. Such information can significantly facilitate the optimization of the submodular set function. However, both the number and the types of the scenes are unavailable during model training.

The proposed BN-SVP addresses the above issue through non-parametric partition of a video. It builds upon and extends an HDP-HMM model that places a Hierarchical Dirichlet Process (HDP) prior on the state transition distribution of a Hidden Markov Model (HMM) model [20]. By using an HMM to model a video (as a sequence of segments), each discovered hidden state can be naturally interpreted as a scene in the video. The HDP prior allows us to determine the optimal number of states (i.e., scenes) according to the nature of the data. However, real-world videos may be highly noisy and directly using an HDP-HMM model may extract too many scenes with less significant visual characteristics (e.g., spatial changes of objects or addition/removal of a small number of objects). To address this issue, we follow the sticky HDP-HMM to encourage a stronger self-transition of a state [4]. This will result in temporal persistence of states to produce longer and semantically coherent scenes. To further accommodate spatial changes or variations in certain objects, we allow the emission distribution to follow another non-parametric DP that automatically determines the number of mixture components (i.e., sub-scenes) within the same scene. For example, scene 1 in Figure 4 is comprised of two sub-scenes: the first with a clear sky and the second with smoke in the sky.

More specifically, consider a collection of hidden states (i.e. scenes in a video), the transition probability of state $j$ to other states is governed by a DP:

$$\mathcal{G}_{j}=\sum_{l=1}^{\infty}\hat{\pi}_{jl}\delta_{\hat{\phi}_{jl}},\;\hat{\boldsymbol{\pi}}_{j}\thicksim\text{GEM}(\alpha)$$

(3)

where $\text{GEM}(\alpha)$ is formed through a stick breaking construction process with parameter $\alpha$ [20], $\hat{\phi}_{jl}$ is drawn from a base distribution $\mathcal{G}_{0}$, which follows another DP

$$\mathcal{G}_{0}=\sum_{k=1}^{\infty}\beta_{k}\delta_{\phi_{k}},\;\boldsymbol{\beta}\thicksim\text{GEM}(\gamma),\;\phi_{k}\thicksim H$$

(4)

Because of the discrete nature of $\mathcal{G}_{0}$, there can be multiple $\hat{\phi}_{jl}$’s taking an identical value of $\phi_{k}$. Considering the unique set of atoms $\phi_{k}$, we can rewrite $\mathcal{G}_{j}$ as

$$\mathcal{G}_{j}=\sum_{k=1}^{\infty}\pi_{jk}\delta_{\phi_{k}},\;\boldsymbol{\pi}_{j}\thicksim\text{DP}(\alpha,\boldsymbol{\beta}),\phi_{k}\thicksim H$$

(5)

Given the highly dynamic and noisy nature of many real-world surveillance videos, directly applying the standard HDP-HMM model to partition a video may result in many redundant scenes and rapidly switches among them. This is problematic in our setting in which it is critical to infer semantically coherent scenes along with a slower transition among them. As a result, it is essential to ensure temporal persistence of the discovered scenes [4]. This can be achieved through enhanced self transitions. In particular, the transition probability of the $j$’s state is augmented by

$$\boldsymbol{\pi}_{j}\thicksim\text{DP}\left(\alpha+\rho,\frac{\alpha\boldsymbol{\beta}+\rho\delta_{j}}{\alpha+\rho}\right)$$

(6)

This has the effect of increasing the expected probability of staying in the same state.

$$\mathbb{E}[\pi_{jk}|\boldsymbol{\beta}]=\left\{\begin{array}[]{@{}ll@{}}\frac{\alpha\beta_{j}+\rho}{\alpha+\rho}&\text{if}\ k=j\\ \frac{\alpha\beta_{k}}{\alpha+\rho},&\text{otherwise}\end{array}\right.$$

(7)

To allow certain levels of variations within the same scene and accommodate the highly dynamic nature of a video sequence, we propose to model the emission process using a mixture distribution governed by another non-parametric DP. This design offers three unique advantages. First, it further ensures the temporal persistence of a scene as for a segment with less significant visual differences, it can stay in the same scene by switching to a different mixture component instead of transitioning to another (redundant) scene. Second, it offers a fine-grained partition of the video sequence, which is instrumental to separate abnormal segments (e.g., frames (b) and (e) in Figure 4) from normal ones (e.g., frames (a) and (d) in Figure 4) that share a common background. Last, the number of mixture components is automatically determined by the DP (e.g., scenes 1 & 3 have two mixture components while scene 2 only has one). For the $k$-th scene, there is an unique stick-breaking distribution $\boldsymbol{\psi}_{k}\thicksim\text{GEM}(\tau)$ that defines the weights of the mixture components within the scene. Then, given the scene and mixture component assignment $(z_{i}=k,s_{i}=t)$ of a segment ${\bf x}^{+}_{i}$ in a video, it is drawn from a specific multivariate Gaussian: $\mathcal{N}(\boldsymbol{\mu}_{k,t},\Sigma_{k,t})$.

Posterior inference of the augmented HDP-HMM model with a DP mixture for emission can be achieved through direct assignment [20] or blocked sampling with an increasing mixing rate [3]. Hyper-parameters can also be inferred by placing a vague prior on them and conduct Gibbs sampling.

The scene and component assignments of BN-SVP induces a pairwise similarity among segments in a video:

$$\left\{\begin{array}[]{@{}ll@{}}S_{i,j}=({\bf x}_{i}^{+})^{\top}\Sigma^{-1}_{z_{i},s_{i}}{\bf x}_{j}^{+}&\text{if}\ s_{i}==s_{j}\wedge z_{i}==z_{j}\\ S_{i,j}=0&\text{otherwise}\end{array}\right.$$

(8)

It is worth to note that the similarity between two segments ${\bf x}_{i}^{+}$ and ${\bf x}_{j}^{+}$ is evaluated using the learned feature representations (through a DNN) instead of the raw features. The induced similarity allows us to define a submodular set function [11, 10] summarized by the follow proposition.

Based on the definition of $S_{i,j}$ as shown above, it is straightforward to show that $F({\mathcal{C}})$ is a special instance of the location facility function [14], which is submodular. Since each mixture component captures a unique sub-scene, maximization of $F({\mathcal{C}})$ can extract a diverse set of segments that best represent the all the scenes (and sub-scenes) in the entire video. By further integrating the margin loss given in (2), we achieve a submodularity diversified MIL loss:

$\displaystyle\min_{{\bf w},\mathcal{C}^{+}\in\mathcal{B}_{pos},|\mathcal{C}^{+}|\leq{\kappa}}L(\mathcal{C}^{+})-\lambda F(\mathcal{C}^{+})$

(10)

where the margin loss is defined over instances in a set $\mathcal{C}^{+}$ with size no larger than $\kappa$:

$$L(\mathcal{C}^{+})=\Bigl{[}1-\frac{1}{|\mathcal{C}^{+}|}\sum_{i\in\mathcal{C}^{+}}f({\bf x}^{+}_{i})+\max_{j\in\mathcal{B}_{neg}}f({\bf x}_{j}^{-})\Bigl{]}_{+}$$

(11)

In essence, $\mathcal{C}^{+}$ includes the set of instances in a positive bag that participate in the model training. The constraint $|\mathcal{C}^{+}|\leq{\kappa}$ has the effect of excluding some representative segments from the margin loss as these segments are less likely to be abnormal (e.g., with a very low predicted score). Including these segments will increase the margin loss and coefficient $\lambda$ controls the balance between the margin loss and the diversity among the chosen segments.

We propose a greedy algorithm for optimizing the submodular function in (9) to ensure efficient model training. The proposed algorithm leverages the special structure of the state and mixture component space resulted from the HDP-HMM partition of the video segments. The performance guarantee of the greedy algorithm is ensured by our theoretical result presented at the end of this section.

Recall that we use $s_{i}$ to denote the mixture component of segment ${\bf x}^{+}_{i}$. Let $f^{*}_{s}$ denote the maximum score among all the segments assigned to the same mixture component and $i^{*}_{s}$ be the index of the corresponding representative segment:

$\displaystyle i^{*}_{s}=\arg\max_{\forall i:s_{i}=s}f({\bf x}_{i}^{+}),\;f_{s}^{*}=f({\bf x}_{i^{*}_{s}}^{+})$

(12)

We construct a representative set $\widehat{\mathcal{C}^{+}}$ as follows. Let $\widehat{\mathcal{C}^{+}}=\Phi$ and for each mixture component $s$, we set

$$\left\{\begin{array}[]{@{}ll@{}}\widehat{\mathcal{C}^{+}}\leftarrow\widehat{\mathcal{C}^{+}}\cup\{i^{*}_{s}\},&\text{if}\ f^{*}_{s}\geq\epsilon\\ \widehat{\mathcal{C}^{+}}\leftarrow\widehat{\mathcal{C}^{+}},&\text{otherwise}\end{array}\right.$$

(13)

where $\epsilon$ is a threshold to exclude segments with a low prediction score, which plays an equivalent role as constraint $|\mathcal{C}^{+}|\leq{\kappa}$ in (10). In our experiments, we use $\epsilon$ equal to the output of the segment staying in the 35th percentile among video specific segments so as to avoid skipping any potential abnormal segments. Once a representative set $\widehat{\mathcal{C}^{+}}$ is constructed, model training can proceed by solving the following MIL loss:

$\displaystyle\min_{{\bf w}}\Bigl{[}1-\frac{1}{|\widehat{\mathcal{C}^{+}}|}\sum_{i\in\widehat{\mathcal{C}^{+}}}f({\bf x}^{+}_{i})+\max_{j\in\mathcal{B}_{neg}}f({\bf x}_{j}^{-})\Bigl{]}_{+}\vspace{-2mm}$

(14)

Given the state and mixture component assignment of each segment in a video, the representative set can be quickly constructed by sorting segments within each component according to their predicted scores and choosing the representative segment from each component by comparing its score with the threshold $\epsilon$. Next, we provide a strong theoretical guarantee that the greedy algorithm can ensure the inclusion of a diverse set of segments for model training.

The detailed proof is provided in the Appendix.

Our experimentation includes three video datasets of different scales: ShanghaiTech [17], Avenue [16], and UCF-Crime [19]. Table 4 in the Appendix shows how the videos are partitioned into the training/testing sets in each dataset.

• •

  ShanghaiTech consists of 437 videos (330 normal and 107 abnormal). In the original setting, all training videos are normal. To fit into our setting, we follow the data split in [27] to assign normal and abnormal videos in both training and testing sets.

• •

  Avenue consists of 16 training and 21 testing videos. We perform 80:20 split separately in the abnormal and normal video sets to generate training and testing instances.

• •

  UCF-Crime consists of 13 different anomalies with a total of 1900 videos, where 1610 are training videos and 290 are testing videos. In this dataset, frame labels are available only for the testing videos.

To show the robustness of the proposed approach in the multimodal and outlier scenarios, we also generate the Multimodal and Outlier datasets. Specifically, we create a multimodal scenario by extending the UCF-Crime dataset. For the outlier scenario, we deliberately impose some outliers in the ShanghaiTech dataset. More details of these two datasets are provided in Section 4.3. For evaluation, we report the frame-level receiver operating characteristics (ROC) curve along with the corresponding area under curve (AUC). The AUC score indicates the robustness of the performance at various thresholds.

For Avenue and ShanghaiTech datasets, we extract visual features from the FC7 layer of a pre-trained C3D network [23]. We re-size each video frame to $240\times 340$ pixels and fix the frame rate to 30 fps. We compute the C3D features for every 16-frame video clip. This may yield a different number of clips (each clip having a 2048 dimensional feature vector) depending on the number of frames in each video. Thus, we fit any number of clips to the 32 segments by taking an average of clip features in a specific segment. In case of UCF-Crime, we extract the features using an I3D network [1] by using the pretrained network as described in [25]. For all datasets, we use parallel GCN networks to capture the feature similarity and temporal consistency. The outputs of the parallel branches are combined and passed through a 5-layer LSTM network where each layer has 32 hidden units followed by batch normalization. Finally, an FC layer with sigmoid activation is applied to bring the prediction score to $(0,1)$. For model training, we use SGD with a learning rate of $0.001$ and $l_{2}$ regularization with parameter $\lambda=0.001$. Detailed information about the network architecture is provided in the Appendix.

To show the effectiveness of extracting a diverse set of segments for model training, we present illustrative sample frames in a stealing video from UCF-Crime, where BN-SVP correctly identifies all abnormal frames and a top-$k$ approach (e.g., Avg Topk) misses some of them. In Figure 7, both frames are of abnormal types and but they occur in two distinct time intervals within the video. The first frame is more obvious for a stealing event. Consequently, both the proposed BN-SVP and Avg Topk are able to correctly identify it. In contrast, the second frame is less obvious for a stealing activity. Nevertheless, it is still chosen by BN-SVP due to its diverse coverage of potentially abnormal frames during the training process. On the other hand, Avg Topk only focuses on frames with high prediction scores that are usually co-located in the same time interval. This will narrow the scope of the model being exposed to other abnormal frames. Therefore, Avg Topk is not able to correctly predict the second frame and falsely classify it as normal. More qualitative analysis that demonstrates the robustness of the proposed approach on multimodal and outlier scenarios is provided in the Appendix along with an ablation study for the prediction score threshold $\epsilon$ defined in (13).