Real-Time Robust Video Object Detection System Against Physical-World
Adversarial Attacks

Video object detection, recognizing instances of visual objects (e.g, humans, cars, animals) and their locations in digital videos, is a fundamental important computer vision task. It forms the basis of many other computer vision tasks, such as instance segmentation [13], object tracking [14], and image captioning [15], etc.

Then, warps the key-frame features with the resized and scaled optic flow information to approximately compute the non-key-frame features. The predicted non-key-frame feature map is input to the DNN-suffix for the non-key frame result computation. Due to the large gap between the computation of DNN-prefix and DNN-suffix, the computation overhead of non-key frames is significantly reduced. These two VOD frameworks (image-based and feature-based) are adopted in different scenarios that are optimized for accuracy or performance. It is important to support effective and efficient defensive mechanisms in both these two cases. We propose Themis framework to achieve this goal.

Attack Formalization. Adversarial patch attack can largely damage video recognition tasks by evading video object detectors. It manipulates the victim model to output malicious results by adding the patch perturbation in the object of video data. Formally, the goal of adversarial patch attack is to generate the adversarial patch, $\hat{P}$, to maximize the expectation of possibility for classifier $h$ to output targeted malicious label $y_{p}$ with all adversarial inputs $x_{p}$ derived from data set $X$.

Patched image $x_{p}$ is generated by applying patch $p$ to $x$ in the input dataset $X$, which can be formalized as:

where $p$ is the adversarial patch, $x$ is the clean image, and A is the transformation function applying the adversarial patch on the clean image (environmental noises, resizing, rotations, and deformations).

The adversarial patch determines the prediction results with a very small region of pixels (adversarial region) for a relatively broad range of input images. By moving out the adversarial region, the adversarial effects are eliminated and the prediction results are recovered. Hence, autonomously identifying the adversarial region in the video data is the key foundation of robust object detection.

Patch Attacks vs. Example Attacks. Compared to adversarial example attacks [25] that have been largely studied in image classification tasks, patch attacks have the following advantages: 1) Better universality, because the adversarial patch is independent of input images. Such a scene-independent feature enables physical-world attack without prior knowledge of the scene. Adversarial example attacks, on the other hand, generate perturbation noises highly dependent on the input images, which hinders their deployment in the physical world. 2) Better robustness to environmental noises and geometric distortion. For example, human being wearing the T-shirt printed with the adversarial patch can be ignored in the object detection systems in different environments and body gestures [4]. It is not only effective in a static figure input, but also in complex scenarios like walking, sitting, and running [4]. More studies prove that adversarial patches are robust to not only environmental noises but also geometric distortion.

Adversarial example attacks, however, are scene-dependent and transfer poorly in different inputs. In real cases, the adversary neither can obtain the attack scene in advance nor compute the adversarial examples for every frame in real-time. Therefore, adversarial example attack is not a practical attack model in physical environments for video recognition tasks. In this following, we do not consider the adversarial example attacks.

Limitation of Existing Defenses. Although some prior researches propose the adversarial example detection methodology [26, 27], the differences between adversarial patch and the adversarial example noises hinder these countermeasures’ applicability in the patch attacks. Additionally, these countermeasures can only recognize whether the input is a benign image or not, but cannot recover the DNN system from the attack effect. In this study, we aim to rescue the video recognition tasks by not only detecting the malicious input but also eliminating the adversarial effect by moving out the malicious patches. More adversarial defense comparisons are introduced in Section 7.

The overall Themis system is shown in Fig. 5 with algorithm, framework, and hardware designs.

Algorithms: LISF-based methodology effectively targets the adversarial regions in input images and recovers the prediction results by moving out those regions.

Framework: Although the Themis algorithm already reduces the adversarial region searching space with LISF-based methodology, multiple inferences are introduced in the occluding testing stage and incur large overhead. To achieve the goal of real-time robust object detection, Themis proposes systematic design to efficiently remove redundant computations for better computing efficiency.

To further reduce the detection overhead, Themis proposes the inter-frame and intra-frame optimizations to reduce redundant computations. 1) Inter-frame optimization: By leveraging the spatial and temporal locality between frames, Themis only performs complete adversarial detection in key frames. For non-key frames, Themis framework either predicts the adversarial region locations based on regions detected in key frames (image-based warping) or reuse the clean features in key frames after eliminating the adversarial effects (feature-based warping). These two warp strategies can be easily integrated with existing AO and PO video object detection frameworks. 2) Intra-frame optimization: The algorithm introduces a large volume of redundant calculations of benign features during occluding prediction stage for the key frames. Hence, Themis scheduler reduces the computing overhead with computation reuse of benign features (Section 4.2). Additionally, Themis provides the interfaces of setting the following configurations for better feasibility: key frame proportion, the video object detector model segmentation strategy, and the configurable parameters in adversarial patch detection algorithms.

Hardware: Although Themis algorithm can be implemented in pure software, it is inefficient because of the following reasons: 1) Searching the heat-map of the input activation for the adversarial candidates is inefficient in the DNN accelerator due to the lack of the computing parallelism between the searching process and typical inference process; 2) During the voting stage, multiple patch candidate regions will be occluded to calculate the inference results. In this process, a significant volume of benign features is computed repeatedly and introducing large unnecessary performance overhead. To address these issues, we propose the $Themis$ hardware architecture for efficient defense. The top-level block diagram of hardware architecture is shown in Fig. 5 (c). Apart from the typical DNN accelerators with PE array, scalar function unit (SFU), global buffer, and control logic, Themis is augmented with the LISF searching logic to search the candidate regions according to the heat map of the first superficial layer feature map, Masked Neuron Buffer to optimize the data and computation reuse of benign features, and the voting logic to decide the final prediction results.

1) Image-based Warping in AO Frameworks: For AO framework, we approximately estimate the patch location in non-key frames by warping the patch location in key-frame images with the optical flow. Specifically, every frame will be forwarded to the object detector for a complete inference. $Themis$ only performs the adversarial patch detection in key frames and obtains the adversarial region location. Then warps the detected adversarial region in key-frames with the optical flow information to estimate the adversarial region in non-key frames. The corresponding area of non-key frames is masked and the masked non-key frames are forwarded to the object detector for inference. The rationality is that the adversarial regions in the input data also exhibit temporal and spatial redundancy in live vision. Moreover, compared to the object to be segmented, the adversarial region is much less sensitive to the derivation brought by the inaccurate optical flow information (more validation in Section 6.1).

2) Feature-based Warping in PO Frameworks: For PO framework, we directly warp the clean features in key-frames with the resized and scaled optical flow to compute the non-key-frames features. Then, the predicted feature map is forwarded to the DNN-suffix for the final object detection results. Under both AO and PO cases, the defensive overhead of non-key frames is minimized.

Key frame: We set the key frame rate as 10% with fixed length. Themis framework can support the adaptive key frame strategies proposed by prior video object detection studies [21]. However, how to choose the key frame is out of the scope of this work.

PO framework splitting: Designing DNN prefix and suffix in PO frameworks is the trade-off between performance and detection accuracy. When the prefix is close to the classification layers, the feature map is too small and may introduce larger accuracy degradation during optical flow estimation. We test extensive datasets and set the splitting spot when the feature map size is smaller than 56$\times$56.

Adversarial region detection parameters: In the adversarial patch detection stage, the more adversarial candidates, the more rounds of the additional inferences and larger performance overhead are. The number of the patch candidates is determined by the heat spot selected threshold ($\beta$ and $\theta$). We set $\beta=0.75$ and $\theta=0.85$.

In observing that important superficial features exhibit different spatial distribution characteristics and exert distinct influence on the prediction results in benign and adversarial input data, we propose the LISF-based patch candidate searching methodology and then detect and eliminate the adversarial effect by occluding testing.

LISF-based Patch Candidate Searching. We perform the LISF searching in the output feature map of the first layer. The size of the searching window is the upper limit of the patch sizes and the searching stride is 1. When the number of important neurons in one searching window is larger than the threshold ($\theta$), it is recognized as an important window and marked as the patch candidate. When several important windows are overlapped, the central important window will be retained as patch candidate with all the other deleted. The detailed searching logic implementation is described in Section 4.4.

Occluding Testing and Recovery. We recover the prediction results with the following two steps:

1) Masked image execution. After obtaining the candidate locations of adversarial patches, we generate masked images by occluding the patch candidate locations individually from the original images. These masked images are taken into the victim model to produce the prediction decisions.

2) Monopolist-occluded voting. Themis performs the prediction decision analysis to detect patched images by examining whether there is a monopolist patch candidate that determines the prediction results. It is true that such a methodology will introduce the true-negatives. However, the results show that such cases are rare and Themis can achieve good detection effectiveness. The detailed patch candidate searching and voting mechanisms are introduced as follows. There are $k$ candidates: $P_{1}$, $P_{2}$, …, $P_{k}$. The prediction result of the original image is $L_{0}$, and the corresponding prediction results of masked images: $L_{1}$, $L_{2}$, …, $L_{k}$. We first detect the patch by checking whether there is a $L_{i}$ distinct from others, while all the other labels are the same: If positive, the $P_{i}$ is the monopolist that dominates the prediction results, which is the adversarial patch. Only when tearing it off the image, the classifier predicts the robust and benign label $L_{i}$.

If there is only one candidate, i.e. $k=1$, we compare $L_{1}$ and $L_{0}$ to determine whether there is an adversarial patch. If they are different, $P_{1}$ is the adversarial patch and $L_{1}$ would be recovered label.

If there is no such particular label (either all the labels are the same, or several labels are different), no monopolist is detected. Then the image is recognized as a benign image. Themis then performs the majority voting to obtain the predicted label.

The Themis architecture can be integrated into the typical DNN hardware accelerator to support real-time detection with small overhead.

LISF searching logic. The LISF searching logic outputs the coordinates of clustered important neurons, which infers the possible candidate locations in the input image. LISF searching consists of the following three steps:

1) Obtain the binary important (output) feature map of the first layer. Computing Top-K neurons in the feature map is time-consuming and introduces large hardware overhead. Therefore, we make the estimation about the adaptive threshold according to the maximum value of the feature map. All neurons with activation values larger than the threshold ($\beta*feature_{max}$) are selected as important neurons. The important neurons map is stored in the buffer of LISF searching logic.

2) Identify the important windows. We slide the fixed-size window to make statistic counting about the important neuron numbers in one window. When the important neuron numbers occupy more than the threshold ($\theta$) of the total neurons, we mark this window as an important neuron window, which will be highlighted as the patch candidates. To reduce the hardware overhead, we make the incremental accumulation of the important neurons in one window. As shown in Fig. 6(a), with the number of important neurons in Window[i,j,i+s,j+s], to compute the Window[i,j+1,i+s,j+1+s], we simply subtract the important neurons in column j+1 and add the important neurons in column j+1+s between row i and row i+s.

3) Delete the overlapped important windows. When two sliding important windows have more than 30% of the area overlapped, we take them as one single patch candidate.

Noted, the LISF searching is not on the critical path of DNN inference, which is parallelized with the processing of the original images.

Masked Neuron Buffer (MNB). To support efficient computation reuse of benign features and optimize the data accesses of the adversarial features, the masked neuron buffer is proposed to buffer the padded bounding area for feature computation of candidate adversarial regions.

The data reuse flow of computing candidate adversarial regions is as follows (Fig. 8): With the coordinates of potential adversarial region candidates, the padded bounding areas through all the neural network layers in Fig. 7(a) are determined. Through the typical inference, all the activation values of the padded bounding areas are stored in Masked Neuron Buffer. Then, during the inferences of masked images, all the masked regions are batched for computation. For every layer, the PE array reads the candidate adversarial features from global buffers and the padded bounding areas from masked neuron buffers, as the red box and purple box shown in Fig. 8②. The PE array combines these neurons by padding the red box with the purple box, takes it as input, and computes the candidate adversarial feature of next layer. After completing the computation of this layer, the adversarial features are stored back to global buffers. Because the padded bounding area is very small compared to the full activation map, we configure the mask neuron buffer with a size of 8KB for every PE array.

Voting Logic. The basic voting logic is as shown in Fig. 6(b). We use a compactor array to perform the pairwise comparison between the prediction labels of masked images ($L_{0}$, $L_{1}$, …, $L_{k}$). If there is an orphan label $L_{i}$, this image is identified as a patched image and the recovered label is $L_{i}$. Otherwise, it is a benign image and Themis performs the majority voting to obtain the recovered label.

Computing Flow. The overall computing flow is as follows: taking in a new input image, the DNN accelerator first performs the normal inference procedure on the image. Meanwhile, the searching logic obtains the candidate masked regions in the input based on the localized superficial important features. During the inference procedure of the original images, the feature data of corresponding padded bounding box are stored in the masked neuron buffer. After the completion of the original image inference, masked image inference rounds start to get their prediction results. After that, taking in those prediction results, voting logic detects the adversarial patches and output the recovered results.

For one input image, searching and voting operations are only performed once, while the inference rounds are determined by the searching candidates.Among these stages, multiple inferences introduce the most performance cost. Voting is simple and performed within several cycles. Although the searching algorithm is time-consuming when offloaded to the CPU platform, its customized hardware largely boosts the performance and the overhead is less than 0.5% of the multiple inferences. The detailed experimental evaluation is illustrated in Fig. 15.

We test the attack success rate and the defensive effectiveness in both single-frame object detection tasks and the video object recognition tasks. For the previous scenario, we focus on exploring and validating the adversarial patch detection capability of Themis algorithm on static images. In the latter scenario, we focus on exploring the defensive effectiveness on the non-key frames when Themis framework leverages the temporal and spatial information in video data.

Attack Methodologies: For single-frame testing scenario, we adopt the digital-synthesized attack methodology that randomly attaches the digital adversarial patches onto the bounding box regions of the objects in MS COCO, FLIC, LSP datasets and random locations of images in ImageNet dataset with random rotated angles. The patch size is scaled with the area of bounding boxes ranging from 33x33 to 130x130 pixels. Since we focus on the defensive effectiveness of Themis, we only perform attacks on the objects or images that can be correctly identified by the detector. For video data testing scenario, we adopt physical attack videos released in the state-of-the-art attack methodology [4] (Adv T-shirt), which significantly damages the functionality of YOLOv2 object detector. The patch size is variable and the adversarial patch has been significantly deformed during the human movement.

Optical Flow Methodologies: Themis framework is compatible with different optical flow methodologies. We use both the CV-based and DNN-based optic flow methodologies: DIS [22] and SpyNet [23], to validate the defensive effectiveness and architecture efficiency. The SpyNet is customized with scaled input image sizes and reduced pyramid levels.

Evaluation Metrics: We adopt the commonly-used metrics, the detection rate and the standard mean average precision (mAP) score, to measure the accuracy of video object detection. In terms of performance efficiency, the frame per second (fps) is used to indicate how fast the framework process the video data.

We implement the Themis hardware design in Verilog RTL. To obtain the area and power, we synthesis and place&route the RTL code with Synopsys toolchains under TSMC 28nm technology. We use CACTI 7 and DESTINY to model the DRAM memory and on-chip SRAM buffers. Due to the unbearable long duration of silicon simulation, we also tailor an open-sourced simulator, nn-dataflow to support Themis for the total execution latency simulation. The simulator also reports the exact memory traces and module activities, which are then used to calculate dynamic energy consumption.

Besides, we deploy Themis system on an off-the-shelf FPGA, Zynq UltraScale+ MPSoC ZCU104 board, to show the real performance. Specifically, we implement the proposed architecture components like Adv Candidate Search Logic and integrate them with a DPU (a soft IP for NN inference) using Vitis-ai framework.

We validate the defensive effectiveness of Themis under the following two scenarios: single-frame data and the video data with sequential frames.

Video object detection is the fundamental important computer vision task and grows rapidly with the increasing demands in autonomous driving and video surveillance [34]. Extended from image to video domain, DNN techniques largely boost video recognition capability. YOLO series [16], SSD [17], RCNN series [35], are proposed in rapid succession. Compared to single image object detection, video object detection has new attributes of existing both spatial and temporal correlations within consecutive frames. Previous studies leverage such attributes to improve the performance of video recognition tasks [19, 21, 20]. In addition to the optimized video recognition frameworks that make use of temporal-spatial information for computing efficiency, some recent studies enhance feature maps with tempo-spatial information to improve detection accuracy that originally degraded by motion blur, rare poses, video defocus, etc [34, 36]. Besides, LSTM-based [37],attention-based [38],tracking-based [39]video object detectors are also proposed. Themis algorithm can support such video recognition frameworks well, because it is able to directly identify the adversarial region of input and supports feature aggregation or propagation operations during leveraging the tempo-spatial video information.

Adversarial patch attack is one of the most practical DNN attack models that can effectively damage object detectors even in physical environments [4]. Envisioning its importance, prior studies contribute to the defense techniques against the adversarial patch attack by 1) building certified robust neural networks that resist the attack effect, such as IBP, de-randomized smoothing, etc [12, 40, 41]; However, all of these certified studies achieve distinctly low certified accuracy for large scale datasets such as ImageNet. A certified accuracy of 20.5%-36.3% from these methods can hardly be applied in the real world systems. 2) adversarial training to obtain robust model again patch attack [42], in which online re-training process introduces unacceptable cost. 3) performing patch detection based on empirical observations to eliminate the attack effect, such as digital watermarking (DW) [43], local gradient smoothing (LGS) [32]. However, these defenses proved to be invalidated when confronting with the strong adaptive attacker that has the white-box knowledge of the defense [40]. Note that since locality is the intrinsic property of patch attack, our LISF-based detection algorithm prevents the adversary from maintaining both strong attack effect and stealthiness, which promises the effectiveness of Themis even under the adaptive attack.

The most related recovery methodology are MRD [10] and ObjectSeeker [11]. MRD [10] requires a large amount of iterative inference passes to obtain the prediction map by masking every small region sliding across the entire original input images. It is extremely time-consuming and costs 1446s for detection of one single image with sizes of 224$\times$224 on an Eyeriss-scale accelerator. Similarly, ObjectSeeker [11] proposes patch-agnostic masking for certified objection detection that needs more than 100 inferences for different bands of the input image. Therefore, both MRD and ObjectSeeker are unacceptable for real-time detection due to their additional large cost.

In summary, existing countermeasures are unable to perform online defense for video recognition tasks. This work proposes a high-efficient and effective detection and recovery system to defend the adversarial attacks that practically introduce damaging consequences in video scenarios.

Neuron importance has been widely used for abnormal input detection [44, 45]in previous studies. However, the metric (superficial feature importance) in our methodology is distinct from previous work. Previous studies focus more on the neurons that contribute significantly to the inference output (deep feature importance). We envision that deep feature importance is not a good candidate from the following two aspects: 1) both the benign images and the adversarial images have the deep feature importance. Its discrimination in the benign images and adversarial images is not straightforward, so that it requires more complex computation to identify the benign and adversarial images. 2) calculating such deep feature importance is time-consuming, which demands the gradient information and the complete backward propagation process. We propose the superficial input feature importance as the metric for discrimination analysis based on the intuition that in order to efficiently manage the output prediction results with a very small region of the input data, the adversarial patch must incur large activation from the first place instead of the accumulation of the deep feature extraction.