Relevance Attack on Detectors

• 1.

  We propose a novel attack framework on relevance maps for detectors. We extend DNN interpreters to detectors, find out the most suitable outputs to attack by relevance maps, and explore the best update techniques to increase the transferability.

• 2.

  We evaluate RAD on 8 black-box models and find its state-of-the-art transferability, which exceeds existing results by above 20% in mAP. Detection and segmentation performance is greatly impaired in various metrics, invalidating the state-of-the-art DNN to a very rudimentary counterpart.

• 3.

  By RAD, we create the first adversarial dataset for object detection and instance segmentation, i.e., AOCO. As a potential benchmark, AOCO is generated from COCO and contains 10K high-transferable samples. AOCO helps to quickly evaluate and improve the robustness of detectors.

We present the framework of RAD in Fig. 2. Initialized by the original sample $x_{0}$, the adversarial sample $x_{k}$ in the $k^{\text{th}}$ iteration is forward propagated in the surrogate model, getting the prediction $f(x_{k})$. Current attacks generally suppress the prediction values of all attacked output nodes in $T$, where an output node stands for an output scalar of the detector. In contrast, RAD suppresses the corresponding relevance map $h(x_{k},T)$. To restrain that, gradients of $h(x_{k},T)$ back propagate to $x_{k}$, which is then modified to $x_{k+1}$.

Notably, RAD is a complete framework to attack detectors, and its components require special design. Besides the calculation of relevance maps of detectors, other components in RAD, e.g., the attacked nodes or the update techniques, also need a customized analysis. The reason is that no existing work directly attacks the relevance of detectors, and the experience in attacking predictions is not applicable here. For example, [17] emphasizes classification loss and localization loss equally, but the former is validated to be significantly better in attacking the relevance in Section 3.4.

RAD’s transferability comes from the attack goal: changing the common properties, i.e., the relevance maps, which are the same for different detectors because they are developed to interpret the salient parts in the data, and thus is data-dependent and model-independent [47, 38, 39] for diverse well-trained models, which could be observed in Fig. 1 and E. As shown in Fig. 3, the relevance maps are clear and structured for the original sample in both detectors. After RAD, the relevance maps are induced to be meaningless without a correct focus, leading to wrong predictions, i.e., no or false detection. Because relevance maps transfer well across models, those for black-box detectors are also significantly influenced, causing a great performance drop, which is illustrated visually in Section 4.2.

RAD also attacks quite “precisely”, i.e., the perturbation pattern is significantly focused on distinct areas and has a clear structure as shown in Fig. 4. That is to say, RAD accurately locates the most discriminating parts of a sample and concentrates the perturbation on them, leading to a great transferability when the perturbations are equally bounded.

We analyze the potential of RAD above, below we make it feasible by addressing three crucial issues. To conduct the relevance attack, we first need to know the relevance maps for detectors.

Currently, there exist lots of interpreters to calculate the relevance maps for classifiers as described in Section 2, but none of them are suitable for detectors. We take SGLRP [44] as an example to introduce the relevance map for classifiers and then modify it for detectors because it excels in discriminating against irrelevant regions.

For a given deep classifier, the relevance map is defined as a normalized heat map with the same dimensions as input $x$, visualizing how $x$ contributes to a specific output node $t$ in pixel-wise manners. We denote this map as $h(x,t)$, which, in SGLRP, is obtained by back-propagating the “relevance” $R$ from the output layer (layer $L$) to the input layer (layer $1$) after the forward inference. Suppose layer $l$ has $N$ nodes (dimension of features) and layer $l+1$ has $M$ nodes, the relevance $R_{n}^{(l)}$ at node $n$ in layer $l$ is defined recursively by

for nodes with definite positive values (such as after ReLU), and

for nodes that may have negative values. In the formulas above, $a_{n}^{(l)}$ is the post-activation output of node $n$ in layer $l$ and $z_{n}^{(l)}$ is the pre-activation one. The range $[b_{n}^{(l)},h_{n}^{(l)}]$ stands for the minimum and maximum of $z_{n}^{(l)}$, and $w_{n,m}^{+(l)}=\max\left(w_{n,m}^{(l)},0\right)$, $w_{n,m}^{-(l)}=\min\left(w_{n,m}^{(l)},0\right)$.

According to the relevance propagation rules above, the relevance map $h(x,t)=R^{(1)}$ is calculated by recursively back-propagating the relevance in the output layer $R^{(L)}$, of which the $n^{\text{th}}$ component $R_{n}^{(L)}$ is defined in SGLRP as

where $y_{n}$ is the predicted probability of class $n$, and $y_{t}$ is that for the single target class $t$.

In detectors, however, we need the pixel-wise contributions from the input to $m$ bounding boxes. This multi-node relevance map could not be directly calculated by (3), so we naturally modify SGLRP as

where ${y}_{t_{i}}$ is the predicted probabilities for one target output node $t_{i}$. $T$ is the set containing all target output nodes $\{t_{1},t_{2},...,t_{m}\}$. With iNNvestigate Library [48] to implement Multi-Node SGLRP and Deep Learning platforms supporting auto-gradient, the gradients from RAD loss $L_{\text{RAD}}(x)=h(x,T)$ to sample $x$ could be obtained according to the relevance propagation rules.

We illustrate the difference between SGLRP and our Multi-Node SGLRP in Fig. 5. SGLRP only displays the relevance map for one bounding box, e.g., “TV”, “chair” and “bottle”. Multi-Node SGLRP, in contrast, visualizes the overall relevance.

Besides the calculation of relevance maps, it is also important to choose a proper node-set $T$ to attack. Specifically, we need to select certain bounding boxes and the corresponding output nodes for RAD.

Heuristically, the most “obvious” bounding boxes are desired to be eliminated, so we select the bounding boxes with the highest confidence, following [22]. Concretely, it is feasible to statically choose several bounding boxes to attack in each iteration, or dynamically attack all bounding boxes whose confidence exceeds a threshold. In our evaluation, the two strategies differ a little in performance and are not sensitive to hyper-parameter as demonstrated in A. This shows that RAD does not require a sophisticated tuning of parameters, which is user-friendly. In our following experiments, we statically attack 20 nodes.

After selecting bounding boxes, we could attack their size, leading them to shrink; or their localization, leading them to shift; or their confidence, leading them to be misclassified. To adopt the best strategy, we conduct a toy experiment by attacking YOLOv3 [49], denoted as M2 (other models are specified in Table 3), following the settings later in Sec. 4. Given the results in Table 1, the classification loss induces a better black-box transferability. This may be because detectors generally include a pre-trained classification as the feature extractor, and relevance maps are believed to be an indicator of successful attacks [28, 29]. Note that our method is applicable to both one-stage detectors and two-stage ones because they both have classification outputs which we target on.

With the relevance map $h(x,T)$ for certain attacked nodes $T$, we are able to attack, i.e., update the original sample to become adversarial by suppressing the relevance map according to the attack gradients $g(x)$ as

Some update techniques are validated to be effective for enhancing the transferability in classification. For example, Scale-Invariant (SI) [15] proposes to average the attack gradients by scale copies of the samples as

Besides SI, Diverse Input (DI) [14], Translation-Invariant (TI) [28] are also promising in classification. We are curious about whether they also work well in object detection. To explore this, we adopt these techniques in RAD as the setting suggested by their designers (see Sec. 4 and C). From the results in Table 2, we discover that SI is quite effective, further decreasing the mAP from the baseline significantly. Accordingly, RAD adopts (8) to update.

With the calculated gradient, we update the sample like PGD attack [4] as

where $\alpha$ stands for the step length. $x$ is $\ell_{\infty}$-norm bounded by $\varepsilon$ from the original sample in each iteration as in [14, 15, 28]. Gradient $g(x)$ is normalized by its average $\ell_{1}$-norm,i.e., $||g(x)||_{1}/N$ to prevent numerical errors and control the degree of perturbations. $N$ is the dimension of the image, i.e., $N=height\times width\times channel$. Division by $N$ is necessary because $\ell_{1}$-norm sums all components of the tensor $x$, which is too large as a normalization factor. We do not adopt the mainstream sign method because it is not suitable to generate small perturbations as shown in other attacks in detectors [22].

Our experiments are based on Keras [50], Tensorflow [tensorflow2015-whitepaper] and PyTorch [51] in 4 NVIDIA GeForce RTX 2080Ti GPUs. Library iNNvestigate [48] is used to implement Multi-Node SGLRP.

We conduct experiments on MS COCO 2017 dataset [31], which is a large-scale benchmark for object detection, instance segmentation, and image captioning. For a fair evaluation, we generate adversarial samples from all 5K samples in its validation set and test several black-box models on their mAP, a standard measure in many works [52, 53].

All attacks are conducted with the step length $\alpha=2$ for 10 iterations and the perturbation is $\ell_{\infty}$-bounded in $\varepsilon=16$ to guarantee the imperceptibility as in [28] if not particularly specified. To validate that the mAP drop comes from the attack instead of resizing or perturbation, we add large Gaussian noises ($\sigma=9$) to the resized images, and report it as “Ablation”.

We choose 8 typical detectors ranging from the first end-to-end detector to state-of-the-art counterparts for attack and test. The variety of models guarantees the validity of results. We specify their information in Table 3 and the corresponding pre-processing or details in C.

We first intuitively illustrate the attack process in Fig. 6 and the attack transferability in Fig. 7.

By RAD, the relevance map is attacked to be meaningless and loses its focus. In Fig. 6, the initial prediction is correct and the relevance map is clear. RAD constantly misleads the relevance map to be unstructured without the outline of objects. Finally, all bounding boxes vanish.

In Fig. 7, we visualize several predictions on the same adversarial sample by black-box models. The objects in the image, e.g., the laptop and keyboard, are quite large and obvious to detect. However, with a small perturbation from RAD, 5 black-box models all fail to detect the laptop, keyboard, and mouse. Surprisingly, 4 of them even detect a non-existent “bed”, which is neither relevant nor similar in the image.

To evaluate the in-domain transferability of detection attacks and cross-domain transferability of classification attacks, we test the detection mAP of 8 models in COCO adversarial samples generated in the setting stated before.

For detection attacks, adversarial samples are crafted by attacking surrogate model M2 (YOLOv3 [49]). For classification attacks, we use the model output on the clean sample as the label. By several state-of-the-art attacks on surrogate classifiers (InceptionV3 [59] here as in [14, 28]), the adversarial samples are generated and tested the mAP as the transferability towards detectors. Details of implementation are described in C.

We present the results in Table 4. Among the classification attacks and detection ones, cross-domain attack [60] is effective, but RAD is more aggressive. RAD enjoys a state-of-the-art transferability towards most black-box models, outperforming other methods for above 20%. The detection mAPs are more than halved, making state-of-the-art detectors similar to the early elementary counterpart. Also, the adversarial samples from the one-stage detector (M1) could be transferred to two-stage ones (M4 - M8, seeing Table 3) as the case in attacking classifiers [13, 14, 15].

Although mAP is a general metric to test detectors under a fixed bound of perturbations, it is also interesting to investigate RAD’s transferability under different bounds and sub-tasks, i.e., classification accuracy, the shift of bounding boxes, and their invisibility to detectors.

Here we first vary the $\ell_{\infty}$ bound of RAD, and report the mAP in Fig. 8. With the $\ell_{\infty}$ bound increases, the resulting mAP greatly decreases for all black-box models especially for $\varepsilon$ from 8 to 12.

We further study RAD’s transferability in different metrics, which is necessary because mAP measures the overall performance of detectors, and thus could not decouple the RAD’s aggression on, e.g., shifting and hiding bounding boxes. For subsequent evaluations on sub-tasks, we define the metrics below.

• 1.

  Accuracy for classification of bounding boxes: Without considering locations, we use the predicted box classes to calculate the classification accuracy. For each image, $a$ predicted bounding boxes with the highest confidence (after non-maximum suppression) are considered, where $a$ is the number of ground truth bounding boxes for rationality. For example, if the detector predicts 3 cats, 2 dogs, and 1 car in an image, which has 2 cats, 3 dogs, and 1 person, we count $2+2=4$ hits and 2 misses.

• 2.

  Intersection over Union (IoU) for shifting of bounding boxes: Focused on locations, we average the IoU, the measure of location correctness, on all predictions to quantify the shifting. Also, we select $a$ predicted bounding boxes with the highest confidence, and compute the IoU for each box with the nearest ground truth in the same class (IoU=0 for wrong prediction).

• 3.

  Mean Average Recall (mAR) for hiding of bounding boxes: To see how attacks hide the object, the common recall value is appropriate, and the mAR is calculated from recall in the same way that mAP is from precision, which has been adopted in detection libraries.

To the best of our knowledge, existing works mostly focus on mAP [23, 24, liu2018dpatch], and we here suggest a more comprehensive way to evaluate attacks on detectors. The results are reported in Table 5, from which one could observe that RAD also outperforms its counterparts in different sub-tasks to a large extent, i.e., up to 10%. The predicted bounding boxes in attacking white-box M2 are too few to evaluate fairly, so we do not show their results.

Detection and segmentation are similar in some aspects, so they could be implemented in one network [52, 57, 53]. Also, adversarial samples for object detection tend to transfer to instance segmentation [22]. Accordingly, we evaluate this cross-task transferability by RAD on surrogate detectors YOLOv3 ([49], M2), RetinaNet ([55], M3) and Mask R-CNN ([52], M5). From the results in Table 6, we find that RAD also greatly hurts the performance of instance segmentation, leading to a drop in mAP of over 70%. This inspires the segmentation attackers to indirectly attack detectors.