Saliency Attack: Towards Imperceptible Black-box Adversarial Attack

The imperceptibility of AEs is essential for practical attackers, which has been investigated by previous studies from different perspectives. Guo et al. (Guo et al., 2019) consider low frequency perturbations as imperceptible, thus searching for AEs in frequency domain. But Zhang et al. (Zhang et al., 2020) regard imperceptibility as visual smoothness in an image and integrate Laplacian smoothing into optimization. Croce and Hein (Croce and Hein, 2019) use a combination of $L_{0}$ and $L_{\infty}$ norms to generate sparse and imperceptible perturbations. Besides, some studies also leverage color distance (Zhao et al., 2020) and image quality assessment (IQA) (Wang et al., 2021; Li and Chen, 2021) to improve attack imperceptibility.

On the other hand, determining how to assess the imperceptibility of AEs is still an open question. Most existing adversarial attacks use $L_{p}$ norms ($L_{0}$, $L_{2}$ and $L_{\infty}$) to measure the human perceptual distance between the perturbed image and the original one. Nonetheless, it has been shown that $L_{p}$ norms are not suitable enough for human vision system (Sharif et al., 2018). A recent study in (Fezza et al., 2019) systematically examines various IQA metrics including $L_{p}$ norms through large-scale human evaluation on the imperceptibility of different AEs, and finds that among all the metrics the most apparent distortion (MAD) (Larson and Chandler, 2010) metric is closest to subjective scores (see Appendix A.1 for details of MAD). Hence, in this research, we adopt MAD as our main metric to assess attack imperceptibility.

As aforementioned, there have been white-box attacks or transfer-based attacks that restrict the perturbations to a small salient region. Specifically, white-box attack JSMA (Papernot et al., 2016) constructs a BP-saliency map by calculating derivatives of the model output w.r.t input pixels (Simonyan et al., 2014), while the two transfer-based attacks (Dong et al., 2020; Xiang et al., 2021) utilize CAM and Grad-CAM to extract salient regions, respectively. CAM (Zhou et al., 2016) replaces the final fully connected layers with convolutional layers and global average pooling of a CNN, and localizes class-specific salient regions through forward propagation. Grad-CAM (Selvaraju et al., 2017) improves CAM with on need to modify the network architecture, but it still requires access to the inner parameters of the model to calculate the gradients. Thus, CAM and Grad-CAM can only be applied to white-box attacks or transfer-based attacks, which first construct a transparent substitute model, craft AEs with gradient-based white-box attacks, and then transfer the generated AEs to the target model. It is conceivable that for transfer-based attacks, the transferability of both AEs and saliency maps highly depends on the similarity between the substitute model and the target model, which further depends on the prior knowledge of the training data distribution. Unfortunately, neither such knowledge nor model gradients are available in black-box settings. Considering the limitations of the above methods, we adopt a salient object detection model (Zhao and Wu, 2019) to directly generate the saliency map given an input image with no need to access the target model’s architecture or parameters.

Given a well-trained DNN classifier $h:[0,1]^{d}\rightarrow\mathbb{R}^{K}$, where $d$ is the dimension of the input $x$, and $K$ is the number of classes. Let $h_{k}(x)$ denote the predicted score that $x$ belongs to class $k$. The classifier assigns the class that maximizes $h_{k}(x)$ to the input $x$. The goal of an untargeted attack is to find an AE $x_{adv}$, that results in the model misclassification from the ground-truth class $y_{gt}$, and meanwhile keeps the distance between $x_{adv}$ and the benign input $x$ smaller than a threshold $\epsilon$:

Note the last constraint indicates that $x_{adv}$ is a valid image. In this study, we focus on $L_{\infty}$ norm, following (Moon et al., 2019; Andriushchenko et al., 2020). Conventionally, the task of finding $x_{adv}$ can be rephrased as solving a constrained continuous problem:

where $f(x)=L(x,y_{gt})$ is the loss function. Similar to (Moon et al., 2019), we transform the continuous problem into a discrete surrogate problem, where the perturbation $x_{adv}-x=\delta\in\{-\epsilon,+\epsilon,0\}^{d}$. Note unlike (Moon et al., 2019), where all pixels are perturbed either by $-\epsilon$ or $+\epsilon$, we allow the pixels to remain unperturbed to refrain from global perturbation. Besides, only the pixels in salient regions can be perturbed. The final problem is defined as the following set maximization problem.

where $\mathcal{V}$ denotes the ground set which is the set of all pixel locations ($|\mathcal{V}|=d$), $\mathcal{S}$ denotes the set of pixels in salient regions, $\mathcal{S}^{+}$ and $\mathcal{S}^{-}$ denote the set of selected pixels with $+\epsilon$ and $-\epsilon$ perturbations respectively, and $e_{i}$ is the $i$-th standard basis vector. Note that $\mathcal{S}\subseteq\mathcal{V}$, and $\mathcal{S}\setminus(\mathcal{S}^{+}\cup\mathcal{S}^{-})$ is the set of pixels in salient regions that are unperturbed. The goal of Equation (3) is to find $\mathcal{S}^{+}$ and $\mathcal{S}^{-}$ that will maximize the objective set function $F$.

Salient object detection aims to automatically and accurately extract salient object(s) in an image. Compared with other salient region extraction methods like BP-saliency map (Simonyan et al., 2014), CAM (Zhou et al., 2016) and Grad-CAM (Selvaraju et al., 2017), this type of models do not require any information other than the input image, which is very suitable for the black-box setting. We adopt the Pyramid Feature Attention (PFA) network¹¹1Implementation of PFA: https://github.com/sairajk/PyTorch-Pyramid-Feature-Attention-Network-for-Saliency-Detection (Zhao and Wu, 2019) that achieves the SOTA performance on multiple datasets via capturing high-level context features and low-level spatial structural features simultaneously (see Appendix A.2 for details). Specifically, given an input image, PFA can generate a saliency score between 0 and 1 for each pixel. The higher value denotes higher visual saliency. We then use a threshold $\phi$ to transform the saliency scores into a binary saliency mask, which determines the salient region. The binarization can be expressed as

where $s_{i}$ and $s^{*}_{i}$ are the saliency score and the binary mask at the $i$-th pixel position, respectively. Thus, the salient region is the set of the pixels masked by 1.

The proposed Saliency Attack is outlined in Algorithm 1. The search process (lines 6-35) recursively refines perturbations based on a tree structure for an image, which is illustrated in Figure 4. Concretely, an input image is first split into some initial blocks (a coarse grid in Figure 4), and only the blocks in salient regions are kept (lines 7-11). Then we try to add $+\epsilon$ or $-\epsilon$ perturbations on each initial block individually (lines 12-19) and sort them according to their influence ($F$) on model output (line 26). After that, we choose the block with most influence for further refinement if its $F$ is better than the current best $\hat{F}$ (lines 27-35). We then recursively refine the current best block (line 32), split this block into smaller blocks (a finer grid in Figure 4), and again try to add a perturbation on each block individually (at this time we just flip the perturbation (lines 21-24) for convenience, e.g., $+\epsilon$ to $-\epsilon$) to find the best smaller block. The refinement process recurs until the minimal block (e.g., 1 pixel, line 30) or no smaller block has a better $F$ (line 28). Afterwards, we backtrack to the last level of split blocks and use the second-best block for further perturbation. In this way, the most important block will be explored first, and the perturbations could be as small as possible, with no need to initialize a global perturbation like Parsimonious Attack (Moon et al., 2019).

To make full use of query budget and combine the advantages of different initial block sizes $k_{int}$ (large $k_{int}$ can quickly lead to successful attacks with large perturbations, while small $k_{int}$ enables the refinement for small perturbations in finer grids), we leverage an outer iteration to run the aforementioned Refine search with decreasing $k_{int}$ (lines 2-5). During the iterations, the algorithm will stop if the generated $x_{adv}$ succeeds to fool the model ($F>0$) or the termination condition is reached (line 2).

We use the loss function from CW attack (Carlini and Wagner, 2017) of untargeted attack for Equation (2):

where $Z(x_{adv})_{y_{gt}}$ is the logit with respect to the ground-truth class of the original image. In this way, the loss function is imposed to leave a margin between the ground-truth class and other classes.

We compare our Saliency Attack against SOTA black-box attacks including Boundary Attack (Brendel et al., 2018), TVDBA (Li and Chen, 2021), Parsimonious attack (Moon et al., 2019) and Square attack (Andriushchenko et al., 2020). Among them, Boundary Attack can constantly reduce the perturbation to a very small magnitude with enough queries, TVDBA tries to minimize the perturbation via integrating Structural SIMilarity (SSIM) (Wang et al., 2004) into the loss, Parsimonious Attack proposes discrete optimization to improve query efficiency, and Square Attack can achieve the current SOTA query efficiency and success rate in the black-box attack setting. In addition, we design Parsimonious-sal Attack and Square-sal Attack as baselines that restrict their perturbations in salient regions. For our Saliency Attack, the threshold $\phi$ to produce binary saliency masks is chosen to be 0.1. For splitting, original images are resized to $256\times 256$, and we set the initial block size $k_{int}$ to 16. All parameters of the compared attacks remain consistent with those recommended in their papers.

For performance metrics, we employ commonly used success rate (SR) and average number of queries (Avg. queries). To evaluate the imperceptibility of AEs, we consider $L_{0}$, $L_{2}$ and MAD. All these three imperceptibility metrics are the smaller the better. In practice, a successful AE with imperceptible perturbation is what we need. So besides $\rm SR$, we use a new metric $\rm SR_{true}$. It indicates the rate of successful AEs whose MAD values are below a human-like threshold, i.e., 30, that AEs with $\rm MAD\leq 30$ are basically imperceptible to human eyes (See Appendix A.3).

In this study, we consider $L_{\infty}$ threat model on ImageNet dataset (Deng et al., 2009), and set the perturbation magnitude $\epsilon$ to 0.05 in [0,1] scale. We use Inception v3 (Szegedy et al., 2016) as the target model, and test different query budgets {3,000, 10,000, 30,000} for untargeted attack. We provide our implementation publicly²²2https://github.com/Daizy97/SaliencyAttack.

Some examples are shown in Figure 5. We can easily find the perturbations of original Parsimonious Attack and Square Attack are very obvious in the entire image due to global perturbation, while their modified versions are relatively more imperceptible since no perturbation exists in background regions. However, their perturbations are still complex and irregular, taking up almost all salient regions. In comparison, even restricted in the same salient regions, the perturbations generated by Saliency Attack are smaller and more critical, roughly corresponding to the bright pixels in BP-saliency map. They further represent the positions of dogs’ noses or ears, which accord with our inspiration before. Besides, the AEs of Boundary Attack contain noticeable coarse textures due to inadequate query budget, and TVDBA produces global and irregular perturbations similarly.

The quantitative results are reported in Table 1. In this experiment, we randomly choose 10,000 images from the ImageNet validation set and divide them into ten groups. Then we calculate the mean and standard deviation (SD). The best results are recorded in bold based on Wilcoxon signed-rank test with significance level at 0.05. We can find in all query budgets, our Saliency Attack statistically significantly outperforms all the baselines with a huge gap in terms of $\rm SR_{true}$ and three imperceptibility metrics, which demonstrates the superiority of our method to refine the perturbations in salient regions. For baselines, although TVDBA, Parsimonious Attack and Square Attack can obtain a better $\rm SR$, their successful AEs are not truly imperceptible and will be easily detected by some defenses or humans. For Boundary Attack, it can gradually make progress on $\rm SR_{true}$, $L_{2}$ and MAD as the query budget increases, but they are still much worse than Saliency Attack. This is because it takes at least hundreds of thousands of queries for Boundary Attack to converge (Brendel et al., 2018), which is infeasible in practice. It is also obvious that Parsimonious-sal Attack and Square-sal Attack obtain better imperceptibility performance than their original version with little degradation in SR. This suggests the use of salient regions for black-box attacks is practicable.

We also plot the convergence curve of $\rm SR_{true}$ versus query number under different MAD thresholds in Figure 6. It shows that Saliency Attack can lead other attacks stably most of the time. For some baselines such as Square Attack and Parsimonious Attack, they can generate some successful AEs with very few queries due to global perturbations and some tricks like initializing perturbations with random vertical stripes (Andriushchenko et al., 2020), but they lack the ability to further improve the imperceptibility. Instead, Saliency Attack conservatively selects small regions to perturb, hence its query efficiency is a little lower than some baselines at the beginning but soon exceeds them. We further study the change of their MAD scores in Appendix A.4 and find most AEs can be improved after restricting in salient region, which also validates the effectiveness of our method.

We carry out ablation studies of Saliency Attack, including refining in salient region, in non-salient region, and without saliency (refining in the whole image). We also design a greedy search as a baseline to verify our Refine search. We test multiple block sizes for greedy search and use 32 as the best choice (see Appendix A.5). The results on 1,000 randomly selected images and some examples are given in Table 2 and Figure 7. Note that refining in salient region and refining without saliency generate the same or almost the same perturbations, which means the salient regions indeed contain useful parts and enhance the query efficiency by limiting the search space. But for refining in non-salient region, its perturbation is more complex and visible with worse query efficiency and SR due to unuseful search space. Compared with greedy search, our Refine search has much better query efficiency and $\rm SR_{true}$, which demonstrates its superiority. Therefore, we can conclude that both salient region and Refine search facilitate Saliency Attack.

The hyperparameter $k_{int}$ is the initial block size that determines the first level of split blocks. We test the effect of different $k_{int}$ on SR, query number, and imperceptibility ($L_{2}$ and MAD) over 1,000 randomly selected examples in Figure 8. As $k_{int}$ decreases, SR and imperceptibility can be improved, and SR reaches the peak when $k_{int}$ equals 16. This is because, with smaller initial blocks, Saliency Attack can search for perturbations more finely and accurately leading to higher SR and better imperceptibility. Meanwhile, inevitably more queries are needed, especially for sorting initial blocks. Under a limited query budget like 10,000, the remaining query budget for searching in finer blocks could be inadequate. That’s why a turning point occurs in SR.

We also show some examples in Figure 9. It can be observed that as $k_{int}$ decreases, the generated perturbations become smaller but more salient. For instance, in the first row, the perturbation with $k=128$ roughly covers the region of the dog face, which is also the brightest region in the BP-saliency map. While the perturbation with $k=32$ or $k=16$ focuses on smaller region of the dog ear. This indicates that our Saliency Attack could find smaller and more important perturbations progressively as the $k_{int}$ decreases, which coincides with our assumption before. The perturbations are also interpretable to some extent, like perturbing the specific regions of a dog’s ears, eyes or nose.

To further validate the imperceptibility of AEs, we attack against two detection-based defenses that attempt to detect AEs from clean images. The first is Feature Squeezing³³3Implementation of Feature Squeezing: https://github.com/mzweilin/EvadeML-Zoo (Xu et al., 2018). Its basic idea is to squeeze out unnecessary input features that may be utilized by an adversary to generate AEs. Comparing the model’s prediction on the original image with its prediction on the image after squeezing, the input is likely to be adversarial if the original and squeezed inputs produce substantially different outputs. Feature Squeezing adopts color bits reduction of each pixel and spatial smoothing (local and non-local smoothing) as squeezers. We use the recommended parameters ($5-bit,2\times 2,11-3-4$) for ImageNet dataset. Another is a CNN-based binary classifier based on a Steganography detector (Boroumand et al., 2019), which is designed for detecting small noise specifically and outperforms classic classification models such as Inception-v3 or ResNet. We finetune the pretrained model⁴⁴4https://github.com/brijeshiitg/Pytorch-implementation-of-SRNet with AEs generated by different attacks. Concretely, we randomly select 1,000 clean images and generate the corresponding AEs with Saliency Attack and six baselines, respectively. Thus, we use $2\times 7,000$ pairs of clean images and AEs as the training set to finetune the model. The training is run for 100 epochs, and the learning rate is 0.001. We test another 1,000 samples and record the detection rate (DR), which is the lower the better.

From Table 3, we can find our Saliency attack can achieve lower detection rate against both Feature Squeezing and binary classifier detection than other baselines. For Feature Squeezing, although the area of perturbation generated by Saliency Attack is quite small and thus suffering a higher risk of being denoised by squeezers, the detection rate of Saliency Attack is still better than other attacks due to our effective perturbation. For the binary classifier detector, our Saliency Attack obtains nearly $50\%$ accuracy/detection rate, which means the perturbation is invisible enough that the classifier will be difficult to converge and results in random guess. Therefore, our Saliency Attack is able to generate imperceptible AEs that evade different detection-based defenses.