Analyzing Adversarial Robustness of Deep Neural Networks in Pixel Space: a Semantic Perspective

The problem of generating adversarial examples can be formalized as an optimization problem of finding a minimum adversarial perturbation vector $\boldsymbol{r}$ with constraints. Let $\widehat{f}\colon\mathbb{R}^{n}\rightarrow\{1\textellipsis K\}$ be a classifier with the $n$-dimensional input $\boldsymbol{x}=(x_{1},\textellipsis,x_{n})\in R^{n}$ and the output $\widehat{f}(\boldsymbol{x})$. Each component of $\widehat{f}(\boldsymbol{x})$ represents the probability that the input $\boldsymbol{x}$ belongs to each of the $K$ classes and therefore $\widehat{f}_{i}(\boldsymbol{x})$ is used to denote the probability of $\boldsymbol{x}$ belongs to the class $i$. The classifier assigns the label $\widehat{y}(\boldsymbol{x})=\mathop{argmax}\widehat{f}_{i}(\boldsymbol{x})$ to the input $\boldsymbol{x}$, which means that the classifier believes that $\boldsymbol{x}$ is most likely to belong to class $\widehat{y}(\boldsymbol{x})$. The adversaries are able to launch two types of attacks, the untargeted attack and targeted attack. In the case of untargeted attack, the adversarial goal is to cause the classifier to misclassify the adversarial example $\boldsymbol{x}^{\prime}=\boldsymbol{x}+\boldsymbol{r}$ into any incorrect label $\widehat{y}(\boldsymbol{x^{\prime}})\neq\widehat{y}(\boldsymbol{x})$ with a minimal perturbation $\boldsymbol{r}$. This is equivalent to finding the optimized solution for the following question:

where $d(\cdot)$ is some distance metric to measure the similarity between $\boldsymbol{x}$ and $\boldsymbol{x^{\prime}}$. For targeted attack, the purpose of which is to make the classifier classify the adversarial input $\boldsymbol{x^{\prime}}$ as some specific target class $t$ incorrectly, the corresponding optimization problem can be formalized as follows:

The most widely-used distance metrics to quantify similarity for generating adversarial examples are $L_{0}$, $L_{2}$ and $L_{\infty}$ norms. In this paper, we use $L_{0}$ norm as all the literature in few-pixel perturbations did. To be more specific, we only modify one pixel and perform three-pixel and five-pixel attack as a comparison. The reason for doing this is because in order to more accurately compare the attack effect of adversarial attacks in different areas of the image, we have to strictly limit the location of the perturbation. Besides, there are many possible ways to express equation 1 and 2 in a different form that is more suitable for optimization. Therefore, we apply another alternative formulation that is very commonly used and more intuitive for untargeted attack:

where $y$ represents the true class predicted by the classifier before being attacked, $l$ constrains the strength of the modification and for $L_{0}$ norm it denotes the number of pixels that are allowed to be modified. In the case of targeted attack, use $t$ denotes the target class that the adversary wants the classifier to misclassify as, the equation is as follows:

Previous works commonly search for the direction and the size of the modification over the entire input space while our approach partition the input space according to semantic information before generating adversarial perturbations. It means that the perturbation vector $\boldsymbol{r}$ only modify $l$ directions of $m$ ($m\textless n$) dimensions of the input $x$ with the other dimensions of $\boldsymbol{r}$ left to zeros.

GrabCut [36] is a classic method that based on Graph Cut [37] for interactive image segmentation. GrabCut combines both texture (colour) information and edge (contrast) information, it can reduce the user interaction required to complete the segmentation while extracting foreground of good quality. The method firstly obtains a hard segmentation using iterative graph-cut optimisation and then uses border matting to deal with the problem of transparency around the object boundaries. In specific, GrabCut models the interactive segmentation task as a graph cut optimization problem just like other approached based on graphical models did. Let $\alpha=(\alpha_{1},\textellipsis,\alpha_{N})$ be an array of opacity values at each pixel to express the segmentation of the image ($0\leq\alpha_{n}\leq 1$ ordinarily, or $\alpha_{n}\in\{0,1\}$ for hard segmentation), $\theta$ be parameters that describe image foreground and background distributions. The segmentation problem can be regarded as equivalent to inferring the unknown opacity variables $\alpha$ from the given image data $\boldsymbol{z}=(z_{1},\textellipsis,z_{N})$ and $\theta$. The opacity value $\alpha$ can be estimated as a global minimum of the following energy function $\boldsymbol{E}$ that is defined so that its minimum corresponding to a good segmentation.

In order to apply to colour image, GrabCut uses color Gaussian Mixture Model (GMM) models in the data term $U$, as

where $\boldsymbol{k}$ is a vector of the GMM component variables used to assign a unique GMM component to each pixel either from the foreground or the background model. Let $p(\cdot)$ be a Gaussian probability distribution and $\pi(\cdot)$ be the mixture weighting coefficients, so the equation 6 can be rewritten as

$V$ is the smoothness term and its contrast term is calculated based on Euclidean distance in colour space, as

Then the energy function $\boldsymbol{E}$ is minimized iteratively to obtain a good hard segmentation. After that, by fitting a polyline to the hard segmentation boundary, a closed contour $C$ is obtained. Now $\alpha_{n}$ ($n\in T_{U}$)is computed based on a new trimap $\{T_{B},T_{U},T_{F}\}$, where $T_{U}$ is the set of pixels that located in a ribbon of width $\pm w$ ($w$ is an empirically chosen parameter) pixels either side of $C$. This process is named border matting and GrabCut does this with regularisation and a dynamic programming (DP) algorithm. Foreground pixel stealing is also included to better estimate foreground pixel colours.

U²-Net [38] is a deep learning based method to segment the most visually attractive objects in an image, it belongs to Salient Object Detection (SOD) which is widely used in image segmentation. U²-Net uses a two-level nested U-structure architecture designed for SOD, which enables high resolution feature maps to be maintained at a low memory and computing cost. It nests two repeated U-Net modules instead of cascaded stacking them and comes up with a novel Residual U-block named RSU for capturing intra-stage multi-scale features. Thus the method is very flexible and easy to be adapted to different tasks with little performance loss. The three main components of U²-Net are a six stages encoder, a five stages decoder and a saliency map fusion module which is attached with the decoder stages and the last encoder stage. Besides, U²-Net use deep supervision in the training process and the training loss as follows:

here $l_{side}^{m}$ and $l_{fuse}$ are two loss terms, where $l_{side}^{m}$ is the loss of the side output saliency map and $l_{fuse}$ is the loss of the final fusion output saliency map. $w_{side}^{m}$ and $w_{fuse}$ are the wights of $l_{side}^{m}$ and $l_{fuse}$ respectively. Each loss term is calculated using the standard binary cross-entropy.

In this paper, GrabCut is used to segment the foreground and the background of CIFAR-10 images, and U²-Net is used for ImageNet. U²-Net works in a non-interactive way and it can be used to segment the ImageNet images very efficiently. On the other hand, the images of CIFAR-10 are low-resolution and the foreground and background colour distributions of which are not well separated. The distribution of these images is very different from the distribution of images used in the training of the deep learning based segmentation models, which makes the deep learning based method (such as U²-Net) perform very poorly on CIFAR-10. GrabCut can obtain segmentations of good quality on CIFAR-10 with less user effort among the existed classical methods.

When we discuss the problem of generating adversarial examples from the perspective of optimization, there are some algorithms to choose from, and the solution to optimization problems is different for different methods. Since it becomes more difficult to use gradient information when we calculate the adversarial perturbation on a part of the image (foreground or background) rather than the whole, the method that does not use the gradient information is a reasonable option. Differential Evolution (DE) is one such approach, a very powerful derivative-free optimization algorithm for black-box optimization, which is about finding the minimum of a function without knowing its analytical form. DE does not require the objective function of the optimization problem to be differentiable, and its population selection mechanism preserves diversity so that it may be able to find better solutions than gradient-based methods or even other kinds of evolutionary algorithms more efficiently. These properties make DE can be used to solve complex multi-modal optimization problems, and very suitable for generating adversarial images where the networks are sometimes highly complex and non-differentiable or calculating gradient can be hardly realistic. In addition, DE was also used to generating one pixel adversarial perturbations for the whole image in [14], this gives a fairer comparison of the results of our attacks that modifies one pixel in a part of the image.

When minimizing a function $f(x)\colon\mathbb{R}^{n}\rightarrow\mathbb{R}$, DE firstly represents the solutions as population of $P$ individuals and initializes them, where each individual is a parameter vector of $f(x)$. For each generation $G$, a population can be represented as

Then an operation called mutation is used to generate new parameter vectors. A mutant vector is generated by adding the weighted difference between two different population vectors $\mathit{x}_{\mathit{r}_{2},G}$ and $\mathit{x}_{\mathit{r}_{3},G}$ to a third vector $\mathit{x}_{\mathit{r}_{1},G}$ as

where $F\in\left[0,2\right]$ is called mutation factor, $\mathit{r}_{1}\neq\mathit{r}_{2}\neq\mathit{r}_{3}\neq\mathit{i}$ and $\mathit{r}_{1},\mathit{r}_{2},\mathit{r}_{3},\mathit{i}\in\left\{1,2,\textellipsis,P\right\}$ are randomly chosen. The next step is called recombination, which aims to increase the diversity of the population. In this step, DE create a trial vector by mixing the information of the mutant with the information of the current population vector as

where $D$ represents the dimension of the population vector and each component in $\mathit{u}_{\mathit{i},G+1}$ can be decided by a method called crossover as follows:

In (13), $CR\in\left[0,1\right]$ is called crossover constant, $rnbr(i)\in 1,2,\textellipsis,D$, and $randb(j)$ is the $j$th evaluation of a random number generator (different distribution followed corresponding to different crossover variations). Finally, DE compare the trial vector $\mathit{u}_{\mathit{i},G+1}$ with the current population vector $\mathit{x}_{\mathit{i},G}$. If $\mathit{u}_{\mathit{i},G+1}$ is better measured by the fitness function $f(x)$, $\mathit{x}_{\mathit{i},G}$ is replaced by $\mathit{u}_{\mathit{i},G+1}$; otherwise, $\mathit{x}_{\mathit{i},G}$ is preserved and $\mathit{u}_{\mathit{i},G+1}$ discarded. The whole population will eventually converge towards the solution after many iterations.

Let $C(\boldsymbol{x})$ be the set of pixels of an entire image $\boldsymbol{x}$, we first divide $C(\boldsymbol{x})$ into two sets $C_{F}(\boldsymbol{x})$ and $C_{B}(\boldsymbol{x})$ that satisfied

where $C_{F}(\boldsymbol{x})$ consists of pixels belong to the foreground of the image $\boldsymbol{x}$ and $C_{B}(\boldsymbol{x})$ is the set of pixels in the background. When the adversarial perturbation is optimized by DE, each candidate solution is encoded as a tuple containing five elements: $(x,y,r,g,b)$. $x\raisebox{0.0pt}{-}y$ denotes the coordinates of the modified pixel and $(r,g,b)$ is the RGB value of the perturbation. In the standard DE algorithm, the individuals in the population are coded as continuous real numbers, which were originally used to solve optimization problems for continuous function. Each of these five elements can be regarded as taking values in a continuous interval when searching for the perturbation on the entire image. However, when the perturbation is optimized in the space corresponding to the the foreground or the background pixels, the interval of the coordinates $x\raisebox{0.0pt}{-}y$ is no longer continuous. Some methods encode individuals as discrete sequences of length $n$

and generate mutant individuals by randomly inserting and moving the sequences. The method proposed in this paper is somewhat different for that:

This means that the coordinates $x\raisebox{0.0pt}{-}y$ in the foreground are neither discrete nor take values on a continuous interval, but take values on multiple intervals (the same is true for the background). For simplicity, we solve this problem by making the optimization algorithm automatically constrain the position of the coordinates, instead of modifying the DE algorithm itself. Specifically, when we attack an image on its foreground, we first execute the DE algorithm for the first generation on the entire image $\boldsymbol{x}$ to get the trial vector $\mathit{u}_{\mathit{i},2}$ and the current population vector $\mathit{x}_{\mathit{i},1}$ in Section III-C. Then we use $\mathit{u}_{\mathit{i},2}$ and $\mathit{x}_{\mathit{i},1}$ respectively to modify the image $\boldsymbol{x}$, and get the modified images $\boldsymbol{x}{{}_{u}}^{{}^{\prime}}$ and $\boldsymbol{x}{{}_{x}}^{{}^{\prime}}$, just like the attack on the entire image. The difference is that when we compare the vector $\mathit{u}_{\mathit{i},2}$ and $\mathit{x}_{\mathit{i},1}$, the fitness function is computed on a new image $\bar{\boldsymbol{x}}$ and the set of pixels that make up the image as follows:

where $\boldsymbol{x}^{{}^{\prime}}$ represents the modified images $\boldsymbol{x}{{}_{u}}^{{}^{\prime}}$ or $\boldsymbol{x}{{}_{x}}^{{}^{\prime}}$. Similarly, when attacking on the background,

Thus, at the replacement stage of each generation, we evaluate the vector $\bar{\boldsymbol{x}}$ with the fitness function. This can ensure that the adversarial perturbation is in the segmented foreground or background, and force the DE algorithm to find the optimal solution in the corresponding area.

It is worth noting that when we looking for perturbations in the foreground (or background) we also used pixels of the background (or foreground) at the beginning, but from the second generation the DE algorithm actually only looks for the solution in the foreground (or background) instead of the entire image. This can be confirmed by observing the change of the value of the fitness function over the generations during evolution, because when attacking on the foreground (or background), if the perturbed pixel is located in the background (or foreground), the value of the fitness function will not change at all in our method, but the actual experimental results show that the value of the fitness function drops rapidly from the beginning. Here we only show the feasibility of using only part of the pixels (foreground or background pixels) of an image for a successful one-pixel attack and leave the specific implementation for future work.

In this section, we first introduce the datasets and the deep learning models used for the experimental investigation. The details of the experiment are described afterwards, and the experimental results are analyzed in depth at the end of this section.

Dataset description. The experiments were performed on two datasets of different scale, CIFAR-10 and ImageNet, both of which are widely used. The two datasets contain a different number of color images and each pixel of these images takes the value of a real number between $0$ and $255$ for three color channels. Specifically, the CIFAR-10 dataset is a collection of tiny images in $10$ classes which contains $50,000$ training images and $10,000$ test images, each with a resolution of $32\times 32$. The ImageNet project is a large visual database and ILSVRC 2012 [39], which uses a subset of ImageNet with $1.2$ million training images, $50,000$ validation images and $150,000$ test images in $1000$ categories, is one of the most commonly used version of ImageNet. For each of the attacks on the three types of deep neural networks trained on CIFAR-10, we randomly selected $500$ images from CIFAR-10 test dataset. Analogously, for ImageNet, $420$ images from ILSVRC 2012 test set were randomly selected for each of the attacks. Besides, we also resized the ImageNet images to $224\times 224$ resolution since the original ILSVRC 2012 images are not uniform in size.

Architecture characteristics. In order to conduct the adversarial attacks, we trained three of common networks on CIFAR-10: all convolutional network (AllConv) [40]; network in network (NiN) [41] and VGG-16 network [42]. And we also trained three networks on ILSVRC 2012: BVLC AlexNet [43]; Residual Network with 50 layers (ResNet-50) [44] and VGG-16. We trained the network VGG-16 on both the two datasets for comparing the performance of the same architecture on different data sets. To distinguish between the two, we use VGG-16 (C) represent the VGG-16 network trained on CIFAR-10 and use VGG-16 (I) represent the network trained on ImageNet in this paper. We will also omit the letters in parentheses when there is no ambiguity. The architectures of the networks follow those described in the original works. The baseline accuracy of all networks on clean test sets is shown by Table I which is comparable to state-of-the-art performance.

CIFAR-10. We launched untargeted adversarial attacks and targeted attacks on $500$ randomly selected CIFAR-10 test images for the three CIFAR-10 neural networks. For untargeted attacks, if an image can be perturbed to be misclassified as any other class, the untargeted attack on this image succeeds. For targeted attacks, each image is perturbed to all other nine CIFAR-10 classes separately, and we count it as a successful targeted attack only if the image is perturbed to the specified target class. That is, when an image classified as class $A$ without being attacked is to be perturbed to a target class $B$, but it is finally misclassified as another target class $C\neq B\neq A$, this cannot be counted as a successful targeted attack.

Then we segmented the foreground and the background of these test images use GrabCut. The interaction with GrabCut (the amount of which is usually small) stops until a satisfactory result for human eyes is obtained. The untargeted and targeted adversarial attacks using the same technique were conducted in the foreground and in the background. In order to compare the effects of the adversarial attack in the foreground and the background, it is necessary to strictly control the position where the adversarial perturbation is added. We thus performed the attacks in an extremely limited scenario where the perturbation only can be added to one pixel. In addition, we also conducted the experiments by perturbing the images with three and five pixel-modification in the original natural images as well as their foreground and background on all the three CIFAR-10 networks. The purpose of this is to explore the situation of different perturbation scale.

The perturbation used to modify the pixel (or pixels) is computed by the population-based optimization algorithm DE. The initial number of population is set to $400$ and a stratified sampling method called latin hypercube sampling (LHS) is used to initialize the initial population to maximize coverage of the available parameter space. The maximum number of iteration is set to $100$ and early-stop criterion will be triggered when the predicted label is no longer the true class in the case of untargeted attacks, and when the predicted label is the target class in the case of targeted attacks. We also employ dithering when set the value of mutation factor $F$ to help speed convergence significantly, it means $F$ takes a uniform random number between $[0.5,1)$ in each iteration. The binomial crossover is included. The fitness function is the confidence value of true class for untargeted attack and the confidence value of target class for targeted attack.

ImageNet. We only performed one-pixel untargeted attacks on three ImageNet neural networks with the same DE parameter settings considering the time constraints. Although the search space of ImageNet is much larger than that of CIFAR-10 (approximately $50$ times larger), it is enough to see a clear trend from the experimental results even without proportionally increasing the number of evaluations. This low computational cost setting corresponds to a computationally tractable attack method on ImageNet, which is more realistic. Of cause, we also believe that more evaluations may result in a higher attack success rate.

In addition, the most different between the experiment on ImageNet and CIGAR-10 is that we used a different method called U²-Net to segment the foreground and background from ImageNet natural images. The setting of U²-Net was kept as similar as possible to the original with a few modifications to adapt to changes in input size. To ensure that the foreground and background of each image used for attack are properly segmented, we finally performed a manual inspection as an additional guarantee.

Some examples of perturbed images are visualized in Fig. 1 and Fig. 2. It can be clearly seen that vulnerable points in the pixel space corresponding to an image are diverse and not unique. For the same architecture trained on the same data set, the perturbed images obtained by modifying one pixel in the foreground and modifying one pixel in the background may be misclassified by the model into the same class (see the images with labels in blue in Fig. 1, or (a-1) and (b-1), (c-1) and (d-1), (a-2) and (b-2), (c-3) and (d-3), etc in Fig. 2) or different classes (images with labels in red in Fig. 1, (e-1) and (f-1) in Fig. 2). For models with different architectures trained on the same data set, there is more than one candidate pixel both in the foreground and background when performing the one-pixel attack, which makes different models incorrectly classify the perturbed images into the same class (see (a-1) and (a-2), (b-1) and (b-2), (c-1) and (c-3), (d-1) and (d-3) in Fig. 2) or different classes ((g-1) and (g-3) in Fig. 2)).

Success rate. In order to evaluate the attack performance on the foreground and background of the image, we introduce several metrics to measure the effectiveness of the attacks, with only a few differences for CIFAR-10 and ImageNet datasets. Specifically, we report the success rate of untargeted attacks and targeted attacks as well as their corresponding confidence for CIFAR-10 in Table II. The success rate is defined as the percentage of adversarial examples that are successfully misclassified as any incorrect class label in the case of untargeted attacks. And in the case of targeted attacks, it is defined as the probability of adversarial examples being misclassified into a specific target class. In addition to the success rate calculated based on the actual untargeted and targeted attack results, we also report the untargeted success rate evaluated based on targeted attack results. That is to say, if the targeted attack can be launched successfully on an image at least for one target class, the untargeted attack on this image is considered successful. When calculating the confidence value, we accumulate the values of probability of the predicted class after each successful attack, then divided by the number of successful attacks. For ImageNet, we report the success rate and the corresponding confidence of top1 predicted class after being attacked in the case of untargeted attack. Besides, since we focus on decreasing the probability of the original true class, in order to further investigate the effect of the attacks, the average decrease in confidence of the original top1, top3, top5 classes before and after being attacked as well as the probability that the original top1, top3, top5 classes is still in top3, top5 prediction after being attacked are also shown in Table III.

On CIFAR-10, the results of one-pixel untargeted attacks and targeted attacks both show that the attack performance on the foreground of the image is comparable to the attack effect on the original entire image, while the attack success rate on the background is significantly lower. For example, there is $43.40\%$ chance that the one-pixel untargeted attack on an arbitrary CIFAR-10 original test image can cause the image to be misclassified for model VGG-16 (C), $41.60\%$ chance that on the foreground of the image the attack can success, while the success rate drops to $18.20\%$ for the attack on the background of the image. For all the three CIFAR-10 models, the difference between the success rate on the original image and the success rate on its background is much larger than the difference between the success rate on the original image and the success rate on its foreground. The success rate of the three models on the foreground of the image reduces by an average of $6.84\%$ in the case of untargeted attacks and by an average of $2.86\%$ in the case of targeted attacks compared with the success rate on the original image. In contrast, the success rate on the background of the image decreases by an average of $54.32\%$ for untargeted attacks and $51.56\%$ for targeted attacks respectively compared with the success rate on the original image. Moreover, for stronger attacks obtained by increasing the number of pixels that can be modified to three and five, the value of success rate increases significantly, while the observation about the success rate of the attacks on the original image compared with its foreground and background has not changed. Besides, for a successful untargeted attack or targeted attack, attacking on the original image or its foreground or background does not significantly change the confidence value corresponding to the predicted class after being attacked, nor does the confidence value change with the number of pixels that can be modified.

On ImageNet, we observed that the success rate of the three models on the foreground of the image decreases by an average of only $5.86\%$ compared with the success rate on the original image, while the success rate on the background reduces by an average of $58.04\%$ compared with on the original image, which is almost 10-fold greater. Regardless of whether the modified pixels are located in the entire image or its foreground or the background, the confidence value of the predicted class after being attacked successfully does not change much. These results are in line with our previous finding of CIFAR-10 and show that the observation generalizes well to large size images. For successful attacks, the average of the reduction in confidence corresponding to the original true top1 class of the three ImageNet models are, $15.51\%$ for attacks on the original image, $15.21\%$ for attacks on the foreground of the image and $8.92\%$ for attacks on the background respectively. Even for the unsuccessful attacks, the drop in confidence values corresponding to the true top3 and top5 classes after being attacked on the background of the image is smaller than that after being attacked on the entire image and its foreground for all the three models. All these results tell us that attacking on the background of the image is much less effective than on the entire image or its foreground. But it should also be noted that, although the average probability that the original top3 or top5 classes are still in the top3 or top5 predictions is highest after being unsuccessfully attacked on the background than on the foreground and the entire image, the original top3 or top5 classes are still in the top3 or top5 predictions in more than $90\%$ of the attacks for all the three models. If the attacks are launched successfully, the probability that the original top1 class is still in top3 or top5 predictions is almost $100\%$. This small change in the rank order of the predicted classes is caused by the fact that we only utilized a simple implementation to modify the pixel of the images for such a large scale dataset. More calculations, other setting of parameters and fitness functions should give different results.

Number of target classes (successful targeted attack results). The number of target classes that the percentage of the images can be perturbed to are shown in Fig. 3 where one-, three-, five-pixels are modified on the entire image, its foreground and background respectively. This metric is only included in the CIFAR-10 results since there are too many classes of ImageNet classification up to $1000$ classes. It can be seen that with only one-pixel modification on the entire image, a certain amount of images can be misclassified up to three target classes by the AllConv network and NiN network, and a small number of images can even be perturbed to six target classes by VGG-16 (C) network. The results of the number of target classes when attacking on the entire image are almost as good as the results when attacking on the foreground of the image, whereas the results of the attacks that modify the pixel on the background of the image are much worse.

Perturbing the images to more target classes can be achieved when increasing the number of pixels that can be modified. At the same time, there is still a large gap between the attack effect on the background of the image and that in the entire image or its foreground. These results suggest that the parts of an image that contain different amounts of semantic information are vulnerable to adversarial attacks to different degrees.

Original-target class pairs. We display the number of the successful untargeted and targeted one-pixel attacks corresponding to different original-target class pairs in Fig. 4 and Fig. 5. The heat-maps in Fig. 4 and Fig. 5 are based on the results of the three CIFAR-10 networks when attacking on the original entire image as well as on its foreground and background. It can be observed that the results of untargeted and targeted attacks show no significant difference in patterns. When attacking on the entire image, the degree of vulnerability varies for different original-target class pairs and there are some specific original-target class pairs which are much more vulnerable than others. For example, images of class $5$ can be much more easily misclassified to class $3$ but can hardly be perturbed to class $8$ for the targeted attacks on the VGG-16 (C) network. Most of these specific class pairs are still more vulnerable than others to attacks on the foreground of the image, but are no longer significantly different from other class pairs when attacking on the background. It suggests that for a single image, the vulnerable directions are more likely found in the space corresponding to the part of the image that contains more semantic information, such as the foreground.

In addition, it is more difficult to be attacked successfully for the images of some classes than of others (such as class $9$ for AllConv and NiN, class $1$ for VGG-16 (C)), and some classes are harder to reach (such as class $2$ for AllConv, class $9$ for NiN, class $8$ for VGG-16 (C)). For the classes that are hard to be attacked and to be perturbed to, the difficulty is further increased when attacking on the background of the image. This indicates that the vulnerable directions shared by different data points that belong to the same class are also closely related to the amount of semantic information. It is more difficult to search for the vulnerable directions on the background, which contains less semantic information than the foreground of the image. Therefore, when the data points going across the input space through the direction parallel to any of the directions corresponding to the pixels in the background, the original classes are more likely to keep robust along these directions. Meanwhile, for the classes that are already hard to reach, finding a reachable direction in the space corresponding to the background will be even more difficult. This phenomenon might be due to the shape of the decision boundary and the relative angle between each direction of the data point and the boundary. It means that, for a wide boundary that the data points may be far away from the boundary, the distance from the boundary may be further along the directions in the space corresponding to the background. On the other hand, if the boundary shape is thin, it is more difficult to reach along the direction corresponding to the pixel with less semantic information.

Change in fitness values and number of generations. In order to observe more clearly what happened during the attack, how the fitness values change during the evolution of different networks and different datasets in the case of untargeted attack is illustrated in Fig 6. More specifically, we randomly selected $15$ images that can be successfully attacked not only on the original entire image but also on its foreground and background for all the three CIFAR-10 networks: AllConv, NiN, VGG-16(C), and selected $10$ ImageNet images for the model AlexNet. Remember that the fitness value is set to be the confidence of the true class of the image, and the goal of the untargeted attack is to minimize this fitness value, as previously mentioned. We also report the average number of generations required for all images successfully attacked on the original image, its foreground and background for all the CIFAR-10 and ImageNet networks in Table IV.

As shown in Fig 6, the fitness values can drop off a cliff between certain two generations and decrease steadily at other generations. For the sample that can be successfully attacked both on the original entire image and its foreground and background, the number of generations required to drop the fitness value sufficiently to make the image be misclassified is much less than $100$. The average of fitness decreases monotonically with the number of generations, moreover, the average fitness value drops almost as much for attacking on the foreground as it does for the entire image. However, when attacking on the background of the image, the average fitness value drop is much smaller. The results in Table IV also show that for all the networks on CIFAR-10 and ImageNet, the average number of generations that required for the successful untargeted attack by modifying one pixel on the background of the image is significantly higher than on the entire image and its foreground. Besides, it can be find that the fitness value decrease less for the AlexNet network compared with the networks on CIFAR-10. This means that under the current settings, the AlexNet network is more difficult to fool.