Interpreting Attributions and Interactions of Adversarial Attacks

Given an input image, the regional attribution measures the importance of each image region to the decrease of the attacking cost $\|\delta\|_{p}$. [27] has discussed that the commonly used gradient w.r.t. the attacking loss [41] cannot objectively reflect the regional attribution due the highly non-linear representation of the DNN. Please see section A in supplementary materials for more discussions. Considering the high computational burden, we investigate the regional attribution instead of the pixel-wise attribution. We divide the entire image into $L\times L$ grids (regions), denoted by $\Lambda=\{\Lambda_{11},\Lambda_{12},\dots,\Lambda_{LL}\}$. Each region $\Lambda_{uv}\in\Lambda$ ($1\leq u,v\leq L$) is a set of pixels. $\phi_{uv}$ denotes the attribution of the region $\Lambda_{uv}$. The total attribution to the decrease of the attacking cost is allocated to each region:

$$\phi_{11}+\phi_{12}+\dots+\phi_{LL}=\text{cost}(\Lambda)-\text{cost}(\emptyset)$$

(3)

where $\text{cost}(\Lambda)=\|\delta^{(\Lambda)}\|_{p}$ and $\text{cost}(\emptyset)=\|\delta^{(\emptyset)}\|_{p}$. $\delta^{(\Lambda)}$ represents the adversarial perturbation generated by allowing all pixels to be perturbed. $\delta^{(\emptyset)}$ represents the adversarial perturbation generated without perturbing any pixels in $\Lambda$. Note that it is impossible to conduct adversarial attacks, when all pixels in the image are not allowed to be perturbed. Thus, as Fig. 2 (a) shows, we approximate $\phi_{uv}$ by extending the input image to $(1+\beta)\cdot width\times(1+\beta)\cdot height$, where $\beta$ is a small scalar constant. We regard $\delta^{(\emptyset)}$ as the adversarial perturbation generated when only the extended regions are perturbed. We compute such regional attribution as the Shapley value [40].

$$\phi_{uv}\!=\!\frac{1}{L^{2}}\!\!\!\!\sum_{S\subseteq\Lambda\setminus\{\Lambda_{uv}\}}\!\!\!\binom{L^{2}-1}{|S|}^{-1}\!\!\!\!\!\!\Big{[}\text{cost}{(S\cup\{\Lambda_{uv}\})}\!-\!\text{cost}{(S)}\Big{]}$$

(4)

where $S$ denotes a set of regions excluding $\Lambda_{uv}$. The $\text{cost}(S)$ is the $L_{p}$ norm ($p=2$, or $+\infty$) of the adversarial perturbation generated when only regions in $S$ are allowed to be perturbed during attacking. We formulate such an attack using masking operation, $\hat{\delta}=\arg\min_{\delta}\;\|\delta\circ M^{(S)}\|_{p}+c\cdot f(x+\delta\circ M^{(S)})\ \text{s.t.}\ x+\delta\circ M^{(S)}\in[0,1]^{n}$, where $\circ$ represents the element-wise multiplication. $M^{(S)}$ is a mask, which satisfies $\forall\Lambda_{uv}\in S,\,\forall i\in\Lambda_{uv},\,M^{(S)}_{i}=1$, and for all other pixels $i$, $M^{(S)}_{i}=0$. $\delta^{(S)}=\hat{\delta}\,\circ\,M^{(S)}$ is the adversarial perturbation generated when only regions in $S$ are allowed to be perturbed, thereby $\text{cost}(S)=\|\delta^{(S)}\|_{p}$.

Based on Equation (4), in order to quantify the regional attribution, we sample different $M^{(S)}$ to conduct adversarial attacks. Note that the adversarial attack with masking operation is conducted to compute $\text{cost}{(S)}$ and $\text{cost}{(S\cup\{\Lambda_{uv}\})}$, instead of obtaining more robust perturbations.

In this section, given a perturbation map of an input image, we aim to decompose the perturbation map into several relatively independent image regions (perturbation components), in order to explore the true pixel-wise collaborative behavior behind of the adversarial attack. Note that the collaborative behavior itself may not be adversarial robust. Given the set of all perturbation pixels $\Omega$, we aim to extract $m$ components, i.e. $\{C^{(1)},C^{(2)},\dots,C^{(m)}\}$, where $\forall i,C^{(i)}\subseteq\Omega$; $\forall i,j,i\neq j,C^{(i)}\cap C^{(j)}=\emptyset$, and $C^{(1)}\cup C^{(2)}\cup\dots C^{(m)}\subseteq\Omega$. Perturbation pixels within each component are supposed to have strong interactions, while perturbation pixels in different components are supposed to have weak interactions. Thus, each component can be roughly considered independent in the adversarial attack.

The interaction among perturbation pixels reflect cooperative or adversarial relationships of perturbation pixels in attacking. Inspired by [30], we define interactions based on the game theory. Let us consider a game with $n$ players $\Omega^{\prime}=\{1,2,\dots,n\}$, where players aim to gain a reward. We use $2^{\Omega^{\prime}}$ to denote all potential subsets of $\Omega^{\prime}$. For example, if $\Omega^{\prime}=\{a,b\}$, then $2^{\Omega^{\prime}}=\{\emptyset,\{a\},\{b\},\{a,b\}\}$. The reward of the game $z:2^{\Omega^{\prime}}\rightarrow\mathbb{R}$ maps a set of players to a scalar value. Here, given an adversarial example $x^{\prime}=x+\delta\in\mathbb{R}^{n}$ and a DNN $g(\cdot):\mathbb{R}^{n}\rightarrow\mathbb{R}^{T}$¹¹1$g(\cdot)$ denotes the DNN’s output scores before the softmax layer., where $T$ is the number of categories, we regard each perturbation pixel in $\delta$ as a player. The goal of perturbation pixels is to decrease the score of the true category $g_{\ell}(x^{\prime})$ and increase the score of the target category $g_{t}(x^{\prime})$. Given a set of perturbation pixels $S\subseteq\Omega$, we formulate the reward of perturbation pixels in $S$ as $z(S)=g_{t}(x+\delta\circ M^{(S)})-g_{\ell}(x+\delta\circ M^{(S)})$, by assigning values of perturbation pixels in $\Omega\setminus S$ to zero, where $M^{(S)}$ is a mask $\forall i\in S,M^{(S)}_{i}=1;\forall i\in\Omega\setminus S,M^{(S)}_{i}=0$.

The total reward in the game is $\Phi=z(\Omega)-z(\emptyset)$, where $z({\Omega})=g_{t}(x+\delta\circ{\bf{1}})-g_{\ell}(x+\delta\circ{\bf{1}})$ is the reward obtained by all perturbation pixels, and $z(\emptyset)=g_{t}(x+\delta\circ{\bf{0}})-g_{\ell}(x+\delta\circ{\bf{0}})$ is the baseline reward w.r.t. the original image. The total reward can be allocated to each perturbation pixel, $\Phi=\phi_{1}+\phi_{2}+\dots+\phi_{n}$ as the Shapley value. $\phi_{i}$ is referred to as the reward of the perturbation pixel $i$.

We notice that perturbation pixels do not contribute to the attack independently. Instead, some perturbation pixels may form a specific component. In this way, the reward gained by the component is different from the sum of the reward of each perturbation pixel when they contribute to the attack individually. Here, we can consider this component as a singleton player. Thus, the total reward $\Phi$ is allocated to $n-|S|+1$ players in the new game, i.e. $\Phi=\phi^{\prime}_{S}+\sum_{i\in N\setminus S}\phi^{\prime}_{i}$. In this way, the interaction among players in $S$ is defined as

$$I[S]=\phi^{\prime}_{S}-{\textstyle\sum}_{i\in S}\,\phi_{i}$$

(5)

where $\phi_{S}^{\prime}$ is the allocated reward when $S$ is taken as a singleton player, and $\phi_{i}$ is the reward when we consider the perturbation pixel $i$ contributes to the attack independently.

We compute $\phi_{i}$ and $\phi^{\prime}_{S}$ as Shapley values, which will be given in Equation (6). Fig. 2 (b) shows a toy example for the interaction. The total reward is allocated to each perturbation pixel, i.e. $\Phi=\phi_{a}+\phi_{b}+\dots+\phi_{h}+\phi_{i}$. If pixels $a,b,d,e$ are regarded as a singleton player, then the allocation of $\Phi$ would change to $\Phi=\phi^{\prime}_{\{a,b,d,e\}}+\phi^{\prime}_{c}\dots+\phi^{\prime}_{h}+\phi^{\prime}_{i}$. If $\phi^{\prime}_{\{a,b,d,e\}}-(\phi_{a}+\phi_{b}+\phi_{d}+\phi_{e})\neq 0$, then pixels $a,b,d,e$ are considered to have interactions.

Understanding of the interaction: If $I[S]>0$, it means perturbation pixels in $S$ cooperate with each other to change prediction scores of the DNN. If $I[S]<0$, it means perturbation pixels in $S$ conflict with each other. The absolute value $|I[S]|$ indicates the strength of the interaction.

Computation of the reward: According to Equation (1), $\phi_{i}$ and $\phi^{\prime}_{S}$ are computed as follows.

$$\begin{split}\phi_{i}&=\frac{1}{n}\sum_{\tilde{S}\subseteq\Omega\setminus\{i\}}\binom{n-1}{|\tilde{S}|}^{-1}\Big{[}z(\tilde{S}\cup\{i\})-z(\tilde{S})\Big{]}\\ \phi^{\prime}_{S}&=\frac{1}{n^{\prime}}\sum_{\tilde{S}\subseteq\Omega\setminus S}\binom{n^{\prime}-1}{|\tilde{S}|}^{-1}\Big{[}z(\tilde{S}\cup S)-z(\tilde{S})\Big{]}\end{split}$$

(6)

where $n^{\prime}=n-\left|S\right|+1$. In this study, we propose an efficient method to approximate $\phi_{i}$ and $\phi^{\prime}_{S}$, which will be introduced in Equation (7) and Equation (8).

Extraction of perturbation components: We extract perturbation components via hierarchical clustering, in which perturbation pixels have strong interactions. The pseudo code of extracting perturbation components is shown in section C in supplementary materials. In the first step of clustering, we merge $q$ neighboring perturbation pixels with strong interactions into a $q$-pixel component. In the second step, we merge $q$ neighboring components with strong interactions into a component of $q^{2}$ pixels. In this way, we iteratively generate components of $q$, $q^{2}$, $q^{3}$ pixels, etc. A toy example is shown in Fig. 2 (c). We use $C^{(i)}$ to denote a component. The finally merged component is selected from a set of component candidates, each of which is a set of $q$ neighboring components, $S^{c}=C^{(i_{1})}\cup C^{(i_{2})}\cup\dots\cup C^{(i_{q})}$. If $\left|I[S^{c}]\right|>\gamma$, then we merge $C^{(i_{1})},C^{(i_{2})},\dots,C^{(i_{q})}$ into a large component. Considering the local property [6], we only compute interactions among neighboring pixels/components.

Efficient approximation of rewards of perturbation pixels: We can consider the value of each perturbation pixel $i$ as the sum of values of $K$ sub-pixels, denoted by $(i,k)$, $1\leq k\leq K$. The values of the $K$ sub-pixels are uniformly divided, i.e. $\delta/K=\delta_{(i,1)}=\delta_{(i,2)}=\dots=\delta_{(i,K)}$. In this way, the reward $\phi_{i}$ can be approximated as the sum of sub-pixels’ rewards, i.e. $\phi_{i}\approx\sum_{k=1}^{K}\phi_{(i,k)}$. Consider the symmetry property of the Shapley value [30], $\phi_{(i,1)}=\phi_{(i,2)}=\dots=\phi_{(i,K)}$. Thus, we can approximate $\phi_{i}\approx K\cdot\phi_{(i,k)}$. Please see section B.1 in supplementary material for the proof and more discussions. The reward $\phi_{(i,k)}$ can be approximated as follows. Let us consider the marginal reward for computing the Shapley value of the sub-pixel, i.e. $z{(S\cup\{(i,k)\})}-z{(S)}$. Given a large value of $K$, the perturbation magnitude of each sub-pixel $|\delta_{(i,k)}|$ is fairly small. The marginal reward can be approximated as $z{(S\cup\{(i,k)\})}-z{(S)}=\delta_{(i,k)}\cdot\partial{z(S)}/\partial{\delta_{(i,k)}}+{o(\delta_{(i,k)})}$, based on the Taylor expansion. In this way, the Shapley value of each sub-pixel is given as Equation (7).

$$\phi_{(i,k)}\!\approx\!\frac{1}{nK}\!\!\!\sum_{S\subseteq\Omega^{\text{pixel}}\setminus\{(i,k)\}}\!\!\!\!\binom{nK-1}{|S|}^{\!\!-1}\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\underbrace{\left[\frac{\partial z(S)}{\partial\delta_{(i,k)}}\delta_{(i,k)}\right]}_{{\textrm{approximates}\;z(S\cup\{(i,k)\})-z(S)}}\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!$$

(7)

where $\Omega^{\text{pixel}}=\{(i,k)|1\leq i\leq n,1\leq k\leq K\}$.

Efficient approximation of rewards of components: Let there be $m$ components in a certain clustering step, i.e. $\{C^{(1)},C^{(2)},\dots,C^{(m)}\}$. Given a component $C^{(u)}$, we approximate $\phi_{C^{(u)}}$ using combinations of components, instead of perturbation pixels. We further reduce the cost of calculating $\phi_{C^{(u)}}$ in the similar way of calculating $\phi_{i}$. Each perturbation pixel $i$ in $C^{(u)}$ is divided into $K$ sub-pixels, i.e. $\forall i\in C^{(u)},\delta_{i}=\sum_{k}\delta_{(i,k)}$. In this way, $C^{(u)}$ can be divided into $K$ sub-components. The $k$-th sub-component is given as ${C^{(u)}_{k}}=\bigcup_{i\in C^{(u)}}\{(i,k)\}$. The reward of $C^{(u)}_{k}$ is approximated as follows. Please see section B.3 in supplementary materials for the proof.

$$\begin{split}\phi_{C^{(u)}_{k}}\approx\frac{1}{mK}\sum\nolimits_{S\subseteq\Omega^{\text{comp}}\setminus\{C^{(u)}_{k}\}}\binom{mK-1}{|S|}^{-1}A\end{split}$$

(8)

where $\Omega^{\text{comp}}=\{C^{(u)}_{k}|1\leq u\leq m,1\leq k\leq K\}$, and $A=\sum_{(i,k)\in C^{(u)}_{k}}\left[\frac{\partial z(S)}{\partial\delta_{(i,k)}}\delta_{(i,k)}\right]$, which approximates $z(S\cup\{C^{(u)}\})-z(S)$.

Implementation & computational complexity: The computation of Shapley values is NP-hard. The sampling-based method has been widely used for approximation [5]. Thus, we propose to approximate Equation (7) and Equation (8) by a sampling method. The complexity of computing the Shapley value of one pixel is $O(2^{n})$. Equation (9) reduces the computational complexity to $O(nKT)$, where $n$ is the pixel number, and $T$ is times of sampling. Because derivatives to sub-pixels w.r.t. all pixels $i\in\Omega$ can be computed simultaneously, the computational complexity of computing Shapley values of all pixels remains $O(nKT)$.

$$\begin{split}\phi_{(i,k)}\!\approx&\frac{1}{nKT}\!\!\sum_{s=0}^{nK-1}\sum_{t=1}^{T}\delta_{(i,k)}\left.\frac{\partial z(S)}{\partial\delta_{(i,k)}}\right|_{S=S_{st}^{\text{pixel}}}\!\!s.t.\left|S_{st}^{\text{pixel}}\right|=s\!\!\!\!\!\!\!\!\!\!\!\!\\ \phi_{C^{(u)}_{k}}\!\approx&\frac{1}{mKT}\sum_{s=0}^{mK-1}\!\!\sum_{(i,k)\in C^{(u)}_{k}}\sum_{t=1}^{T}\delta_{(i,k)}\left.\frac{\partial z(S)}{\partial\delta_{(i,k)}}\right|_{S=S_{st}^{\text{comp}}}\!\!\!\!\!\!\!\!\!\!\!\!\\ &s.t.\quad\left|S_{st}^{\text{comp}}\right|=s\end{split}$$

(9)

where $t$ represents a sampling step; $s$ controls the size of the set $S$; $S_{st}^{\text{pixel}}\subseteq\Omega^{\text{pixel}}\setminus\{(i,k)\}$ denotes a random subset of $s$ sub-pixels excluding $(i,k)$; $S_{st}^{\text{comp}}\subseteq\Omega^{{\text{comp}}}\setminus\{C^{(u)}_{k}\}$ denotes a random subset of $s$ sub-components excluding $C^{(u)}_{k}$.

Even with above approximation, the computational cost for computing all component candidates in each step of clustering is still high. Thus, we further approximate rewards of component candidates by simplifying contextual relationships of far-away pixels [6]. We use $S^{c}=C^{(i_{1})}\cup C^{(i_{2})}\cup\dots\cup C^{(i_{q})}$ to denote a component candidate. If we compute $\phi^{\prime}_{S^{c}}$ in the set {$S^{c},C^{(i_{k+1})}\dots,C^{(i_{m})}\}$, the computational complexity is $O((m-q)KT)$. The complexity of computing rewards of all candidates is $O(m(m-q)KT)$. Here, instead of computing $\phi^{\prime}_{S^{c}}$ in the set {$S^{c},C^{(i_{k+1})}\dots,C^{(i_{m})}\}$, we randomly merge $\tilde{m}$ components to get $\tilde{m}/q$ component candidates, including $S^{c}$. In this way, the new set includes $\tilde{m}/q$ component candidates and $m-\tilde{m}$ components, i.e. $\{S^{c},\bigcup_{a=q+1}^{2q}C^{(i_{a})},\dots,\bigcup_{a=\tilde{m}-q+1}^{\tilde{m}}C^{(i_{a})},C^{(i_{\tilde{m}+1})},\dots,C^{(i_{m})}\}$. We can simultaneously compute rewards of $\tilde{m}/q$ candidates in the new set, and the computational complexity is $O((m-(q-1)\tilde{m}/q)KT)$. To compute rewards of all potential component candidates, we need to sample $qm/\tilde{m}$ different sets. In this way, the overall complexity for the computation of rewards of candidates is reduced from $O(m(m-q)KT)$ to $O(m(qm/\tilde{m}-q)KT)$. Please see section B.4 supplementary materials for details.

Visualization of regional attributions: Fig. 3 visualizes adversarial perturbations and regional attributions. We compared magnitudes of regional adversarial perturbations with regional attributions. Given a region $\Lambda_{uv}$, the magnitude of the regional perturbation was computed as $(\sum_{i\in\Lambda_{uv}}|\delta_{i}|^{2})^{1/2}$. Fig. 3 shows that attributions to the $L_{2}$ attack and magnitudes of regional perturbations were similar, but attributions to the $L_{\infty}$ attack and magnitudes of regional perturbations were usually different. For example, let us focus on the top left girl image in Fig. 3, $L_{\infty}$ perturbations uniformly distributed over the entire image, while attributions mainly focused on the face region. We have also compared regional attributions with different hyper-parameters ($\beta$ and $L$), which shows that regional attributions were insensitive to the selection of hyper-parameters. Please see section D.1 supplementary materials for details.

Similarity between regional attributions through different attacks: As Fig. 3 shows, although the distribution of $L_{2}$ adversarial perturbations and the distribution of $L_{\infty}$ adversarial perturbations were dissimilar, their regional attributions were similar to each other. Given regional attributions to the $L_{2}$ attack $\phi^{(2)}\in\mathbb{R}^{L\times L}$ and regional attributions to the $L_{\infty}$ attack $\phi^{(\infty)}\in\mathbb{R}^{L\times L}$, we used the $IoU=\frac{\sum_{u}\sum_{v}\min(\phi^{\prime(2)}_{uv},\phi^{(\prime\infty)}_{uv})}{\sum_{u}\sum_{v}\max(\phi^{\prime(2)}_{uv},\phi^{\prime(\infty)}_{uv})}$ to measure the similarity between regional attributions $\phi^{(2)}$ and $\phi^{(\infty)}$, where we normalized the attribution $\phi^{\prime}_{uv}=\frac{\phi_{uv}-\min_{u^{\prime},v^{\prime}}\phi_{u^{\prime}v^{\prime}}}{\max_{u^{\prime},v^{\prime}}\phi_{u^{\prime}v^{\prime}}-\min_{u^{\prime},v^{\prime}}\phi_{u^{\prime}v^{\prime}}}$. We also showed the IoU between $L_{2}$ magnitudes and $L_{\infty}$ magnitudes. Please see Table 1 for quantitative results of similarity between regional attributions and similarity between magnitudes of regional perturbations. In Fig. 3, we found that regional attributions to the $L_{2}$ attack and regional attributions to the $L_{\infty}$ attack were similar, while magnitudes of $L_{2}$ adversarial perturbations and regional magnitudes of $L_{\infty}$ adversarial perturbations were dissimilar.

Extraction of perturbation components: We extracted interactions between perturbation pixels generated in the $L_{2}$ attack, and decomposed the perturbation into components. To reduce the computation cost, we regarded each group of neighboring $4\times 4$ pixels as a super-pixel. We set $\tilde{m}=0.5\cdot m$. In the first step of clustering, $\gamma$ was set to satisfy $\mathbb{E}_{S}[{\mathbbm{1}}(\left|I[S]\right|>\gamma)]=0.2$, where ${\mathbbm{1}(\cdot)}$ is the indicator function. In subsequent steps of clustering, $\gamma$ was set to satisfy $\mathbb{E}_{S}[{\mathbbm{1}}(\left|I[S]\right|>\gamma)]=0.5$. To obtain stable results, when computing the reward of a component candidate $S_{c}$, we kept the nearest component of each component in $S_{c}$ always present in sampling. In each step, we ended up merging components, when component candidates for which the reward had been computed covered more than 90% components. We used the greedy strategy to merge components, and kept conducting clustering until the size of each component was 64.

We first extracted perturbation components on normally-trained DNNs. Fig. 4 visualizes adversarial perturbations and corresponding perturbation components. Note that we enlarged the size of each super-pixel, to clarify the visualization. Fig. 4 shows that components were not aligned with visual concepts. For example, adversarial perturbations on the top left girl image in the first row of Fig. 4 was segmented into five components, in which the pink component covered the neck, hair, and other regions without semantic meanings.

Effects of adversarial training on perturbation components: We also extracted perturbation components on adversarially-trained DNNs, which are visualized in Fig. 4. We used the method proposed by Madry et al. [28] for adversarial training, which was formulated as $\min_{\boldsymbol{\theta}}\mathbb{E}_{x_{i}\in X}\max_{x_{i}^{\prime}:\left\|\mathbf{x}^{\prime}_{i}-\mathbf{x}_{i}\right\|_{p}\leq\epsilon}\ell\left(g_{\boldsymbol{\theta}}\left(\mathbf{x}^{\prime}_{i}\right),c_{i}\right)$, where $c_{i}$ represents the label of $x_{i}$, and $\ell$ denotes the classification loss.

We investigated whether perturbation components were mainly localized in the foreground or the background. A perturbation component was regarded being in the foreground, if perturbation pixels in the component belonging to the foreground were more than perturbation pixels in the component belonging to the background. Table 2 reports the ratio of components in the foreground for adversarially-trained DNNs and normally-trained DNNs. Adversarially-trained DNNs had more perturbation components in the foreground than normally-trained DNNs.

Moreover, we also explored the utility of perturbation components, i.e. whether the component mainly decreased the prediction score of the true category or mainly increased the score of the target category. We classified perturbation components into two types, i.e. components mainly decreasing the prediction score of the true category and components mainly increasing the score of the target category. $\Delta y_{\ell}=\left|g_{\ell}(x+\delta)-g_{\ell}(x+\delta\circ M^{(\Omega\setminus C^{(u)})})\right|$ and $\Delta y_{t}=\left|g_{t}(x+\delta)-g_{t}(x+\delta\circ M^{(\Omega\setminus C^{(u)})})\right|$ measured the decrease of the prediction score of the true category and the increase of the prediction score of the target category caused by the component $C^{(u)}$, respectively. If $\Delta y_{\ell}>\Delta y_{t}$, the component $C^{(u)}$ was regarded mainly decreasing the score of the true category; otherwise, mainly increasing the score of the target category. Table 3 reports the ratio of components that mainly decreased the prediction score of the true category. The ratio of components mainly decreasing the score of the true category in adversarially-trained DNNs were usually greater than that in normally-trained DNNs.