Diagnostics for Deep Neural Networks with
Automated Copy/Paste Attacks

It is important to have scalable methods that allow humans to exercise effective oversight over deep neural networks. Adversarial examples are one type of tool which can be used to study weaknesses in DNNs [13, 53, 45]. Typically, adversarial examples are generated by optimizing perturbations to the input of a network. And some previous works offer examples of adversaries being used to develop generalizable interpretations of DNNs [13, 53, 27, 10].

However, there are limitations to what one can learn about flaws in DNNs from synthesized features [5]. First, synthetic adversarial perturbations are often difficult to describe and thus offer limited help with human-centered approaches to interpretability. Second, even when synthetic adversarial features are interpretable, it is unclear without additional testing whether they fool a DNN due to their interpretable features or due to hidden motifs [7, 27]. This makes developing a generalizable understanding from them difficult. Third, there is a gap between research and practice in adversarial robustness [2]. Real-world failures of DNNs are often due to atypical natural features or combinations thereof [22], but synthesized features are off this distribution.

Here, we work to diagnose weaknesses in DNNs using natural, interpretable features. We introduce using a Search for Natural Adversarial Features Using Embeddings (SNAFUE) to find novel adversarial combinations of natural features. We apply SNAFUE to find copy/paste attacks for an image classifier in which one natural image is inserted as a patch into another to induce a targeted misclassification. Figure 1 outlines this approach. First, we use a generator to synthesize robust feature-level adversarial patches [10] which are designed to make any image from a particular source class misclassified as a target. Second, we use the target model’s latent activations to create embeddings of both these synthetic patches and a dataset of natural patches. Finally we select the natural patches that embed most similarly to the synthetic ones.

We apply SNAFUE at the ImageNet scale. First, we use SNAFUE to replicate all successful known examples of copy/paste attacks from previous works with no human involvement. Second, we demonstrate its scalability by identifying hundreds of vulnerabilities. Figure 2 and Figure 9 show examples which illustrate easily-describable misassociations between features and classes in the network. Overall, this work makes two contributions.

1. 1.

   Algorithmic: We introduce Search for Natural Adversarial Features Using Embeddings (SNAFUE) as a tool for scalable human oversight.

2. 2.

   Diagnostic: We apply SNAFUE by red-teaming an image classifier. We demonstrate that it automatedly identifies weaknesses due to natural features that are uniquely human-interpretable.

Meanwhile, in concurrent work [9], we compare SNAFUE to other interpretability tools by using them to help humans rediscover trojans in a network. We find that SNAFUE offers an effective and unique tool for helping humans interpret and debug DNNs. Code is available at https://github.com/thestephencasper/snafue.

Describable Synthetic Adversarial Attacks: Conventional adversarial attacks are effective but difficult for a human to interpret. They tend to be imperceptible and, when exaggerated, typically appear as random or mildly-textured noise [52, 19]. Thus, from a human-interpretability perspective, they demonstrate little aside from how the network can be vulnerable to this specific class of perturbations. Some works have used perturbations inside the latents of image generators to synthesize more describable synthetic attacks [37, 47, 50, 30, 29, 49, 24, 55]. These works however, have not focused on interpretability and only studied small networks trained on simple datasets (MNIST [34], Fashion MNIST [57], SVHN [42], CelebA [38], BDD [58], INRIA [12], and MPII [1]). In the trojan detection literature, some methods have aimed to reconstruct trojan features using regularization and transformations. Thus far, however, they have been limited to recovering small, few-pixel triggers while failing to reconstruct triggers that take the form of larger objects [54, 20].

Natural Adversarial Features: Several approaches have been used for discovering natural adversarial features. One is to analyze examples in a test set that a DNN mishandles [22, 15, 28], but this limits the search for weaknesses to a fixed dataset and cannot be used for discovering adversarial combinations of features. Another approach is to search for failures over an easily-describable set of perturbations [17, 33, 51], but this requires performing a zero-order search over a fixed set of changes.

Copy Paste Attacks: Copy/paste attacks have been a growing topic of interest and offer another method for studying natural adversarial features. Some interpretability tools have been used to design copy/paste adversarial examples including feature-visualization [8] and methods based on network dissection [3, 41, 23]. Our approach is related to that of [10] who introduce robust feature level adversarial patches and use them for interpreting DNNs and designing copy-paste attacks. However, copy/paste attacks from [8, 41, 23, 10] have been limited to simple proofs of concept with manually-designed copy/paste attacks. They also required a human process of interpretation, trial, and error in the loop. We build off of these with SNAFUE which is the first method that identify adversarial combinations of natural features for vision models in a way that is (1) not restricted to a fixed set of transformations or a limited set of source and target classes and (2) efficiently automatable.

Figure 1 outlines our approach for finding copy/paste adversaries for image classification. For all experiments, we report the success rate defined as the proportion of the time that a patched image was classified as the target class minus the proportion of the time the unpatched natural image was. Additional details are in Appendix A.1.

Synthetic adversarial patches: First, we create synthetic robust feature level adversarial patches as in [10] by perturbing the latent activations of a BigGAN [6] generator. The synthetic adversarial patches were trained to cause any source class image to be misclassified as the target regardless of the insertion location in the source image.

Candidate patches: Patches for SNAFUE can come from any source and do not need labels. Features do not necessarily have to be natural and could, for example, be procedurally generated. Here, we used a total of $N=$ 265,457 natural images from five sources: the ImageNet validation set [46] (50,000) TinyImageNet [32] (100,000), OpenSurfaces [4] (57,500), the non OpenSurfaces images from Broden [3] (37,953), plus four trojan triggers (4) (see Section 4).

Embeddings: We used the $N=$ 265,457 natural patches along with $M=10$ adversarial patches, and passed them through the target network to get an $L$-dimensional embedding of each using the post-ReLU latents from the penultimate (avgpooling) layer of the target network. The result was a nonnegative $N\times L$ matrix $U$ of natural patch embeddings and a $M\times L$ matrix $V$ of adversarial patch embeddings. A different $V$ must be computed for each attack, but $U$ only needs to be computed once. This plus the fact that embedding the natural patches does not require insertion into a set of source images makes SNAFUE much more efficient than a brute-force search. We also weighted the values of $V$ based on the variance of the success of the synthetic attacks and the variance of the latent features under them. Details are in Appendix A.1.

Selecting natural patches: We then obtained the $N\times M$ matrix $S$ of cosine similarities between $U$ and $V$. We took the $K^{\prime}=300$ patches that had the highest similarity to any of the synthetic images, excluding ones whose classifications from the target network included the target class in the top 10 classes. Finally, we evaluated all $K^{\prime}$ natural patches under random insertion locations over all 50 source images from the validation set and subsampled the $K=10$ natural patches that increased the target network’s post-softmax confidence in the target class the most. Screening the $K^{\prime}$ natural patches for the best 10 caused only a marginal increase in computational overhead. The method was mainly bottlenecked by the cost of training the synthetic adversarial patches (for 64 batches of 32 insertions each).

Automation of interpretations: Unlike previous proofs of concept, SNAFUE does not rely on a human in the loop. However, we still rely on a human after the loop for analyzing the final results and making a final interpretation by default. To test how easily this may be automatable, we test a method for turning many images into a single caption in Appendix A.3.

Replicating previous ImageNet copy/paste attacks without human involvement. First, we set out to replicate all known successful ImageNet copy/paste attacks from previous works without any human involvement. To our knowledge, there are 9 such attacks, 3 each from [8], [23]²²2The attacks presented in [23] were not universal within a source class and were only developed for a single source image each. When replicating their results, we use the same single sources. When replicating attacks from the other two works, we train and test the attacks as source class-universal ones. and [10].³³3[10] test a fourth attack involving patches making traffic lights appear as flies, the examples they identified were not successful at causing targeted misclassification.⁴⁴4[41] also test copy paste attacks, but not on ImageNet networks We used SNAFUE to find 10 natural patches for all 9 attacks. Figure 3 shows the results. In all cases, we are able to find successful natural adversarial patches. We also find in most cases that we find similar adversarial features to the ones identified in the prior works. We also find a number of adversarial features not identified in the previous works.

SNAFUE is scalable and effective between similar classes. There are many natural visual features that image classifiers may encounter and many more possible combinations thereof, so it is important that tools for interpretability and diagnostics with natural features are scalable. Here, we test the scalability and effectiveness of SNAFUE with a broad search for vulnerabilities. Based on prior proofs of concept [8, 41, 23, 10] copy/paste attacks tend to be much easier to create when the source and target class are related (see Figure 3). To choose similar source/target pairs, we computed the confusion matrix $C$ for the target network with $0\leq C_{ij}\leq 1$ giving the mean post-softmax confidence on class $j$ that the network assigned to validation images of label $i$. Then for each of the 1,000 ImageNet classes, we conducted 5 attacks using that class as the source and each of its most confused 5 classes as targets. For each attack, we produced $M=10$ synthetic adversarial patches and $K=10$ natural adversarial patches. Figure 2 and Figure 4 show examples from these attacks with many additional examples in Appendix Figure 9. Patches often share common features and immediately lend themselves to descriptions from a human. In Appendix A.3, we find that these patches also often lend themselves to common descriptions from neural captioning models.

At the bottom of Figure 4, are histograms for the mean attack success rate for all patches and for the best patches (out of 10) for each attack. The synthetic feature-level adversaries were generally highly successful, and the the natural patches were also successful a significant proportion of the time. In this experiment, 3,451 (6.9%) out of the 50,000 total natural images from all attacks were at least 50% successful at being targeted adversarial patches under random insertion locations into random image of the source class. This compares to a 10.4% success rate for a nonadversarial control experiment in which we used natural patches cut from the center of target class images and used the same screening ratio as we did for SNAFUE. Meanwhile, 963 (19.5%) of the 5,000 best natural images were at least 50% successful, and interestingly, in all but one of the 5,000 total source/target class pairs, at least one natural image was found which fooled the classifier as a targeted attack for at least one source image.

Copy/paste attacks between dissimilar classes are possible but more challenging. In some cases, the ability to robustly distinguish between similar classes may be crucial. For example, it is important for autonomous vehicles to effectively tell red and yellow traffic lights apart. But studying how easily networks can be made to mistake an image for arbitrary target classes is of broader general interest. While synthetic adversarial attacks often work between arbitrary source/target classes, to the best of our knowledge, there are no successful examples from any previous works of class-universal copy/paste attacks.

We chose to examine the practical problem of understanding how vision systems in vehicles may fail to detect pedestrians [43] because it provides an example where failures due to novel combinations of natural features could realistically pose safety hazards. To test attacks between dissimilar classes, we chose 10 ImageNet classes of clothing items (which frequently co-occur with humans) and 10 of traffic-related objects.⁵⁵5{academic gown, apron, bikini, cardigan, jean, jersey, maillot, suit, sweatshirt, trenchcoat} $\times$ {fire engine, garbage truck, racer, sports car, streetcar, tow truck, trailer truck, trolleybus, street sign, traffic light} We conducted 100 total attacks with SNAFUE using each clothing source and traffic target. Figure 5 shows these results. Outcomes were mixed.

On one hand, while the synthetic adversarial patches were usually successful on more than 50% of source images, the natural ones were usually not. Only one out of the 1,000 total natural patches (the leftmost natural patch in Figure 5) succeeded for at least 50% of source class images. This suggests a limitation of either SNAFUE or of copy/paste attacks in general for targeted attacks between unrelated source and target classes. On the other hand, 54% of the natural adversarial patches were successful for at least one source image, and such a natural patch was identified for 87 of all 100 source/target class pairs.

SNAFUE is unique and relatively effective compared to other interpretability/diagnostic tools: Finally, we test how SNAFUE compares to related tools meant to help humans better understand and diagnose bugs in models. In concurrent work [9], we evaluate interpretability tools for DNNs based on how effective they are at helping humans identify trojans that have been implanted into DNNs. We test SNAFUE against 8 other interpretability tools for DNNs based on feature synthesis/search. We find that SNAFUE and robust feature level adversaries [10] are the most successful overall and that combinations of methods are more helpful than individual ones.

Implications for scalable human oversight. Having effective diagnostic tools to identify problems with models is important for trustworthy AI. The most common way to evaluate a model is with a test set. But good testing performance does not imply that a system will generalize safely in deployment. Test sets do not typically reveal failures such as spurious features, out of distribution inputs, and adversarial vulnerabilities. Thus, it is important to have scalable tools that allow humans to exercise effective oversight over deep neural networks.

Interpretability tools are useful for building more trustworthy AI because of the role they can play in helping humans exercise oversight. But many techniques from the literature suffer from limitations including a lack of scalability and usefulness for identifying novel flaws. This has led to criticism, with a number of works noting that few interpretability tools are used by practitioners in real applications [14, 36, 40, 31, 45]. Toward practical methods to find weaknesses in DNNs, we introduce SNAFUE as an automated method for finding natural adversarial features.

SNAFUE identifies distinct types of problems. In some cases, networks may learn flawed solutions because they are given the wrong learning objective while in other cases, they may fail to converge to a desirable solution even with the correct objective [26]. SNAFUE can discover both types of issues. In some cases, it discovers failures that result from dataset biases. Examples include when it identifies that cats make envelopes misclassified as cartons or that young children make bicycles-built-for-two misclassified as tricycles (Figure 2 rows 1-2). In other cases, SNAFUE identifies failures that result from the particular representations a model learns, presumably due to equivalence classes in the DNN’s representations. Examples include equating black and white birds with killer whales, parallel lines with spatulas, and red/orange cars with fiddler crabs (Figure 2 rows 3-5).

Limitations. We find that it scales well and can easily identify hundreds of sets of copy/paste vulnerabilities that are very easy for a human to interpret and describe. However, we also find limitations including how SNAFUE is less effective for dissimilar source and target classes. In Appendix A.4, we catrogize and discuss different types of failures with SNAFUE.

Three directions for future work.

1. 1.

   Diagnostics in the wild: Vision datasets are full of biases, including harmful ones involving human demographic groups [16]. A compelling use of SNAFUE and similar techniques could be for discovering these in deployed systems. This could be valuable for exploring the practical relevance of diagnostic tools.

2. 2.

   Debugging: In addition to is use for interpretability, SNAFUE also produces adversarial data which can be used for adversarial training or probing the network to guide targeted procedural edits. Correcting vulnerabilities to copy/paste attacks could be useful applications or tests for model editing tools (e.g. [18, 11, 56, 39]). In the real world, vision systems often fail due to distractor features, atypical contexts, and occlusion [22]. Debugging with copy/paste attacks may be well-equipped to address these failures.

3. 3.

   NLP: Using a version of SNAFUE in natural language processing could be helpful for identifying natural phrases that could cause language models to fail. This could be valuable for language models because the discrete nature of their inputs makes it difficult to use gradient based methods to construct adversarial perturbations in input space. However, SNAFUE could be used to produce interpretable adversaries from synthetic adversarial insertions to the latents. Other recent works have aimed to generate adversarial triggers for language models that appear as natural language phrases. However, these often depend on either using reinforcement learning to train language generators which can be unstable and computationally expensive [44] or on using humans in the loop [59]. Because SNAFUE is automated and can work flexibly with any dataset of candidate features, it may offer a competitive alternative to existing tools.

All of the proposals for building safe AI outlined in [25] explicitly call for adversarial robustness and/or oversight via interpretability tools. Finding and fixing bugs in advanced AI systems will hinge on interpretability, adversaries, and adversarial training. Consequently, continued work toward scalable techniques for interpretability and diagnostics will be important for safer AI.

We thank Rui-Jie Yew and Phillip Christoffersen for feedback and Evan Hernandez for discussing how to best replicate results from [23]. Stephen Casper’s work was supported by the Future of Life Institute and Kaivalya Hariharan’s work was supported by the Open Philanthropy Project.