Preventing Unauthorized Use of Proprietary Data:
Poisoning for Secure Dataset Release

The topic of data manipulation for the purposes of performance degradation has been investigated in the data poisoning literature. Specifically, this work is closely related to indiscriminate (availability) poisoning attacks wherein an attacker wishes to degrade performance on a large number of samples (Barreno et al., 2010). Early works on this type of attack show that data can be maliciously modified to degrade test-time performance of simple classical algorithms, such as support vector machines, principle component analysis, clustering, logistic regression, etc., or in the setting of binary classification. (Muñoz-González et al., 2017; Xiao et al., 2015; Biggio et al., 2012; Koh et al., 2018; Steinhardt et al., 2017b).

In scenarios involving simple learning models, the optimal perturbation to training data can often be explicitly calculated via the implicit function theorem. However, this becomes computationally intractable for modern deep networks. As a consequence, little work has been done on indiscriminate attacks on deep networks. Recently, Shen et al. (2019) proposed a heuristic to avoid having to explicitly solve the full bi-level objective. The method, TensorClog, crafts perturbations to cause gradient vanishing with the aim of preventing a deep network from training on the perturbed data, thus degrading test time performance of the network. However, this work only performs their method in the setting of transfer learning where a known feature extractor is used, limiting the viability of this attack.

In contrast to indiscriminate attacks, targeted (integrity) poisoning attacks aim to cause a network trained on modified data to mis-classify a few pre-selected target samples. Unlike the indiscriminate attack setting, recent work on targeted poisoning has successfully attacked modern deep networks trained from scratch on poisoned data (Geiping et al., 2020; Huang et al., 2020). These attacks do not noticeably degrade validation accuracy, despite the victim network mis-classifying the selected target example(s). Moreover, simply performing a large number of targeted attacks to degrade overall validation accuracy is not feasible. In some cases, these attacks perturb up to $10\%$ of the training data in order to mis-classify a single target image, and they often fail to attack more than a handful of targets (Geiping et al., 2020). These attacks rely upon the expressiveness of deep networks to “gerrymander” the decision boundary of the victim network around the selected target examples - a strategy that will not work in the general indiscriminate setting.

We develop an indiscriminate data poisoning attack which works on deep networks trained from scratch in a black-box setting. Our method allows practitioners to minimally modify data which, when released, causes models trained on this data to generalize poorly. Our method allows companies to release data, either for transparency purposes, or via user upload, which does not compromise the competitive advantage the company gains from asymmetric access to the clean data. For a general overview of data poisoning attacks, defenses, and terminology, see Goldblum et al. (2020).

Also parallel to the goals of this work are defenses to model stealing attacks. Model stealing attacks often aim duplicate a machine learning model or its functionality (Tramèr et al., 2016). Defenses vary depending upon the attack scenario. For example, a defense proposed in Orekondy et al. (2019) perturbs the prediction outputs of a network to prevent a model stealing attack which aims to mimic the performance of a service like a cloud prediction API. Other defenses aim to prove theft of a model has occurred after the fact by “watermarking” a network (Uchida et al., 2017). However, these defenses do little to stop malicious actors from using scraped data to train their own models (Hill, 2020).

Formally, we seek to compute perturbations $\Delta=\{\Delta_{i}\}$ to elements $x_{i}$ of a dataset $\mathcal{S}$ in order to make a network, $F$, trained on the dataset generalize poorly to the distribution $\mathcal{D}$ from which $\mathcal{S}$ was sampled. Achieving this goal entails solving the following bi-level objective,

where $\mathcal{C}$ denotes the constraint set which bounds the perturbations so that the perturbed data is perceptually similar to the clean data. In our work, we employ the $\ell_{\infty}$ constraint $\|\Delta\|_{\infty}<\epsilon$ as is standard in the adversarial literature (Madry et al., 2017; Zhu et al., 2019; Geiping et al., 2020). Constraining the perturbations in this fashion allows practitioners like social media companies to employ our method in order to release minimally changed user data while still protecting the performance of their proprietary models.

Directly solving for $\Delta$ which minimizes this objective is intractable as this would require backpropagating through the entire training procedure found in the inner objective (2) for each iteration of gradient descent on the outer objective. Thus, the bilevel objective must be approximated.

It has been demonstrated in Witches’ Brew (Geiping et al., 2020) that bounded perturbations to training data can be crafted to manipulate the gradient of a network trained on this data. We adapt gradient manipulation to the problem of general performance degradation. We estimate the outer objective $(1)$ by training a network $F$ on a clean dataset $\mathcal{S}$ and then crafting perturbations to minimize the following objective:

where $\mathcal{L}^{\prime}(F(x_{i};\theta),y_{i})$ is the “reverse” cross entropy loss (Huang et al., 2019) which discourages the network $F$ from classifying $x_{i}$ with label $y_{i}$. Specifically, the reverse cross entropy loss for a sample $(x,y)$ with one-hot label $y$ is given by:

where $p_{\theta}(x)_{y\neq 0}$ denotes the entry of the softmax of $F(x;\theta)$ corresponding to the class specified by the one-hot label $y$. For notational simplicity, we denote the target gradient

and the crafting gradient

Put simply, we seek to align the training gradient of the perturbed data with the gradient of the reverse cross-entropy loss on the clean data, $\mathcal{S}$. Ideally, when a network trains on the perturbed data, the perturbations cause the training gradient of this network at each parameter vector to be aligned with the gradient of the reverse cross entropy loss on the clean data. This in turn would decrease the reverse cross entropy loss on the clean data, causing a network trained on the perturbed data to converge to a minimum with poor generalization on the distribution from which $\mathcal{S}$ was sampled.

In order to enforce the constraints in $\mathcal{C}$, we employ projected gradient descent (PGD) as in Madry et al. (2017) on the perturbations, alternately minimizing Eq. 3 and projecting onto an $l_{\infty}$ ball. Algorithm 1 details this procedure.

Additionally, we employ techniques found in previous poisoning work such as restarts and differentiable data augmentation in the crafting procedure in order to improve the success of our perturbations (Huang et al., 2020; Geiping et al., 2020).

We pre-train a model in order to estimate the target gradient, and the crafting gradients as already trained models have been shown to provide the most stable perturbations in Geiping et al. (2020); Huang et al. (2020). Additional details about the training procedure and the crafting procedure can be found in the appendix A.1.

We establish baselines for our method on both the ILSVRC2012 dataset (ImageNet) (Russakovsky et al., 2015) and the CIFAR-10 dataset (Krizhevsky et al., 2009). ImageNet consists of over $1$ million images coming from $1000$ classes. For these experiments, to save time crafting, we use a pre-trained model from torchvision (see https://pytorch.org/docs/stable/torchvision/models.html). For memory constraints, we then craft poisons in batches of $25,000$, replacing the target gradient estimate each split. We train ImageNet models for $40$ epochs.

CIFAR-10 consists of $50,000$ images coming from $10$ classes. For the baseline experiments, we calculate the target gradient on the entire training set. We train CIFAR-10 models for $50$ epochs.

Unless otherwise stated, we craft our poisons by choosing ResNet-18 (He et al., 2015) as the architecture for $F$ using $8$ restarts and $240$ optimization steps using signed Adam as in Witches’ Brew.

For evaluation, we train a new, randomly initialized network from scratch. We test our method both on the network which was used for crafting, and in a black-box setting where the network architecture is unknown to the practitioner. See subsection 3.4 for more details on evaluation.

To test our method on an industrial-scale dataset, we first craft perturbations to ImageNet. In this setting, we deploy a full crafting procedure which mimics the scenario where a company already has a large corpus of data they wish to release. In this case, the company can train the clean model $F$ on this data, and estimate the average target gradient over this training set.

We find that we are able to significantly degrade the validation accuracy, and in the case of the largest perturbation, decrease it by more than $58\%$ (see Table 1). Visualizations for these perturbations can be found in Figure 1.

Additionally, we establish baseline comparisons for our method on CIFAR-10 by comparing our method to other poisoning methods, data manipulations, and to performance on clean non-poisoned data. We compare our method to the following alternatives:

TensorClog (Shen et al., 2019): We compare our method to TensorClog, which, to our knowledge, is the only previously existing indiscriminate poisoning attack which has been shown to work on modern deep networks. TensorClog was designed primarily for white-box transfer learning based attacks where a known feature extractor is frozen for evaluation. However, to produce a fair comparison, we re-implement their objective into our crafting regime. This objective aims to cause vanishing training gradients to prevent a network from training properly on the dataset.

Random Noise: In order to tease apart the effect of crafted versus non-crafted perturbations on validation accuracy, we also compare our method to fixed, random additive noise. Since our PGD based perturbation usually results in a perturbation close to the “corners” of the $\ell_{\infty}$ ball, i.e. most pixels are perturbed by the maximum allowed value, we enforce that the noise is of the maximum allowable level for our constraint set.

Comparisons are made with the same $\varepsilon$-bound and the same training procedure from a randomly initialized ResNet-18 model. We find that our alignment method significantly outperforms these alternatives with the same $\varepsilon$-bound. In the case of ResNet18, we degrade validation accuracy by close $50$ more percentage points than TensorClog, the next best.

The results in Tables 1 and 2 are in a grey-box setting where we do not know the victim’s model initialization or any information about batching, but we do know their architecture training details like optimizer choice. In order to more strenuously test our method in a realistic setting, we also test our poisons on architectures and training procedures completely unknown during poison crafting. This mimics the black-box setting wherein the practitioner does not know how a victim plans to train on their scraped data. Specifically, we train networks on our crafted poisons using the setup from an independent, well known repository for training CIFAR-10 models ¹¹1Training routine taken from widely used repository: https://github.com/kuangliu/pytorch-cifar..

We validate our method on commonly used architectures including VGG19 (Simonyan & Zisserman, 2014), ResNet-18 (He et al., 2015), GoogLeNet (Szegedy et al., 2015), DenseNet121 (Huang et al., 2016), and MobileNetV2 (Sandler et al., 2018). These results are presented in Table 3. We see that even at very small epsilon constraints, the proposed method is able to nearly halve the clean validation accuracy of many of these popular models.

While our method is able to degrade validation accuracy under normal training conditions, a question remains whether the method is stable to poisoning defenses and modifications in training procedures. Would this improve the validation accuracy of a model trained on the perturbed data? To this end, we investigate several avenues.

First, we investigate whether existing poisoning defenses lessen the effects of our perturbations. Many existing defenses are designed for settings which are significantly different than our proposed attack. For example, many defenses assume that there is a small amount of poisoned data, and that this will be anomalous in feature space (Steinhardt et al., 2017a; Tran et al., 2018; Peri et al., 2020; Chen et al., 2018). These types of defenses are best suited for scenarios in which the same perturbations are applied identically to each data point, as in patch based attacks, or when the defender has access to a large corpus of trustworthy data on which to train a feature extractor to filter out poisoned images based on their similarity to clean reference samples. Moreover, these anomaly detection defenses work under the assumption that the majority of data will not be poisoned. It was demonstrated in (Geiping et al., 2020) that modifications meant to alter gradients of a network trained from scratch do not produce data which is anomalous in feature space of the poisoned model. Furthermore, since we poison the entire dataset, the trained model’s feature space can no longer be thought of as containing clean and perturbed elements. Thus, the heuristic that poisoned data will somehow be outliers in feature space does not apply.

However, another family of defenses leverages differential privacy (DPSGD) as a defense against poisoning (Ma et al., 2019; Hong et al., 2020). Since differentially private models are, by construction, insensitive to minor changes in the training set, they may be resistant to different poisoning methods. By clipping and noising gradients, these defenses aim to limit the effect of perturbations placed on data. In theory, this defense is applicable to our perturbations. However, we test the defense proposed in Hong et al. (2020) against our method and find that the drop in validation accuracy we saw in previous experiments remains even when training with DPSGD.

In addition to methods designed for defending against data poisoning, we also test the potency of our poisons under both Gaussian smoothing and random additive noise (of the same magnitude as our perturbations) during training. These modifications test how brittle our crafted perturbations are to modification. For Gaussian smoothing, we use a radius $r=2$. We find that our attack is stable under all the discussed training modifications, with none of the proposed defenses substantially improving results. These results can be found in Table 5.

The CelebA dataset contains 10177 identities (Liu et al., ). Following standard procedure as in (Cheng et al., 2017), we remove 371 identities with too few images, use 8806 identities for pre-training and 1000 identities for testing. We use ResNet-18 and Resnet-50 as backbone architectures. During pre-training, we use the popular Cosface classification head (Wang et al., 2018). During poison dataset generation, we take 50 gradient steps with signAdam and a single random restart. In additional to the CelebA dataset, we also test the attacked model’s verification accuracy on four other face datasets: Labeled Faces in the Wild (LFW) (Huang et al., 2008), Celebrities in Frontal-Profile data set (CPF) (Sengupta et al., 2016), AgeDB (Moschoglou et al., 2017), VGGFace2 (Cao et al., 2018). We use the face.evoLVe repository (ZhaoJ, 9014) to run all of our facial recognition experiments.

In Table 6, we see that both identification and verification accuracy drop materially when a model is trained on our crafted dataset. When trained on poisoned data with $\epsilon=16/255$, CelebA top 1 accuracy drops by 36% to 61%, . Similarly, verification accuracy also drops by as much as 35%. Note that in many commercial applications, even a few percentage drop can tip the scales for a company to maintain a competitive advantage.

Poisons crafted on a more powerful model also transfer better in our experiments. Specifically, poisons generated with ResNet-50 always reduce both identification and verification accuracy more than poisons generated with ResNet-18 (see Table 7). This experiment suggests that if one wants to make the poisoned dataset more potent, one could simply use a larger capacity model to generate poisons.