Robustness bounds on the successful adversarial examples in probabilistic models: Implications from Gaussian processes

Today, machine learning is widely used in various fields, and concerns about its security have emerged. An adversarial example (AE) is widely investigated among such attack methods [goodfellow_explaining_2015]. AE is a sample that is slightly different from a natural sample that is misclassified by a machine learning classifier [goodfellow_explaining_2015]. AE is often crafted against neural networks by adding a small perturbation (adversarial perturbation) to the original input, and various methods are proposed to make an adversarial perturbation [carlini_towards_2017, goodfellow_explaining_2015].

Since AE is considered an attack method against machine learning, various defense methods against AE have been investigated, including the detection method [feinman_detecting_2017] and adversarial training [cai_curriculum_2018]. However, Carlini & Wagner [carlini_adversarial_2017] reviewed ten detection methods and stated that no defense method could survive a white-box AE attack, an attack using the knowledge of the victim’s architecture. Therefore, both the defense methods and their theoretical basis should be required for the defense against AE.

In the current study, we investigated the theoretical basis of AE using the Gaussian Process (GP), a stochastic process whose output is multivariate normally distributed. GP can be used for a probabilistic model for classification and regression [rasmussen_gaussian_2006], and previous research shows that GP is equivalent to some types of classifier such as linear regression and a relevance vector machine, and especially, GP is equivalent to a certain type of neural networks [williams_computing_1996]. More specifically, the activation function in neural networks corresponds to the kernel function in the NN [williams_computing_1996].

Until now, some studies have been conducted that aim to establish the theoretical basis of AE. Shafahi, Huang, Studer, Feizi & Goldstein [shafahi_are_2019] have suggested that AE is inevitable under a certain condition on the distribution of the training data, using the geometrical method. Blaas et al. [blaas_adversarial_2020] have shown the algorithmic method to calculate the upper and lower bound of the value of GP classification in an arbitrary set. Cardelli, Kwiatkowska, Laurenti & Patane [cardelli_robustness_2019] have shown the upper bound of the probability of robustness of the GP output in an arbitrary set. Wang, Jha & Chandhuri [wang_analyzing_2018] have shown the method of creating a robust dataset when using a k-nearest neighborhood classifier. Fawzi, Fawzi & Fawzi [fawzi_adversarial_2018] have shown the fundamental upper bound on the robustness of the classification, which some studies are based on. Mahloujifar, Diochnos & Mahmoody [mahloujifar_curse_2019] have applied their results to broader types of data distributions, and Zhang, Chen, Gu & Evans [zhang_understanding_2020] have shown the intrinsic robustness bounds for classifiers with a conditional generative model. More specifically, Gilmer et al. [gilmer_relationship_2018] have shown a fundamental bound relating to the classification error rate in the field of neural networks; Bhagoji, Cullina & Mittal [bhagoji_lower_2019] have shown lower bounds of adversarial classification loss using optimal transport theory.

Some studies suggest certified defense methods. Randomized smoothing [cohen_certified_2019] is a method for composing "smoothed" classifiers that are derived from arbitrary classifiers. The prediction of an arbitrary input by smoothed classifiers has a safety radius, within which the classification results of the neighborhood inputs are the same as the central input.

Our approach is different from the related studies in the following viewpoints. First, our approach focuses on the training data for the classifier and is easier to use practically. For example, the other studies focus on the geometry of the classifier [shafahi_are_2019] or the upper bound for which some cumbersome calculation with all training data is necessary [blaas_adversarial_2020, cardelli_robustness_2019]. Using our approach, we can analyze how the robustness will change when the distribution of training data of the classifier changes, for example, when some training data are removed from the dataset or moved within the input space. Cohen, Rosenfeld & Kolter’s randomized classifier [cohen_certified_2019] needs Monte Carlo samplings to calculate the variance used in the calculation of the safety radius. However, our approach can provide the predictive variance using GP. Thus, our approach can be used with Cohen, Rosenfeld & Kolter’s randomized smoothing.

Second, our approach can be applied to a wide range of classifiers thanks to GP. For example, Wang, Jha & Chandhuri ’s approach [wang_analyzing_2018] also focuses on how robustness changes when the distribution of training data of the classifier changes, but their approach is based on the nearest-neighbor method. Our approach is based on GP, which can be applied to broader inference models. Smith, Grosse, Backes & Alvarez [smith_adversarial_2023] have shown adversarial vulnerability bounds for binary GP classification. However, Smith, Grosse, Backes & Alvarez’s proof is based on the Gaussian kernel, while the result of our research is based on the characteristics of the kernel functions, which includes the Gaussian kernel as a special case. Therefore, the current study has a broader application than [smith_adversarial_2023].

Our research has the following contributions.

• •

  Our research shows that the success probability of an AE attack against a certain dataset in the GP classification with a certain kernel function is upper-bounded by the function of the distance of the nearest points that have different labels.

• •

  We confirmed our theoretical result through the experiment using ImageNet with various kernel parameters in GP classification. The experimental result is well-suited to the theoretical result.

• •

  We showed that changing the parameters of the kernel function causes the change of the theoretical upper bound, and thus our result gives the theoretical basis for the enhancement method of robustness.

Let $\mathcal{D}$ be a dataset with $N$ samples that has data points $x_{i}$ and labels of object classes $y_{i}$. The data point $x_{i}$ is a $D$-dimensional vector, and the label $y_{i}$ is binary. Namely,

Let

for further simplicity.

We consider the binary classification of the dataset, using GP regression with a certain kernel function $k(x,x^{\prime})$ [rasmussen_gaussian_2006]. We define a GP regressor $\mathcal{R}(x):\mathbb{R}^{D}\rightarrow\mathcal{N}(y;\mu_{\mathcal{R}},\sigma^{2}_{\mathcal{R}})$. $\mathcal{R}(x)$ gives the output as a Gaussian distribution $\mathcal{N}(y;\mu_{\mathcal{R}},\sigma^{2}_{\mathcal{R}})$. Then we define a probabilistic GP classifier $\mathcal{C}(\cdot):\mathbb{R}^{D}\rightarrow\{+1,-1\}$, which is constructed as below:

where $p$ is a value sampled from the distribution $\mathcal{N}(x)$. Intuitively, $\mathcal{C}(x)$ is a probabilistic binary classifier whose output is $+1$ when a sample from the output distribution of $\mathcal{R}(x)$ is not negative.

Let the maximum value of the kernel function between two input points of $\mathcal{D}$ with different labels be $s$. We write the definition of $s$ as follows, without loss of generality:

The value of the kernel function between 2 points $x_{+}\in\mathcal{D_{+}},x_{-}\in\mathcal{D_{-}}$ is described as s, where $x_{-}\in\mathcal{D}_{-}$ gives the maximum value of $k(x_{+},x_{-})$ when $x_{+}$ is fixed. $s$ can be written as a function of $x_{+}$, that is

Put $x_{*}\in\mathbb{R}^{D}$ such that $k(x_{+},x_{*})=r$, where $r$ is a constant.

In this paper, we introduce the following assumption.

Intuitively, the former part of ˜1 suggests that the predictive mean of the GP regressor with all training data is close to that with only two training data, implying that the effect of training data other than $x_{+}$, $x_{-}$ is small.

The schematic diagram is shown in Fig.˜1.

Proof sketch. First, we prove that the prediction variance increases if the training point increases in GP regression (Lemma 1). In the next, we prove that $\operatorname{Erf}(0;\mu,\sigma)=\int^{0}_{-\infty}\mathcal{N}(x;\mu,\sigma^{2})dx$ increases monotonically with respect to $\sigma^{2}$ if $\mu>0$ (Lemma 2). Then we prove that the probability that $x_{*}$ is classified as $-1$ increases monotonically with respect to $k(x_{*},x_{-})$ (Lemma 3), and the theorem is proved. ∎

In the proof, we set the notations below.

We construct a GP regressor $\mathcal{R}(x)$ with the kernel function $k(x,x^{\prime})$. In GP regressor, the mean $\mu_{\mathcal{R}}$ and the variance $\sigma_{\mathcal{R}}^{2}$ of the new data point $x_{*}$ can be calculated as

where

In the next we construct GP classifier $\mathcal{C}(x)$ whose prediction is $y_{*}=+1$ if a sample from $\mathcal{R}(x)$ is greater than 0, and the prediction is $y_{*}=-1$ otherwise.

For further calculation, set the below notations

The following lemma is a simpler version of the statement Exercise 4 of Chapter 2 from [rasmussen_gaussian_2006]. The proof is in the supplemental material.

Thus, if the two points ($x_{+},x_{-}$) in the training dataset of GP regression are fixed, the maximum predictive variance is the variance calculated with the two points as the training dataset only.

Next, we calculate the predicted mean under the condition that only $x_{+}$ and $x_{-}$ are used as the training dataset. We write the predictive mean as $\mu_{\mathcal{R}2}$.

Then $k(x_{*})$ and y are written as $k(x_{*})=(\theta_{r1},\theta_{r2})^{T}$, $\textbf{y}=(+1,-1)$. Using these, the predicted mean is calculated below, plugging them into Eq.˜10.

The category boundary is regarded as the point where the predicted mean is 0, therefore on the category boundary $\frac{\theta_{r1}-\theta_{r2}}{\theta_{1}-\theta_{s}}=0$ holds.

Therefore, category boundary is located where $\theta_{r1}=\theta_{r2}.$

This implies that the set of the points which satisfies $k(x_{+},x_{*})=k(x_{-},x_{*})$ is the category boundary, and Eq.˜29 implies that where $k(x_{+},x_{*})>k(x_{-},x_{*})$ the mean is positive.

Similarly, the prediction variance with only $x_{+},x_{-}$ as the training dataset is described as follows, plugging the values into Eq.˜11:

Considering Lemma˜1, Eq.˜30 gives the maximum predictive variance of GP regression with the training dataset including $x_{+},x_{-}$.

The probability of $\mathcal{C}(x_{*})=-1$ is the probability that the value of a sample from the output distribution is lower than 0. Therefore, it can be calculated as follows, using the cumulative distribution function of Gaussian distribution

where