Manifold attack

We define a virtual point is a synthetic sample generated in such a way to be likely on the observed samples underlying manifold. An anchor point is a sample used for generating a virtual point. An attack point is a virtual point that maximises locally a chosen measure of model distortion. For example, attack point can be a sample perturbed with an adversarial noise.

We use the same notation as in section 3. Given a dataset $\mathcal{X}=\{\mathbf{x}_{1},...,\mathbf{x}_{N}\}$ and the corresponding embedded set $\mathcal{A}=\{\mathbf{a}_{1},...,\mathbf{a}_{N}\}$, $\mathcal{L}_{e}(\mathbf{a}_{i},\mathcal{A}_{i}^{c})$ denotes the embedding loss, $\mathcal{A}_{i}^{c}$ being the complement of $\{\mathbf{a}_{i}\}$ in $\mathcal{A}$. We consider the objective function of PGS defined as $\mathcal{L}_{t}=\sum_{i=1}^{N}\mathcal{L}_{e}(\mathbf{a}_{i},\mathcal{A}_{i}^{c})$. Let’s consider $p$ anchor points $\mathbf{z}_{1},\mathbf{z}_{2},...,\mathbf{z}_{p}\in\mathbb{R}^{n}$, a virtual point $\tilde{\mathbf{x}}\in\mathbb{R}^{n}$ is defined as:

The anchor points $\mathbf{z}_{i}$ define a region or feasible zone, in which a virtual point $\tilde{\mathbf{x}}$ must be located and $\gamma=[\gamma_{1},\gamma_{2},..,\gamma_{p}]$ is the coordinate of $\tilde{\mathbf{x}}$. Anchor points are sampled from the dataset $\mathcal{X}$ with different strategies, which are defined according to predefined rules (see section 4.4 for several examples). Figure 2 shows an example of anchor points setting and relations between points. For the embedding $\mathbf{a}_{i}$ of observed sample $\mathbf{x}_{i}$, $\mathbf{a}_{i}=g(\mathbf{x}_{i})$, the embedding loss is defined as:

Similarly, for the embedding $\tilde{\mathbf{a}}$ of virtual point $\tilde{\mathbf{x}}$, $\tilde{\mathbf{a}}=g(\tilde{\mathbf{x}})$, the embedding loss is defined as:

The algorithm 1 describes the computation of an attack point. It consists in finding the local coordinates $\gamma$ that maximizes the embedding loss $\mathcal{L}_{e}(\tilde{\mathbf{a}},\mathcal{A})$ for the current state of model $g()$. Hence, $\gamma$ is estimated though a projected gradient ascent.

In order to guarantee the constrains in equation (9), we use the projector $\Pi_{ps}$ defined by the problem:

This convex problem with constraints can be solved quickly by a simple sequential projection that alternates between sum constraint and positive constraint (algorithm 2). The demonstration can be inspired by Lagrange multiplier method. A simple demonstration can be found in Appendix.

By default, each set of anchor points has one attack point. However, we can generate more than one attack point for the same set of anchor points by using different initializations of $\gamma$, so as to find different local maxima. The double embedding loss, on the observed samples and on the attack points, is expected to enforce the the model $g()$ smoothness over the underlying manifold, including in low samples density areas. The general optimization scheme goes as follows: we optimize alternately between attack stages and model update stages until convergence. In attack stage, we optimize the attack points through $\gamma$ while fixing the model $g()$ and in the model update stage, we optimize the model $g()$ while fixing attack points.

Algorithm 4 represents the update step for multiple attack points. We assumed that the embedding loss $\mathcal{L}_{e}$ is smooth with respect to $\gamma$ and used a gradient-based algorithm to estimate the latter. Nevertheless, in PGS, there are several methods whose embedding loss is not even continuous. For example, in LLE (6), the embedding loss takes into account the $k$ nearest neighbors of a point, which might change throughout the estimation of an attack point, producing a discontinuity. To circumvent this problem, we use several strategies to avoid singularities:

• -

  By reducing the gradient step $\xi$ which limits the virtual point displacement.

• -

  By taking a small number of attack points in each subset $\mathcal{B}_{s}$ or by using randomly a part of attack points to perform the attack while fixing other attack points, in an attack stage.

• -

  By updating $\gamma$ only if embedding loss increases.

Besides, some metrics might be approximated by smooth functionals. For instance, in the contrastive loss (3), we can replace the metric $d_{x}()$ which outputs only 0 or 1, with $d_{x}(\mathbf{x}_{i},\mathbf{x}_{j})=\text{exp}\big{(}\frac{-\left\lVert\mathbf{x}_{i}-\mathbf{x}_{j}\right\rVert_{2}^{2}}{2\sigma_{i}^{2}}\big{)}$ to make embedding loss continuous.

For some PGS methods as MDS or LE, the embedding loss $\mathcal{L}_{e}$ can be decomposed into the sum of elementary pairwise loss $l_{e}$:

Then the batch loss $L$ (in algorithm 3) can be modified into:

Following this change, $L$ can be decomposed into three parts:

where $\mathcal{B}_{s}^{d}$ and $\mathcal{B}_{s}^{v}$ are respectively set that contains all embedded data points (or observed samples) and all embedded virtual points of $\mathcal{B}_{s}$.

In some PGS losses which can be decomposed into the sum of elementary pairwise loss, we can balance the effect of the virtual points with respect to the observed samples, not only by tuning the ratio between the number of virtual points and the number of observed samples in $B_{s}$, but also by weighting each of three parts above which corresponds to the settings observed-observed, observed-virtual and virtual-virtual.

In this section, we provide two settings (or rules) for computing anchor points with the corresponding initializations. These settings need to be chosen carefully to guarantee that virtual points are on the sample underlying manifold.

Neighbor anchors: The first anchor point $\mathbf{z}_{1}$ is taken randomly from $\mathcal{X}$, then the next $(p-1)$ anchor points $\mathbf{z}_{2},..,\mathbf{z}_{p}$ are taken as $(p-1)$ nearest neighbor points of $\mathbf{z}_{1}$ in $\mathcal{X}$ (Euclidean metric by default). Here, we assume that the convex hull of a sample and its neighbors is likely comprised in the samples manifold. The number of anchors $p$ needs to be small compared to the number of data points $N$. The initialization for virtual points can be set by taking $\gamma_{i}\sim\mathcal{U}(0,1),\forall i=1,..,p$ then normalize to have $\sum_{i=1}^{p}\gamma_{i}=1$.

Random anchors: The second setting is inspired by Mix-up method [26]. $p$ anchors are taken randomly from $\mathcal{X}$ and we take $\gamma\sim\text{Dirichlet}(\alpha_{1},..,\alpha_{p})$. If $\alpha_{i}\ll 1,\forall i=1,..,p$, the Dirichlet distribution returns $\gamma$ where $\gamma_{i}\geq 0$, $\sum_{i=1}^{p}\gamma_{i}=1$. In particular, there is a coefficient $\gamma_{k}$ much greater than other ones with a strong probability, which implies that virtual points are more probably in the neighborhood of a data sample. Since the manifold attack tries to find only local maximum by gradient-based method, if $\xi$ and $n\_iters$ are both small, we expect that attack points in the attack stage do not move too far from their initiated position, remaining on the manifold of data. Note that, in the case $\alpha_{i}=1,\forall i=1,..,p$, the Dirichlet distribution become the Uniform distribution.

To ensure that the coefficient $\gamma_{k}$ is always much greater than other ones, we apply one more constraint: $\gamma_{k}\geq\tau$, and by taking $\tau$ close to 1. The constraints in (9) become:

Then the projection in algorithm 2 needs to be slightly modified to incorporate this new constraint. We define the projection $\gamma=\Pi_{ps}^{{}^{\prime}}(\kappa)$ as follow:

For this experiment, we use the S curve data and Digit data. The S curve data contains $N=1000$ 3-dimensional samples, as shown in figure 3. The Digit data contains $N=1797$ images, of size $8\times 8$ of a digit. We want to compute 2-dimensional embeddings for these data. Each data is separated into two sets: $N_{tr}$ samples are randomly taken for training set and the $N_{te}$ remaining samples are used for testing. $\mathbf{x}^{tr}$ denotes a training sample and $\mathbf{x}^{te}$ denotes a testing sample. We perform four training modes as described in table 1 with a neural network model $g()$. The evaluation loss, after optimizing model $g()$, is defined as:

The anchoring rule, embedding loss $\mathcal{L}_{e}$ and the model $g()$ are precised in the following.

Anchoring rule. Two settings are considered:

• -

  Neighbor anchors (NA): A set of anchor points is composed by a sample with its 4 nearest neighbors. In this case, we have $M=N_{tr}$ sets of anchor points and $p=5$ anchor points in each set. The coefficient $\gamma$ is initialized by uniform distribution.

• -

  Random anchors (RA): We take randomly 2 points among $N_{tr}$ training points to create a set of anchor points. In this case, we have $M=\binom{N_{tr}}{2}$ sets of anchor points and $p=2$ anchor points in each set. The coefficient $\gamma$ is initialized by the Dirichlet distribution with $\alpha_{i}=0.5,\forall i=1,..,p$.

The embedding loss. We employ the embedding loss MDS and LE as described in section 3, with the default metrics. For similarity metric $d_{x}()$ in LE method, we take $\sigma=0.2$ for S curve data and $\sigma=0.5$ for Digit data.

The model. A simple structure of CNN is used. Here are the detailed architectures for each CNN as the dimension of the inputs are different for the two datasets:

• -

  S curve data: Conv1d[$1,4,2$] $\rightarrow$ ReLu $\rightarrow$ Conv1d[$4,4,2$] $\rightarrow$ ReLu$\rightarrow$ Flatten $\rightarrow$ Fc[$4,2$].

• -

  Digit data: Conv2d[$1,8,3$] $\rightarrow$ ReLu $\rightarrow$ Conv2d[$8,16,3$] $\rightarrow$ ReLu$\rightarrow$ Flatten $\rightarrow$ Fc[$64,2$].

For LE method, two additional constraints are imposed to avoid trivial embeddings:

where $d=2$ is the number of output dimensions, $\mathbf{A}^{tr}=[\mathbf{a}_{1}^{tr},..,\mathbf{a}_{N_{tr}}^{tr}]=[g(\mathbf{x}_{1}^{tr}),..,g(\mathbf{x}_{N_{tr}}^{tr})]$. To adapt these constraints, we add a normalization layer at the end of model $g()$: $(g(\mathbf{x})-\mathrm{E}(\mathbf{A}^{tr}))\Sigma^{-1}(\mathbf{A}^{tr},\mathbf{A}^{tr})$, where $\Sigma^{-1}$ is performed by Cholesky decomposition.

To simulate the case of few training samples, we fix $N_{tr}=100$ for MDS method and $N_{tr}=50$ for LE method. The balance between virtual points and samples is controlled by the couple $\lambda=$ (number of virtual points in $\mathcal{B}_{s}$ , number of observed samples in $\mathcal{B}_{s}$). We set $\lambda=(2,5)$ for MDS method and $\lambda=(5,10)$ for LE method. The gradient step $\xi$ is selected from $\{0.1,1,10\}$ and the number of iterations is fixed at $n\_iters=2$.

The initialization of model $g()$ is impactful, especially since there are few training data. Five different initialization of model for each method are performed. The mean and the standard deviation of the evaluation loss $L_{ev}$ are represented in table 2. Firstly, we see that using random virtual (RV) points as additional data points gives a better loss than using only data points. Secondly, using manifold attack (MA) further improves the results which shows the benefit of the proposed approach to regularize the model.

For the S curve data, initialization by Neighbors anchors (NA) gives a better result compared to initialization by Random anchors (RA). However, for the Digit data, initialization by Random anchors gives a better result. This is due to the fact that in the S curve data, Neighbor Anchors covers better the manifold of data than Random Anchor. On the other hand, in Digit data, Neighbor Anchors (by using Euclidean metric to determine nearest neighbors) can generate, with greater probability, a virtual point that is not in the manifold of data. This leads to a greater evaluation loss compared to Random Anchor.

The five embedded representations, respectively with five different initialization of $g()$, for testing samples in S curve data are found in figure 4 for MDS method and in figure 5 for LE method.

In this subsection, we combine manifold attack with supervised learning to assess the adversarial robustness of model. Let start with a general objective function which is created by a supervised loss and a PGS loss :

where $(\mathbf{x}^{l}_{i},y_{i})$ means a sample and its corresponding label. $\mathcal{L}_{c}$ is a dissimilarity metric, like for instance the Cross Entropy. $\mathcal{L}_{t}$ is a PGS loss that includes eventually all observed samples $\mathbf{x}$ and virtual points $\tilde{\mathbf{x}}$. Note that the coordinates $\gamma$ of all virtual points are constrained by (9).

In the following, we construct $\mathcal{L}_{t}$ by a particular PGS loss, called Mix-up manifold learning loss [27, 28, 26] :

In this configuration, the virtual point $\tilde{\mathbf{x}}=\gamma_{1}\mathbf{x}_{i}+\gamma_{2}\mathbf{x}_{j}$, where $\mathbf{x}_{i}$ and $\mathbf{x}_{j}$ are two anchor points. The distribution Beta is just a particular case of the Dirichlet distribution where the number of anchor points $p=2$ (in section 4.4). In case of supervised learning, we perform PGS between the original samples representation and the final NNM output (which can be considered as a latent representation). Thus, we implement a slightly modified version of equation (20):

Then, we replace this explicit PGS loss in equation (5.2) and take $\lambda=1$:

We call problem (22) adversarial Mix-up because this problem is developed from Mix-up[26], by adding an adversarial learning for $\gamma$. In practice, to optimize the problem (22), we perform an attack stage (algorithm 4) to find $\gamma$ that gives the greater PGS loss, before performing model update stage for $f_{\theta}()$. We repeat alternatively these two stages until convergence. Note that, we can take $\gamma_{2}=1-\gamma_{1}$, so that we only need to deal with one variable $\gamma_{1}$ to maximize PGS loss. The projection (12) for $\gamma_{1}$ is now just the clamping function, to make sure that $\gamma_{1}$ is between 0 and 1.

We compare four supervised training methods: ERM (Empirical Risk Minimization, which is thus classical supervised learning with Cross Entropy loss), Mix-up, Adversarial Mix-up and Cut-Mix [29] on ImageNet dataset with the model ResNet-50 [6], which has about 25.8M trainable parameters. We use ImageNet dataset. We retrieve 948 classes consisting in 400 labelled training samples and 50 testing samples to evaluate the models.

We evaluate the error rate for testing set at the end of each epoch and report the best best error rate (top-1 and top-5) in table 3. We create adversarial examples using Fast Gradient Sign Method (FGSM) [10], on another trained ERM model, with $\epsilon=0.05$. In Adversarial Mix-up, $n\_iters$ is fixed at 1. The attack stage is parametrized by the gradient step $\xi$, which is set up following two configurations, (1) $\xi$ is reduced linearly from 0.1 to 0.01 and (2) $\xi$ is reduced linearly from 0.01 to 0.001. Following original articles, $\alpha$ is set at 0.2 for Mix-Up and Adversarial Mix-up and $\alpha=1$ for Cut-Mix. More details for hyper-parameters can be found in Appendix.

Firstly, Mix-up [26] is a combination between supervised learning task and PGS task loss and it gives a better error rates for both testing set and adversarial examples compared to ERM, which is a standard supervised learning task.

Secondly, in Adversarial Mix-up (1) and (2), a trade-off has to be found between error rate for testing set and error rate for adversarial examples. For strong values of $\xi$ (1), the error rate for testing sample can be about 0.5% worse than Mix-up (without using attack stage), but it gains about 13% for the robustness against adversarial examples. On the other hand, if $\xi$ takes moderate values as in (2), error rates for both testing sample and adversarial examples are smaller than those of Mix-Up, but it gains only about 5% for the robustness against adversarial examples. These two observations show the effect of manifold attack, which is an improved PGS procedure. We conclude that manifold attack not only improves generalization but also significantly improves adversarial robustness of the model.

Finally, Adversarial Mix-up Adversarial (2) provides a slightly worse error rate than Cut-Mix (about 1 % in the case of testing sample), but it gains about 10% in the case of adversarial examples.

It is worth noting that in attack stage, the model $g()$ needs to be differentiable. Thus, in the case of using NNMs with Dropout layer for example , the active connections need to be fixed during an attack stage.

In this subsection, manifold attack is applied to reinforce semi-supervised neural network models that use PGS as regularization. The problem (5.2) is extended to include unlabelled samples. We assume that the training set contains $N$ samples $\mathbf{x}$, the $N_{l}$ first samples $\mathbf{x}^{l}_{i}$ are labelled and the remaining $N_{u}=N-N_{l}$ samples $\mathbf{x}^{u}_{i}$ are unlabelled. Hence, the objective function for a semi-supervised manifold attack method:

where $L_{u}$ refers to other possible unsupervised losses such as pseudo-label, self-supervised learning, etc.

In the following, we explicit the terms of the problem (26). We build upon MixMatch [30], a semi-supervised learning method in the current state-of-the-art. MixMatch use Mix-up manifold learning loss for PGS. Secondly, we add an adversarial learning for $\gamma$ in MixMatch. Hence we refer to the resulting instance of 26 as Adversarial MixMatch. Note that, $\gamma_{1}$ in MixMatch is slightly different from Mix-up as:

Then the projection for $\gamma_{1}$ is now the clamping function between 0.5 and 1. When two separated variables $\gamma_{1}$ and $\gamma_{2}$ are defined, we can use the projector $\Pi_{ps}^{{}^{\prime}}$ (16) defined in subsection 4.4. We use the Pytorch implementation for MixMatch by Yui [31] (with all hyper-parameters by default), then we introduce attack stages, with the number of iterations $n\_iters=1$. In each experiment, for both CIFAR-10 and SVHN dataset, we divide the training set into three parts: labelled set, unlabelled set and validation set. The number of validation samples is fixed at 5000. The number of labelled samples is 250, and the remaining samples are part of the unlabelled set. We repeat the experiment four times, with different samplings of labelled samples, unlabelled samples, validation samples and different initialization of model Wide ResNet-28 [32] which has about 1.47M trainable parameters. More details for hyper-parameters can be found in Appendix.

The error rate on testing set, which corresponds to the best validation error rate, is reported in table 4, for both MixMatch and MixMatch Adersarial. We see that MixMatch Adersarial improves the performance of MixMatch, about $1.5\%$ less on error rate. There is a considerable difference between the error rate of MixMatch reproduced by our experiments and the one reported from the official paper, which might come from the sampling, the initialization of model, the library used (Pytorch vs TensorFlow) and the computation of the error rate (error rate associated to best validation error vs the median error rate of the last 20 checkpoints).