Mixed Nash Equilibria in the Adversarial Examples Game

Consider the binary classification task illustrated in Figure 1. We assume that all input-output pairs $(X,Y)$ are sampled from a distribution $\mathbb{P}$ defined as follows

$$\mathbb{P}\left(Y=\pm 1\right)=1/2\ \mbox{ and }\left\{\begin{array}[]{ll}\mathbb{P}\left(X=0\mid Y=-1\right)=1\\ \mathbb{P}\left(X=\pm 1\mid Y=1\right)=1/2\end{array}\right.$$

Given access to $\mathbb{P}$, the adversary aims to maximize the expected risk, but can only move each point by at most $1$ on the real line. In this context, we study two classifiers: $f_{1}(x)=-x-1/2$ and $f_{2}(x)=x-1/2$¹¹1$(X,Y)\sim\mathbb{P}$ is misclassified by $f_{i}$ if and only if $f_{i}(X)Y\leq 0$. Both $f_{1}$ and $f_{2}$ have a standard risk of $1/4$. In the presence of an adversary, the risk (a.k.a. the adversarial risk) increases to $1$. Here, using a randomized classifier can make the system more robust. Consider $f$ where $f=f_{1}$ w.p. $1/2$ and $f_{2}$ otherwise. The standard risk of $f$ remains $1/4$ but its adversarial risk is $3/4<1$. Indeed, when attacking $f$, any adversary will have to choose between moving points from $0$ to $1$ or to $-1$. Either way, the attack only works half of the time; hence an overall adversarial risk of $3/4$. Furthermore, if $f$ knows the strategy the adversary uses, it can always update the probability it gives to $f_{1}$ and $f_{2}$ to get a better (possibly deterministic) defense. For example, if the adversary chooses to always move $0$ to $1$, the classifier can set $f=f_{1}$ w.p. $1$ to retrieve an adversarial risk of $1/2$ instead of $3/4$.

Now, what happens if the adversary can use randomized strategies, meaning that for each point it can flip a coin before deciding where to move? In this case, the adversary could decide to move points from $0$ to $1$ w.p. $1/2$ and to $-1$ otherwise. This strategy is still optimal with an adversarial risk of $3/4$ but now the classifier cannot use its knowledge of the adversary’s strategy to lower the risk. We are in a state where neither the adversary nor the classifier can benefit from unilaterally changing its strategy. In the game theory terminology, this state is called a Mixed Nash equilibrium.

Let us consider a classification task with input space $\mathcal{X}$ and output space $\mathcal{Y}$. Let $(\mathcal{X},d)$ be a proper (i.e. closed balls are compact) Polish (i.e. completely separable) metric space representing the inputs space²²2For instance, for any norm $\lVert\cdot\rVert$, $(\mathbb{R}^{d},\lVert\cdot\rVert)$ is a proper Polish metric space.. Let $\mathcal{Y}=\{1,\dots,K\}$ be the labels set, endowed with the trivial metric $d^{\prime}(y,y^{\prime})=\mathbf{1}_{y\neq y^{\prime}}$. Then the space $(\mathcal{X}\times\mathcal{Y},d\oplus d^{\prime})$ is a proper Polish space. For any Polish space $\mathcal{Z}$, we denote $\mathcal{M}_{+}^{1}(\mathcal{Z})$ the Polish space of Borel probability measures on $\mathcal{Z}$. Let us assume the data is drawn from $\mathbb{P}\in\mathcal{M}_{+}^{1}(\mathcal{X}\times\mathcal{Y})$. Let $(\Theta,d_{\Theta})$ be a Polish space (not necessarily proper) representing the set of classifier parameters (for instance neural networks). We also define a loss function: $l:\Theta\times(\mathcal{X}\times\mathcal{Y})\to[0,\infty)$ satisfying the following set of assumptions.

It is usual to assume upper-semi continuity when studying optimization over distributions [38, 7]. Furthermore, considering bounded (and positive) loss functions is also very common in learning theory [2] and is not restrictive.

In the adversarial examples framework, the loss of interest is the $0/1$ loss, for whose surrogates are misunderstood [14, 1]; hence it is essential that the $0/1$ loss satisfies Assumption 1. In the binary classification setting (i.e. $\mathcal{Y}=\{-1,+1\}$) the $0/1$ loss writes $l_{0/1}(\theta,(x,y))=\mathbf{1}_{yf_{\theta}(x)\leq 0}$. Then, assuming that for all $\theta$, $f_{\theta}(\cdot)$ is continuous and for all $x$, $f_{\cdot}(x)$ is continuous, the $0/1$ loss satisfies Assumption 1. In particular, it is the case for neural networks with continuous activation functions.

The standard risk for a single classifier $\theta$ associated with the loss $l$ satisfying Assumption 1 writes: $\mathcal{R}(\theta):=\mathbb{E}_{(x,y)\sim\mathbb{P}}\left[l(\theta,(x,y))\right]$. Similarly, the adversarial risk of $\theta$ at level $\varepsilon$ associated with the loss $l$ is defined as³³3For the well-posedness, see Lemma 4 in Appendix.

$\displaystyle\mathcal{R}_{adv}^{\varepsilon}(\theta):=\mathbb{E}_{(x,y)\sim\mathbb{P}}\left[\sup_{x^{\prime}\in\mathcal{X},~{}d(x,x^{\prime})\leq\varepsilon}l(\theta,(x^{\prime},y))\right].$

It is clear that $\mathcal{R}_{adv}^{0}(\theta)=\mathcal{R}(\theta)$ for all $\theta$. We can generalize these notions with distributions of classifiers. In other terms the classifier is then randomized according to some distribution $\mu\in\mathcal{M}^{1}_{+}(\Theta)$. A classifier is randomized if for a given input, the output of the classifier is a probability distribution. The standard risk of a randomized classifier $\mu$ writes $\mathcal{R}(\mu)=\mathbb{E}_{\theta\sim\mu}\left[\mathcal{R}(\theta)\right]$. Similarly, the adversarial risk of the randomized classifier $\mu$ at level $\varepsilon$ is⁴⁴4This risk is also well posed (see Lemma 4 in the Appendix).

$\displaystyle\mathcal{R}_{adv}^{\varepsilon}(\mu):=\mathbb{E}_{(x,y)\sim\mathbb{P}}\left[\sup_{x^{\prime}\in\mathcal{X},~{}d(x,x^{\prime})\leq\varepsilon}\mathbb{E}_{\theta\sim\mu}\left[l(\theta,(x^{\prime},y))\right]\right].$

For instance, for the $0/1$ loss, the inner maximization problem, consists in maximizing the probability of misclassification for a given couple $(x,y)$. Note that $\mathcal{R}(\delta_{\theta})=\mathcal{R}(\theta)$ and $\mathcal{R}_{adv}^{\varepsilon}(\delta_{\theta})=\mathcal{R}_{adv}^{\varepsilon}(\theta)$. In the remainder of the paper, we study the adversarial risk minimization problems with randomized and deterministic classifiers and denote

$\displaystyle\mathcal{V}_{rand}^{\varepsilon}:=\inf_{\mu\in\mathcal{M}^{1}_{+}(\Theta)}\mathcal{R}_{adv}^{\varepsilon}(\mu),~{}\mathcal{V}_{det}^{\varepsilon}:=\inf_{\theta\in\Theta}\mathcal{R}_{adv}^{\varepsilon}(\theta)$

(1)

To account for the possible randomness of the adversary, we rewrite the adversarial attack problem as a convex optimization problem on distributions. Let us first introduce the set of adversarial distributions.

For an attacker that can move the initial distribution $\mathbb{P}$ in $\mathcal{A}_{\varepsilon}(\mathbb{P})$, the attack would not be a transport map as considered in the standard adversarial risk. For every point $x$ in the support of $\mathbb{P}$, the attacker is allowed to move $x$ randomly in the ball of radius $\varepsilon$, and not to a single other point $x^{\prime}$ like the usual attacker in adversarial attacks. In this sense, we say the attacker is allowed to be randomized.

Link with DRO. Adversarial examples have been studied in the light of DRO by former works [33, 37], but an exact reformulation of the adversarial risk as a DRO problem has not been made yet. When $(\mathcal{Z},d)$ is a Polish space and $c:\mathcal{Z}^{2}\rightarrow\mathbb{R}^{+}\cup\{+\infty\}$ is a lower semi-continuous function, for $\mathbb{P},\mathbb{Q}\in\mathcal{M}^{+}_{1}(\mathcal{Z})$ , the primal Optimal Transport problem is defined as

$\displaystyle W_{c}(\mathbb{P},\mathbb{Q}):=\inf_{\gamma\in\Gamma_{\mathbb{P},\mathbb{Q}}}\int_{\mathcal{Z}^{2}}c(z,z^{\prime})d\gamma(z,z^{\prime})$

with $\Gamma_{\mathbb{P},\mathbb{Q}}:=\left\{\gamma\in\mathcal{M}^{+}_{1}(\mathcal{Z}^{2})\mid~{}\Pi_{1\sharp}\gamma=\mathbb{P},~{}\Pi_{2\sharp}\gamma=\mathbb{Q}\right\}$. When $\eta>0$ and for $\mathbb{P}\in\mathcal{M}^{+}_{1}(\mathcal{Z})$, the associated Wasserstein uncertainty set is defined as:

$\displaystyle\mathcal{B}_{c}(\mathbb{P},\eta):=\left\{\mathbb{Q}\in\mathcal{M}^{+}_{1}(\mathcal{Z})\mid W_{c}(\mathbb{P},\mathbb{Q})\leq\eta\right\}$

A DRO problem is a linear optimization problem over Wasserstein uncertainty sets $\sup_{\mathbb{Q}\in\mathcal{B}_{c}(\mathbb{P},\eta)}\int g(z)d\mathbb{Q}(z)$ for some upper semi-continuous function $g$ [41]. For an arbitrary $\varepsilon>0$, we define the cost $c_{\varepsilon}$ as follows

$\displaystyle c_{\varepsilon}((x,y),(x^{\prime},y^{\prime})):=\left\{\begin{array}[]{ll}0&\mbox{if }d(x,x^{\prime})\leq\varepsilon\mbox{ and }y=y^{\prime}\\ +\infty&\mbox{otherwise.}\end{array}\right.$

This cost is lower semi-continuous and penalizes to infinity perturbations that change the label or move the input by a distance greater than $\varepsilon$. As Proposition 1 shows, the Wasserstein ball associated with $c_{\varepsilon}$ is equal to $\mathcal{A}_{\varepsilon}(\mathbb{P})$.

Thanks to this result, we can reformulate the adversarial risk as the value of a convex problem over $\mathcal{A}_{\varepsilon}(\mathbb{P})$.

The adversarial attack problem is a DRO problem for the cost $c_{\varepsilon}$. Proposition 2 means that, against a fixed classifier $\mu$, the randomized attacker that can move the distribution in $\mathcal{A}_{\varepsilon}(\mathbb{P})$ has exactly the same power as an attacker that moves every single point $x$ in the ball of radius $\varepsilon$. By Proposition 2, we also deduce that the adversarial risk can be casted as a linear optimization problem over distributions.

Thanks to Proposition 2, the adversarial risk minimization problem can be seen as a two-player zero-sum game that writes as follows,

$\displaystyle\inf_{\mu\in\mathcal{M}^{1}_{+}(\Theta)}\sup_{\mathbb{Q}\in\mathcal{A}_{\varepsilon}(\mathbb{P})}\mathbb{E}_{(x,y)\sim\mathbb{Q},\theta\sim\mu}\left[l(\theta,(x,y))\right].$

(3)

In this game the classifier objective is to find the best distribution $\mu\in\mathcal{M}_{1}^{+}(\Theta)$ while the adversary is manipulating the data distribution. For the classifier, solving the infimum problem in Equation (3) simply amounts to solving the adversarial risk minimization problem – Problem (1), whether the classifier is randomized or not. Then, given a randomized classifier $\mu\in\mathcal{M}_{1}^{+}(\Theta)$, the goal of the attacker is to find a new data-set distribution $\mathbb{Q}$ in the set of adversarial distributions $\mathcal{A}_{\varepsilon}(\mathbb{P})$ that maximizes the risk of $\mu$. More formally, the adversary looks for

$$\mathbb{Q}\in\operatorname*{argmax}_{\mathbb{Q}\in\mathcal{A}_{\varepsilon}(\mathbb{P})}\mathbb{E}_{(x,y)\sim\mathbb{Q},\theta\sim\mu}\left[l(\theta,(x,y))\right].$$

In the game theoretic terminology, $\mathbb{Q}$ is also called the best response of the attacker to the classifier $\theta$.

Every zero sum game has a dual formulation that allows for a deeper understanding of the framework. Here, from Proposition 2, we can define the dual problem of adversarial risk minimization for randomized classifiers. This dual problem also characterizes a two-player zero-sum game that writes as follows,

$\displaystyle\sup_{\mathbb{Q}\in\mathcal{A}_{\varepsilon}(\mathbb{P})}\inf_{\mu\in\mathcal{M}^{1}_{+}(\Theta)}\mathbb{E}_{(x,y)\sim\mathbb{Q},\theta\sim\mu}\left[l(\theta,(x,y))\right].$

(4)

In this dual game problem, the adversary plays first and seeks an adversarial distribution that has the highest possible risk when faced with an arbitrary classifier. This means that it has to select an adversarial perturbation for every input $x$, without seeing the classifier first. In this case, as pointed out by the motivating example in Section 2.1, the attack can (and should) be randomized to ensure maximal harm against several classifiers. Then, given an adversarial distribution, the classifier objective is to find the best possible classifier on this distribution. Let us denote $\mathcal{D}^{\varepsilon}$ the value of the dual problem. Since the weak duality is always satisfied, we get

$\displaystyle\mathcal{D}^{\varepsilon}\leq\mathcal{V}_{rand}^{\varepsilon}\leq\mathcal{V}_{det}^{\varepsilon}.$

(5)

Inequalities in Equation (5) mean that the lowest risk the classifier can get (regardless of the game we look at) is $\mathcal{D}^{\varepsilon}$. In particular, this means that the primal version of the game, i.e. the adversarial risk minimization problem, will always have a value greater or equal to $\mathcal{D}^{\varepsilon}$. As we discussed in Section 2.1, this lower bound may not be attained by a deterministic classifier. As we will demonstrate in the next section, optimizing over randomized classifiers allows to approach $\mathcal{D}^{\varepsilon}$ arbitrary closely.

In the adversarial examples game, a Nash equilibrium is a couple $(\mu^{*},\mathbb{Q}^{*})\in\mathcal{M}^{1}_{+}(\Theta)\times\mathcal{A}_{\varepsilon}(\mathbb{P})$ where both the classifier and the attacker have no incentive to deviate unilaterally from their strategies $\mu^{*}$ and $\mathbb{Q}^{*}$. More formally, $(\mu^{*},\mathbb{Q}^{*})$ is a Nash equilibrium of the adversarial examples game if $(\mu^{*},\mathbb{Q}^{*})$ is a saddle point of the objective function

$$(\mu,\mathbb{Q})\mapsto\mathbb{E}_{(x,y)\sim\mathbb{Q},\theta\sim\mu}\left[l(\theta,(x,y))\right].$$

Alternatively, we can say that $(\mu^{*},\mathbb{Q}^{*})$ is a Nash equilibrium if and only if $\mu^{*}$ solves the adversarial risk minimization problem – Problem (1), $\mathbb{Q}^{*}$ the dual problem – Problem (6), and $\mathcal{D}^{\varepsilon}=\mathcal{V}_{rand}^{\varepsilon}$. In our problem, $\mathbb{Q}^{*}$ always exists but it might not be the case for $\mu^{*}$. Then for any $\delta>0$, we say that $(\mu_{\delta},\mathbb{Q}^{*})$ is a $\delta$-approximate Nash equilibrium if $\mathbb{Q}^{*}$ solves the dual problem and $\mu_{\delta}$ satisfies $\mathcal{D}^{\varepsilon}\geq\mathcal{R}_{adv}^{\varepsilon}(\mu_{\delta})-\delta$.

We now state our main result: the existence of approximate Nash equilibria in the adversarial examples game when both the classifier and the adversary can use randomized strategies. More precisely, we demonstrate that the duality gap between the adversary and the classifier problems is zero, which gives as a corollary the existence of Nash equilibria.

Theorem 1 shows that $\mathcal{D}^{\varepsilon}=\mathcal{V}_{rand}^{\varepsilon}$. From a game theoretic perspective, this means that the minimal adversarial risk for a randomized classifier against any attack (primal problem) is the same as the maximal risk an adversary can get by using an attack strategy that is oblivious to the classifier it faces (dual problem). This suggests that playing randomized strategies for the classifier could substantially improve robustness to adversarial examples. In the next section, we will design an algorithm that efficiently learn this classifier, we will get improve adversarial robustness over classical deterministic defenses.

Let $\{(x_{i},y_{i})\}_{i=1}^{N}$ samples independently drawn from $\mathbb{P}$ and denote $\widehat{\mathbb{P}}:=\frac{1}{N}\sum_{i=1}^{N}\delta_{(x_{i},y_{i})}$ the associated empirical distribution. One can show the adversarial empirical risk minimization can be casted as:

$\displaystyle\widehat{\mathcal{R}}_{adv}^{\varepsilon,*}:=\inf_{\mu\in\mathcal{M}^{+}_{1}(\Theta)}\sum_{i=1}^{N}\sup_{\mathbb{Q}_{i}\in\Gamma_{i,\varepsilon}}\mathbb{E}_{(x,y)\sim\mathbb{Q}_{i},\theta\sim\mu}\left[l(\theta,(x,y))\right]$

where $\Gamma_{i,\varepsilon}$ is defined as :

$\displaystyle\Gamma_{i,\varepsilon}:=\Big{\{}\mathbb{Q}_{i}\mid~{}\int d\mathbb{Q}_{i}=\frac{1}{N},~{}\int c_{\varepsilon}((x_{i},y_{i}),\cdot)d\mathbb{Q}_{i}=0\Big{\}}.$

More details on this decomposition are given in Appendix E. In the following, we regularize the above objective by adding an entropic term to each inner supremum problem. Let $\bm{\alpha}:=(\alpha_{i})_{i=1}^{N}\in\mathbb{R}_{+}^{N}$ such that for all $i\in\{1,\dots,N\}$, and let us consider the following optimization problem:

	$\displaystyle\widehat{\mathcal{R}}_{adv,\bm{\alpha}}^{\varepsilon,*}:=\inf_{\mu\in\mathcal{M}^{+}_{1}(\Theta)}\sum_{i=1}^{N}$	$\displaystyle\sup_{\mathbb{Q}_{i}\in\Gamma_{i,\varepsilon}}\mathbb{E}_{\mathbb{Q}_{i},\mu}\left[l(\theta,(x,y))\right]$
		$\displaystyle-\alpha_{i}\text{KL}\left(\mathbb{Q}_{i}\Big{|}\Big{|}\frac{1}{N}\mathbb{U}_{(x_{i},y_{i})}\right)$

where $\mathbb{U}_{(x,y)}$ is an arbitrary distribution of support equal to:

$\displaystyle S_{(x,y)}^{(\varepsilon)}:=\Big{\{}(x^{\prime},y^{\prime}):~{}\text{s.t.}~{}c_{\varepsilon}((x,y),(x^{\prime},y^{\prime}))=0\Big{\}},$

and for all $\mathbb{Q},\mathbb{U}\in\mathcal{M}_{+}(\mathcal{X}\times\mathcal{Y})$,

$\displaystyle\text{KL}(\mathbb{Q}||\mathbb{U}):=\left\{\begin{array}[]{lll}\int\log(\frac{d\mathbb{Q}}{d\mathbb{U}})d\mathbb{Q}+|\mathbb{U}|-|\mathbb{Q}|&\mbox{if }\mathbb{Q}\ll\mathbb{U}\\ +\infty&\mbox{otherwise.}\end{array}\right.$

Note that when $\bm{\alpha}=0$, we recover the problem of interest $\widehat{\mathcal{R}}_{adv,\bm{\alpha}}^{\varepsilon,*}=\widehat{\mathcal{R}}_{adv}^{\varepsilon,*}$. Moreover, we show the regularized supremum tends to the standard supremum when $\bm{\alpha}\to 0$.

By adding an entropic term to the objective, we obtain an explicit formulation of the supremum involved in the sum: as soon as $\bm{\alpha}>0$ (which means that each $\alpha_{i}>0$), each sub-problem becomes just the Fenchel-Legendre transform of $\text{KL}(\cdot|\mathbb{U}_{(x_{i},y_{i})}/N)$ which has the following closed form:

	$\displaystyle\sup_{\mathbb{Q}_{i}\in\Gamma_{i,\varepsilon}}\mathbb{E}_{\mathbb{Q}_{i},\mu}\left[l(\theta,(x,y))\right]-\alpha_{i}\text{KL}\left(\mathbb{Q}_{i}||\frac{1}{N}\mathbb{U}_{(x_{i},y_{i})}\right)$
	$\displaystyle=\frac{\alpha_{i}}{N}\log\left(\int_{\mathcal{X}\times\mathcal{Y}}\exp\left(\frac{\mathbb{E}_{\theta\sim\mu}\left[l(\theta,(x,y))\right]}{\alpha_{i}}\right)d\mathbb{U}_{(x_{i},y_{i})}\right).$

Finally, we end up with the following problem:

$\displaystyle\inf_{\mu\in\mathcal{M}^{+}_{1}(\Theta)}\sum_{i=1}^{N}\frac{\alpha_{i}}{N}\log\left(\int\exp\frac{\mathbb{E}_{\mu}\left[l(\theta,(x,y))\right]}{\alpha_{i}}d\mathbb{U}_{(x_{i},y_{i})}\right).$

In order to solve the above problem, one needs to compute the integral involved in the objective. To do so, we estimate it by randomly sampling $m_{i}\geq 1$ samples $(u_{1}^{(i)},\dots,u_{m_{i}}^{(i)})\in(\mathcal{X}\times\mathcal{Y})^{m_{i}}$ from $\mathbb{U}_{(x_{i},y_{i})}$ for all $i\in\{1,\dots,N\}$ which leads to the following optimization problem

$\displaystyle\inf_{\mu\in\mathcal{M}^{+}_{1}(\Theta)}\sum_{i=1}^{N}\frac{\alpha_{i}}{N}\log\left(\frac{1}{m_{i}}\sum_{j=1}^{m_{i}}\exp\frac{\mathbb{E}_{\mu}\left[l(\theta,u_{j}^{(i)})\right]}{\alpha_{i}}\right)$

(7)

denoted $\widehat{\mathcal{R}}_{adv,\bm{\alpha}}^{\varepsilon,\bm{m}}$ where $\bm{m}:=(m_{i})_{i=1}^{N}$ in the following. Now we aim at controlling the error made with our approximations. We decompose the error into two terms

$\displaystyle|\widehat{\mathcal{R}}_{adv,\bm{\alpha}}^{\varepsilon,\bm{m}}-\widehat{\mathcal{R}}_{adv}^{\varepsilon,*}|\leq|\widehat{\mathcal{R}}_{adv,\bm{\alpha}}^{\varepsilon,*}-\widehat{\mathcal{R}}_{adv,\bm{\alpha}}^{\varepsilon,\bm{m}}|+|\widehat{\mathcal{R}}_{adv,\bm{\alpha}}^{\varepsilon,*}-\widehat{\mathcal{R}}_{adv}^{\varepsilon,*}|$

where the first one corresponds to the statistical error made by our estimation of the integral, and the second to the approximation error made by the entropic regularization of the objective. First, we show a control of the statistical error using Rademacher complexities [2].

We deduce from the above Proposition that in the particular case where $\Theta$ is finite such that $|\Theta|=L$, with probability of at least $1-\delta$

$\displaystyle|\widehat{\mathcal{R}}_{adv,\bm{\alpha}}^{\varepsilon,*}-\widehat{\mathcal{R}}_{adv,\bm{\alpha}}^{\varepsilon,\bm{m}}|\in\mathcal{O}\left(Me^{M/\alpha}\sqrt{\frac{\log(L)}{m}}\right).$

This case is of particular interest when one wants to learn the optimal mixture of some given classifiers in order to minimize the adversarial risk. In the following proposition, we control the approximation error made by adding an entropic term to the objective.

The assumption made in the above Proposition states that for any given random classifier $\mu$, and any given point $(x,y)$, the set of $\beta$-optimal attacks at this point has at least a certain amount of mass depending on the $\beta$ chosen. This assumption is always met when $\beta$ is sufficiently large. However in order to obtain a tight control of the error, a trade-off exists between $\beta$ and the smallest amount of mass $C_{\beta}$ of $\beta$-optimal attacks.

Now that we have shown that solving (7) allows to obtain an approximation of the true solution $\widehat{\mathcal{R}}_{adv}^{\varepsilon,*}$, we next aim at deriving an algorithm to compute it.

From now on, we focus on finite class of classifiers. Let $\Theta=\{\theta_{1},\dots,\theta_{L}\}$, we aim to learn the optimal mixture of classifiers in this case. The adversarial empirical risk is therefore defined as:

$\displaystyle\widehat{\mathcal{R}}_{adv}^{\varepsilon}(\bm{\lambda})=\sum_{i=1}^{N}\sup_{\mathbb{Q}_{i}\in\Gamma_{i,\varepsilon}}\mathbb{E}_{(x,y)\sim\mathbb{Q}_{i}}\left[\sum_{k=1}^{L}\lambda_{k}l(\theta_{k},(x,y))\right]$

for $\bm{\lambda}\in\Delta_{L}:=\{\bm{\lambda}\in\mathbb{R}_{+}^{L}~{}\mathrm{s.t.}~{}\sum_{i=1}^{L}\lambda_{i}=1\}$, the probability simplex of $\mathbb{R}^{L}$. One can notice that $\widehat{\mathcal{R}}_{adv}^{\varepsilon}(\cdot)$ is a continuous convex function, hence $\min_{\bm{\lambda}\in\Delta_{L}}\mathcal{R}_{adv}^{\varepsilon}(\bm{\lambda})$ is attained for a certain $\bm{\lambda}^{*}$. Then there exists a non-approximate Nash equilibrium $(\bm{\lambda}^{*},\mathbb{Q}^{*})$ in the adversarial game when $\Theta$ is finite. Here, we present two algorithms to learn the optimal mixture of the adversarial risk minimization problem.

A First Oracle Algorithm. The first algorithm we present is inspired from [33] and the convergence of projected sub-gradient methods [9]. The computation of the inner supremum problem is usually NP-hard ⁶⁶6See Appendix E for details., but one may assume the existence of an approximate oracle to this supremum. The algorithm is presented in Algorithm 1. We get the following guarantee for this algorithm.

The main drawback of the above algorithm is that one needs to have access to an oracle to guarantee the convergence of the proposed algorithm. In the following we present its regularized version in order to approximate the solution and propose a simple algorithm to solve it.

An Entropic Relaxation. Adding an entropic term to the objective allows to have a simple reformulation of the problem, as follows:

$\displaystyle\inf_{\bm{\lambda}\in\Delta_{L}}\sum_{i=1}^{N}\frac{\varepsilon_{i}}{N}\log\left(\frac{1}{m_{i}}\sum_{j=1}^{m_{i}}\exp\left(\frac{\sum_{k=1}^{L}\lambda_{k}l(\theta_{k},u_{j}^{(i)})}{\varepsilon_{i}}\right)\right)$

Note that in $\bm{\lambda}$, the objective is convex and smooth. One can apply the accelerated PGD [3, 36] which enjoys an optimal convergence rate for first order methods of $\mathcal{O}(T^{-2})$ for $T$ iterations.

To illustrate our theoretical findings, we start by testing our learning algorithm on the following synthetic two-dimensional problem. Let us consider the distribution $\mathbb{P}$ defined as $\mathbb{P}\left(Y=\pm 1\right)=1/2$, $\mathbb{P}\left(X\mid Y=-1\right)=\mathcal{N}(0,I_{2})$ and $\mathbb{P}\left(X\mid Y=1\right)=\frac{1}{2}\left[\mathcal{N}((-3,0),I_{2})+\mathcal{N}((3,0),I_{2})\right]$. We sample $1000$ training points from this distribution and randomly generate $10$ linear classifiers that achieves a standard training risk lower than $0.4$. To simulate an adversary with budget $\varepsilon$ in $\ell_{2}$ norm, we proceed as follows. For every sample $(x,y)\sim\mathbb{P}$ we generate $1000$ points uniformly at random in the ball of radius $\varepsilon$ and select the one maximizing the risk for the $0/1$ loss. Figure 2 (left) illustrates the type of mixture we get after convergence of our algorithms. Note that in this toy problem, we are likely to find the optimal adversary with this sampling strategy if we sample enough attack points.

To evaluate the convergence of our algorithms, we compute the adversarial risk of our mixture for each iteration of both the oracle and regularized algorithms. Figure 2 illustrates the convergence of the algorithms w.r.t the regularization parameter. We observe that the risk for both algorithms converge. Moreover, they converge towards the oracle minimizer when the regularization parameter $\alpha$ goes to $0$.

Finally, to demonstrate the improvement randomized techniques offer against deterministic defenses, we plot in Figure 2 (right) the minimum adversarial risk for both randomized and deterministic classifiers w.r.t. $\varepsilon$. The adversarial risk is strictly better for randomized classifier whenever the adversarial budget $\varepsilon$ is bigger than $2$. This illustration validates our analysis of Theorem 1, and motivates a in depth study of a more challenging framework, namely image classification with neural networks.

Adversarial examples are known to be easily transferrable from one model to another [35]. To counter this and support our theoretical claims, we propose an heuristic algorithm (see Algorithm 2) to train a robust mixture of $L$ classifiers. We alternatively train these classifiers with adversarial examples against the current mixture and update the probabilities of the mixture according to the algorithms we proposed in Section 4.2. More details on the heuristic algorithm are available in Appendix D.

Experimental Setup. To evaluate the performance of Algorithm 2, we trained from $1$ to $4$ ResNet18 [20] models on $200$ epochs per model⁷⁷7$L\times 200$ epochs in total, where $L$ is the number of models.. We study the robustness with regards to $\ell_{\infty}$ norm and fixed adversarial budget $\varepsilon=8/255$. The attack we used in the inner maximization of the training is an adapted (adaptative) version of PGD for mixtures of classifiers with $10$ steps. Note that for one single model, Algorithm 2 exactly corresponds to adversarial training [24]. For each of our setups, we made two independent runs and select the best one. The training time of our algorithm is around four times longer than a standard Adversarial Training (with PGD 10 iter.) with two models, eight times with three models and twelve times with four models. We trained our models with a batch of size $1024$ on $8$ Nvidia V100 GPUs. We give more details on implementation in Appendix D.

Evaluation Protocol. At each epoch, we evaluate the current mixture on test data against PGD attack with $20$ iterations. To select our model and avoid overfitting [30], we kept the most robust against this PGD attack. To make a final evaluation of our mixture of models, we used an adapted version of AutoPGD untargeted attacks [15] for randomized classifiers with both Cross-Entropy (CE) and Difference of Logits Ratio (DLR) loss. For both attacks, we made $100$ iterations and $5$ restarts.

Results. The results are presented in Figure 3. We remark our algorithm outperforms a standard adversarial training in all the cases by more $1\%$, without additional loss of standard accuracy as it is attested by the left figure. Moreover, it seems our algorithm, by adding more and more models, reduces the overfitting of adversarial training. So far, experiments are computationally very costful and it is difficult to raise precise conclusions. Further, hyperparameter tuning  [19] such as architecture, unlabeled data [12], activation function, or the use of TRADES [43] may still increase the results.