Adversarial Surrogate Risk Bounds
for Binary Classification

Natalie Frank natalief@uw.edu
Department of Applied Mathematics
University of Washington

Abstract

A central concern in classification is the vulnerability of machine learning models to adversarial attacks. Adversarial training is one of the most popular techniques for training robust classifiers, which involves minimizing an adversarial surrogate risk. Recent work characterized when a minimizing sequence of an adversarial surrogate risk is also a minimizing sequence of the adversarial classification risk for binary classification— a property known as adversarial consistency. However, these results do not address the rate at which the adversarial classification risk converges to its optimal value for such a sequence of functions that minimize the adversarial surrogate. This paper provides surrogate risk bounds that quantify that convergence rate. Additionally, we derive distribution-dependent surrogate risk bounds in the standard (non-adversarial) learning setting, that may be of independent interest.

1 Introduction

A central concern regarding sophisticated machine learning models is their susceptibility to adversarial attacks. Prior work (Biggio et al., 2013; Szegedy et al., 2013) demonstrated that imperceptible perturbations can derail the performance of neural nets. As such models are used in security-critical applications such as facial recognition (Xu et al., 2022) and medical imaging (Paschali et al., 2018), training robust models remains a central concern in machine learning.

In the standard classification setting, the classification risk is the proportion of incorrectly classified data. Rather that minimizing this quantity directly, which is a combinatorial optimization problem, typical machine learning algorithms perform gradient descent on a well-behaved alternative surrogate risk. If a sequence of functions that minimizes this surrogate risk also minimizes the classification risk, then the surrogate risk is referred to as consistent for that specific data distribution. In addition to consistency, one would hope that minimizing the surrogate risk would be an efficient method for minimizing the classification risk. This convergence rate can be bounded by surrogate risk bounds, which are functions that provide a bound on the excess classification risk in terms of the excess surrogate risk.

In the standard binary classification setting, consistency and surrogate risk bounds are well-studied topics (Bartlett et al., 2006; Lin, 2004; Steinwart, 2007; Zhang, 2004). On the other hand, fewer results are known about the adversarial setting. The adversarial classification risk incurs a penalty when a point can be perturbed into the opposite class. Similarly, adversarial surrogate risks involve computing the worst-case value (i.e. supremum) of a loss function over an $\epsilon$ -ball. Frank & Niles-Weed (2024a) characterized which risks are consistent for all data distributions, and the corresponding losses are referred to as adversarially consistent. Unfortunately, no convex loss function can be adversarially consistent for all data distributions (Meunier et al., 2022). On the other hand, Frank (2025) showed that such situations are rather atypical— when the data distribution is absolutely continuous, a surrogate risk is adversarially consistent so long as the adversarial Bayes classifier satisfies a certain notion of uniqueness called uniqueness up to degeneracy. While these results characterize consistency, none describe convergence rates.

Our Contributions:

•

We prove a linear surrogate risk bound for adversarially consistent losses (Theorem 11).
•

If the ‘distribution of optimal attacks’ satisfies a bounded noise condition, we prove a linear surrogate risk bound, under mild conditions on the loss function (Theorems 11 and 12).
•

We prove a distribution dependent surrogate risk bound that applies whenever a loss is adversarially consistent for a data distribution (Theorem 13).

Notably, this last bullet applies to convex loss functions. Due to the consistency results in prior work (Frank, 2025; Frank & Niles-Weed, 2024a; Meunier et al., 2022), one cannot hope for distribution independent surrogate bounds for non-adversarially consistent losses. To the best of the authors’ knowledge this paper is the first to prove surrogate risk bounds for the risks most commonly used in adversarial training, see Section 6 for a comparison with prior work. Understanding the optimality of the bounds presented in this paper remains an open problem.

2 Background and Preliminaries

2.1 Surrogate Risks

This paper studies binary classification on $\mathbb{R}^{d}$ with labels $-1$ and $+1$ . The measures ${\mathbb{P}}_{0}$ , ${\mathbb{P}}_{1}$ describe the probabilities of finding data with labels $-1$ , $+1$ , respectively, in subset of $\mathbb{R}^{d}$ . The classification risk of a set $A$ is the misclassification rate when points in $A$ are classified as $+1$ and points in $A^{C}$ are classified as $-1$ :

R(A)=\int{\mathbf{1}}_{A^{C}}d{\mathbb{P}}_{1}+\int{\mathbf{1}}_{A}d{\mathbb{P}}_{0}

The minimal classification risk over all Borel sets is denoted $R_{*}$ . As the derivative of an indicator function is zero wherever it is defined, the empirical version of this risk cannot be optimized with first order descent methods. Consequently, common machine learning algorithms minimize a different quantity called a surrogate risk. The surrogate risk of a function $f$ is defined as

R_{\phi}(f)=\int\phi(f)d{\mathbb{P}}_{1}+\int\phi(-f)d{\mathbb{P}}_{0}.

In practice, the loss function $\phi$ selected so that it has well-behaved derivative. In this paper, We assume:

Assumption 1.

The loss $\phi$ is continuous, non-increasing, and $\lim_{\alpha\to\infty}\phi(\alpha)=0$ .

The minimal surrogate risk over all Borel measurable functions is denoted $R_{\phi,*}$ . After optimizing the surrogate risk, a classifier is obtained by threshholding the resulting $f$ at zero. Consequently, we define the classification error of a function by $R(f)=R(\{f>0\})$ or equivalently,

R(f)=\int{\mathbf{1}}_{f\leq 0}d{\mathbb{P}}_{1}+\int{\mathbf{1}}_{f>0}d{\mathbb{P}}_{0}.

It remains to verify that minimizing the surrogate risk $R_{\phi}$ will also minimize the classification risk $R$ .

Definition 1.

The loss function $\phi$ is consistent for the distribution ${\mathbb{P}}_{0}$ , ${\mathbb{P}}_{1}$ if every minimizing sequence of $R_{\phi}$ is also a minimizing sequence of $R$ when the data is distributed according to ${\mathbb{P}}_{0}$ and ${\mathbb{P}}_{1}$ . The loss function $\phi$ is consistent if it is consistent for all data distributions.

Prior work establishes conditions under which many common loss functions are consistent.

Theorem 1.

A convex loss $\phi$ is consistent iff it is differentiable at zero and $\phi^{\prime}(0)<0$ .

See (Bartlett et al., 2006, Theorem 2). Furthermore, (Frank & Niles-Weed, 2024a, Proposition 3) establishes a condition that applies to non-convex losses:

Theorem 2.

If $\inf_{\alpha}1/2(\phi(\alpha)+\phi(-\alpha))<\phi(0)$ , then the loss $\phi$ is consistent.

The $\rho$ -margin loss $\phi_{\rho}(\alpha)=\min(1,\max(1-\alpha/\rho,0))$ and the shifted sigmoid loss $\phi_{\tau}(\alpha)=1/(1+\exp(\alpha-\tau))$ , $\tau>0$ , both satisfy this criterion. However, a convex loss $\phi$ cannot satisfy this condition:

\frac{1}{2}\left(\phi(\alpha)+\phi(-\alpha)\right)\geq\phi\big{(}\frac{1}{2}\alpha+\frac{1}{2}\cdot-\alpha\big{)}=\phi(0).

(1)

In addition to consistency, understanding convergence rates is a key concern. Specifically, prior work (Bartlett et al., 2006; Zhang, 2004) establishes surrogate risk bounds of the form $\Psi(R(f)-R_{*})\leq R_{\phi}(f)-R_{\phi,*}$ for some function $\Psi$ . This inequality bounds the convergence rate of $R(f)-R_{*}$ in terms of the convergence of $R_{\phi}(f)-R_{\phi,*}$ .

The values $R_{*}$ , $R_{\phi,*}$ can be expressed in terms of the data distribution by re-writing these quantities in terms of the total probability measure ${\mathbb{P}}={\mathbb{P}}_{0}+{\mathbb{P}}_{1}$ and the conditional probability of the label +1, given by $\eta(x)=d{\mathbb{P}}_{1}/d{\mathbb{P}}$ . An equivalent formulation of the classification risk is

R(f)=\int C(\eta({\mathbf{x}}),f({\mathbf{x}}))d{\mathbb{P}}({\mathbf{x}})

(2)

with

C(\eta,\alpha)=\eta{\mathbf{1}}_{\alpha\leq 0}+(1-\eta){\mathbf{1}}_{\alpha>0},

(3)

and the minimal classification risk is found by minimizing the integrand of Equation 2 at each ${\mathbf{x}}$ . Define

C^{*}(\eta)=\inf_{\alpha}C(\eta,\alpha)=\min(\eta,1-\eta),

(4)

then the minimal classification risk is

R_{*}=\int C^{*}(\eta({\mathbf{x}}))d{\mathbb{P}}({\mathbf{x}}).

Analogously, the surrogate risk in terms of $\eta$ and ${\mathbb{P}}$ is

R_{\phi}(f)=\int C_{\phi}(\eta({\mathbf{x}}),f({\mathbf{x}}))d{\mathbb{P}}

(5)

and the minimal surrogate risk is

R_{\phi,*}=\int C_{\phi}^{*}(\eta({\mathbf{x}}))d{\mathbb{P}}({\mathbf{x}})

with the conditional risk $C_{\phi}(\eta,\alpha)$ and minimal conditional risk $C_{\phi}^{*}(\eta)$ defined by

C_{\phi}(\eta,\alpha)=\eta\phi(\alpha)+(1-\eta)\phi(-\alpha),\quad C_{\phi}^{*}(\eta)=\inf_{\alpha}C_{\phi}(\eta,\alpha).

(6)

Notice that minimizers to $R_{\phi}$ may need to be $\overline{\mathbb{R}}$ -valued— consider the exponential loss $\phi(\alpha)=e^{-\alpha}$ and a distribution with $\eta({\mathbf{x}})\equiv 1$ . Then the only minimizer to $R_{\phi}$ would be $+\infty$ .

The consistency of $\phi$ can be fully characterized by the properties of the function $C_{\phi}^{*}(\eta)$ .

Theorem 3.

A loss $\phi$ is consistent iff $C_{\phi}^{*}(\eta)<\phi(0)$ for all $\eta\neq 1/2$ .

Surprisingly, this criterion has not appeared in prior work. See Appendix A for a proof.

In terms of the function $C_{\phi}^{*}$ , Theorem 2 states that any loss $\phi$ with $C_{\phi}^{*}(1/2)<\phi(0)$ is consistent.

The function $C_{\phi}^{*}$ is a key component of surrogate risk bounds from prior work. Specifically, Bartlett et al. (2006) show:

Theorem 4.

Let $\phi$ be any loss satisfying Assumption 1 with $C_{\phi}^{*}(1/2)=\phi(0)$ and define

\Psi(\theta)=\phi(0)-C_{\phi}^{*}\left(\frac{1+\theta}{2}\right).

Then

\Psi(C(\eta,f)-C^{*}(\eta))\leq C_{\phi}(\eta,f)-C_{\phi}^{*}(\eta)

(7)

and consequently

\Psi(R(f)-R_{*})\leq R_{\phi}(f)-R_{\phi}^{*}.

(8)

Equation 8 is a consequence of Equation 7 and Jensen’s inequality. Furthermore, a result of (Bartlett et al., 2006) implies a linear bound when $C_{\phi}^{*}(1/2)<\phi(0)$ :

R(f)-R_{*}\leq\frac{1}{\phi(0)-C_{\phi}^{*}(1/2)}(R_{\phi}(f)-R_{\phi,*})

(9)

Furthermore, a distribution with zero classification error $R_{*}$ has the surrogate risk bound

R(f)-R_{*}\leq\frac{1}{\phi(0)}(R_{\phi}(f)-R_{\phi,*})

(10)

so long as $\phi(0)>0$ . Such distributions are referred to as realizable. A proof of this result that transfers directly to the adversarial scenario is provided in Section B.1.

A distribution is said to satisfy Massart’s noise condition (Massart & Nédélec, 2006) if there is an $\alpha\in(0,1/2]$ such that $|\eta-1/2|\geq\alpha$ holds ${\mathbb{P}}$ -a.e. Under this condition, Massart & Nédélec (2006) establish improved sample complexity guarantees. Furthermore, such distributions exhibit a linear surrogate loss bound as well. These linear bounds, the realizable bounds from Equation 10, and the linear bounds from Equation 9 are summarized in a single statement below.

Proposition 1.

Let $\eta$ , ${\mathbb{P}}$ be a distribution that satisfies $|\eta-1/2|\geq\alpha$ ${\mathbb{P}}$ -a.e. with a constant $\alpha\in[0,1/2]$ , and let $\phi$ be a loss with $\phi(0)>C_{\phi}^{*}(1/2-\alpha)$ . Then for all $|\eta-1/2|\geq\alpha$ ,

C(\eta,f)-C^{*}(\eta)\leq\frac{1}{\phi(0)-C_{\phi}^{*}(\frac{1}{2}-\alpha)}(C_{\phi}(\eta,f)-C_{\phi}^{*}(\eta))

(11)

and consequently

R(f)-R_{*}\leq\frac{1}{\phi(0)-C_{\phi}^{*}(\frac{1}{2}-\alpha)}(R_{\phi}(f)-R_{\phi,*})

(12)

When $\alpha\neq 0$ , this surrogate risk bound proves a linear convergence rate under Massart’s noise condition. If $\alpha=0$ and $C_{\phi}^{*}(1/2)<\phi(0)$ , then the bound in Equation 12 reduces to Equation 9 while if $\alpha=1/2$ then this bound reduces to Equation 10. See Section B.2 for a proof of this result. One of the main results of this paper is that Equation 12 generalizes to adversarial risks.

Note that the surrogate risk bound of Theorem 4 can be linear even for convex loss functions. For the hinge loss $\phi(\alpha)=\max(1-\alpha,0)$ , the function $\phi$ computes to $\phi(\theta)=|\theta|$ . Prior work (Frongillo & Waggoner, 2021, Theorem 1) observed a linear surrogate bound for piecewise linear losses: if $\phi$ is piecewise linear, then $C_{\phi}^{*}(\eta)$ is piecewise linear and Jensen’s inequality implies a linear surrogate bound so long as $\phi$ is consistent (due to Theorem 3). On the other hand, (Frongillo & Waggoner, 2021, Theorem 2) show that convex losses which are locally strictly convex and Lipschitz achieve at best a square root surrogate risk rate.

Mahdavi et al. (2014) emphasize the importance of a linear convergence rate in a surrogate risk bound. Their paper studies the sample complexity of estimating a classifier with a surrogate risk. They note that typically convex surrogate losses exhibiting favorable sample complexity do not satisfy favorable surrogate risk bounds, due to the results of (Frongillo & Waggoner, 2021). Consequently, Proposition 1 hints that proving favorable sample complexity guarantees for learning with convex surrogate risks could require distributional assumptions, such as Massart’s noise condition.

2.2 Adversarial Risks

This paper extends surrogate risk bounds of Equations 8, 10 and 12 to adversarial risks. The adversarial classification risk incurs a penalty of 1 whenever a point ${\mathbf{x}}$ can be perturbed into the opposite class. This penalty can be expressed in terms of supremums of indicator functions— the adversarial classification risk incurs a penalty of 1 whenever $\sup_{\|{\mathbf{x}}^{\prime}-{\mathbf{x}}\|\leq\epsilon}{\mathbf{1}}_{A}({\mathbf{x}}^{\prime})=1$ or $\sup_{\|{\mathbf{x}}^{\prime}-{\mathbf{x}}\|\leq\epsilon}{\mathbf{1}}_{A^{C}}({\mathbf{x}}^{\prime})=1$ . Define

S_{\epsilon}(g)({\mathbf{x}})=\sup_{\|{\mathbf{x}}-{\mathbf{x}}^{\prime}\|\leq\epsilon}g({\mathbf{x}}^{\prime}).

The adversarial classification risk is then

R^{\epsilon}(A)=\int S_{\epsilon}({\mathbf{1}}_{A^{C}})d{\mathbb{P}}_{1}+\int S_{\epsilon}({\mathbf{1}}_{A})d{\mathbb{P}}_{0}

and the adversarial surrogate risk is¹¹1In order to define the risks $R_{\phi}^{\epsilon}$ and $R^{\epsilon}$ , one must argue that $S_{\epsilon}(g)$ is measurable. Theorem 1 of (Frank & Niles-Weed, 2024b) proves that whenever $g$ is Borel, $S_{\epsilon}(g)$ is always measurable with respect to the completion of any Borel measure.

R_{\phi}^{\epsilon}(f)=\int S_{\epsilon}(\phi(f))d{\mathbb{P}}_{1}+\int S_{\epsilon}(\phi(-f))d{\mathbb{P}}_{0}.

A minimizer of the adversarial classification risk is called an adversarial Bayes classifier. After optimizing the surrogate risk, a classifier is obtained by threshholding the resulting function $f$ at zero. Consequently, the adversarial classification error of a function $f$ is defined as $R^{\epsilon}(f)=R^{\epsilon}(\{f>0\})$ or equivalently,

R^{\epsilon}(f)=\int S_{\epsilon}({\mathbf{1}}_{f\leq 0})d{\mathbb{P}}_{1}+\int S_{\epsilon}({\mathbf{1}}_{f>0})d{\mathbb{P}}_{0}.

(13)

Just as in the standard case, one would hope that minimizing the adversarial surrogate risk would minimize the adversarial classification risk.

Definition 2.

We say a loss $\phi$ is adversarially consistent for the distribution ${\mathbb{P}}_{0}$ , ${\mathbb{P}}_{1}$ if any minimizing sequence of $R_{\phi}^{\epsilon}$ is also a minimizing sequence of $R^{\epsilon}$ . We say that $\phi$ is adversarially consistent if it is adversarially consistent for every possible ${\mathbb{P}}_{0}$ , ${\mathbb{P}}_{1}$ .

Theorem 2 of (Frank & Niles-Weed, 2024a) characterizes the adversarially consistent losses:

Theorem 5.

The loss $\phi$ is adversarially consistent iff $C_{\phi}^{*}(1/2)<\phi(0)$ .

Theorem 2 implies that every adversarially consistent loss is also consistent. Unfortunately, Equation 1 shows that no convex loss is adversarially consistent. However, the data distribution for which adversarial consistency fails presented in (Meunier et al., 2022) is fairly atypical: Let ${\mathbb{P}}_{0}$ , ${\mathbb{P}}_{1}$ be the uniform distributions on $\overline{B_{\epsilon}({\mathbf{0}})}$ . Then one can show that the function sequence

f_{n}=\begin{cases}\frac{1}{n}&{\mathbf{x}}\neq 0\\ -\frac{1}{n}&{\mathbf{x}}=0\end{cases}

(14)

minimizes $R_{\phi}^{\epsilon}$ but not $R^{\epsilon}$ whenever $C_{\phi}^{*}(1/2)=\phi(0)$ (See Proposition 2 of (Frank & Niles-Weed, 2024a)). A more refined analysis relates adversarial consistency for losses with $C_{\phi}^{*}(1/2)=\phi(0)$ to a notion of uniqueness of the adversarial Bayes classifier for losses satisfying $C_{\phi}^{*}(1/2)=\phi(0)$ .

Definition 3.

Two adversarial Bayes classifiers $A_{1}$ , $A_{2}$ are equivalent up to degeneracy if any set $A$ with $A_{1}\cap A_{2}\subset A\subset A_{1}\cup A_{2}$ is also an adversarial Bayes classifier. The adversarial Bayes classifier is unique up to degeneracy if any two adversarial Bayes classifiers are equivalent up to degeneracy.

Theorem 3.3 of (Frank, 2024) proves that whenever ${\mathbb{P}}$ is absolutely continuous with respect to Lebesgue measure, then equivalence up to degeneracy is in fact an equivalence relation. Next, Theorem 4 of (Frank, 2025) relates this condition to the consistency of $\phi$ .

Theorem 6.

Let $\phi$ be a loss with $C_{\phi}^{*}(1/2)=\phi(0)$ and assume that ${\mathbb{P}}$ is absolutely continuous with respect to Lebesgue measure. Then $\phi$ is adversarially consistent for the data distribution given by ${\mathbb{P}}_{0}$ , ${\mathbb{P}}_{1}$ iff the adversarial Bayes classifier is unique up to degeneracy.

(a)

(b)

(c)

(d)

Figure 1: Adversarial Bayes classifiers for the example considered in Equation 14. The adversarial Bayes classifiers in (a) and (b) are equivalent up to degeneracy and the the adversarial Bayes classifiers in (c) and (d) are equivalent up to degeneracy, but the adversarial Bayes classifiers in (a) and (c) are not equivalent up to degeneracy.

The extension of Theorem 4 to the adversarial setting must reflect the consistency results of Theorems 5 and 6.

2.3 Minimax Theorems

A central tool in analyzing the adversarial consistency of surrogate risks is minimax theorems. These results allow one to express adversarial risks in a ‘pointwise’ manner analogous to Equation 5. We will then combine this ‘pointwise’ expression together with the proof of Theorem 4 to produce surrogate bounds for adversarial risks.

These minimax theorems utilize the $\infty$ -Wasserstein ( $W_{\infty}$ ) metric from optimal transport. Let ${\mathbb{Q}}$ and ${\mathbb{Q}}^{\prime}$ be two finite positive measures with the same total mass. Informally, the measure ${\mathbb{Q}}^{\prime}$ is within $\epsilon$ of ${\mathbb{Q}}$ in the $W_{\infty}$ metric if one can achieve the measure ${\mathbb{Q}}^{\prime}$ by moving points of ${\mathbb{Q}}$ by at most $\epsilon$ .

The $W_{\infty}$ metric is formally defined in terms of the set of couplings between ${\mathbb{Q}}$ and ${\mathbb{Q}}^{\prime}$ . A Borel measure $\gamma$ on $\mathbb{R}^{d}\times\mathbb{R}^{d}$ is a coupling between ${\mathbb{Q}}$ and ${\mathbb{Q}}^{\prime}$ if its first marginal is ${\mathbb{Q}}$ and its second marginal is ${\mathbb{Q}}^{\prime}$ , or in other words, $\gamma(A\times\mathbb{R}^{d})={\mathbb{Q}}(A)$ and $\gamma(\mathbb{R}^{d}\times A)={\mathbb{Q}}^{\prime}(A)$ for all Borel sets $A$ . Let $\Pi({\mathbb{Q}},{\mathbb{Q}}^{\prime})$ be the set of all couplings between the measures ${\mathbb{Q}}$ and ${\mathbb{Q}}^{\prime}$ . Then the $W_{\infty}$ between ${\mathbb{Q}}$ and ${\mathbb{Q}}^{\prime}$ is then

W_{\infty}({\mathbb{Q}},{\mathbb{Q}}^{\prime})=\inf_{\gamma\in\Pi({\mathbb{Q}},{\mathbb{Q}}^{\prime})}\operatorname*{ess\,sup}_{({\mathbf{x}},{\mathbf{y}})\sim\gamma}\|{\mathbf{x}}-{\mathbf{y}}\|.

(15)

Theorem 2.6 of (Jylhä, 2014) proves that the infimum in Equation 15 is always attained. The $\epsilon$ -ball around ${\mathbb{Q}}$ in the $W_{\infty}$ metric is

{{\mathcal{B}}^{\infty}_{\epsilon}}({\mathbb{Q}})=\{{\mathbb{Q}}^{\prime}:W_{\infty}({\mathbb{Q}}^{\prime},{\mathbb{Q}})\leq\epsilon\}.

The minimax theorem below will relate the adversarial risks $R_{\phi}^{\epsilon}$ , $R^{\epsilon}$ to dual problems in which an adversary seeks to maximize some dual quantity over Wasserstein- $\infty$ balls. Specifically, one can show:

Lemma 1.

Let $g$ be a Borel function. Let $\gamma$ be a coupling between the measures ${\mathbb{Q}}$ and ${\mathbb{Q}}^{\prime}$ supported on $\Delta_{\epsilon}=\{({\mathbf{x}},{\mathbf{x}}^{\prime}):\|{\mathbf{x}}-{\mathbf{x}}^{\prime}\|\leq\epsilon\}$ . Then $S_{\epsilon}(g)({\mathbf{x}})\geq g({\mathbf{x}}^{\prime})$ $\gamma$ -a.e. and consequently

\int S_{\epsilon}(g)d{\mathbb{Q}}\geq\sup_{{\mathbb{Q}}^{\prime}\in{{\mathcal{B}}^{\infty}_{\epsilon}}({\mathbb{Q}})}\int gd{\mathbb{Q}}^{\prime}.

See Appendix C for a proof. Thus, applying Lemma 1, the quantity $\inf_{A}R^{\epsilon}(A)$ can be lower bounded by an infimum followed by a supremum. Is it possible to swap this infimum and supremum? (Pydi & Jog, 2021) answers this question in the affirmative. Let $C^{*}(\eta)$ be as defined in Equation 4 and let

\bar{R}({\mathbb{P}}_{0}^{\prime},{\mathbb{P}}_{1}^{\prime})=\inf_{A\text{ Borel}}\int{\mathbf{1}}_{A^{C}}d{\mathbb{P}}_{1}^{\prime}+\int{\mathbf{1}}_{A}d{\mathbb{P}}_{0}^{\prime}=\int C^{*}\left(\frac{d{\mathbb{P}}_{1}^{\prime}}{d\left({\mathbb{P}}_{1}^{\prime}+{\mathbb{P}}_{0}^{\prime}\right)}\right)d\left({\mathbb{P}}_{0}^{\prime}+{\mathbb{P}}_{1}^{\prime}\right).

(16)

Theorem 7.

Let $\bar{R}$ be as defined in Equation 16. Then

\inf_{A\text{ Borel}}R^{\epsilon}(A)=\sup_{\begin{subarray}{c}{\mathbb{P}}_{1}^{\prime}\in{{\mathcal{B}}^{\infty}_{\epsilon}}({\mathbb{P}}_{1})\\ {\mathbb{P}}_{0}^{\prime}\in{{\mathcal{B}}^{\infty}_{\epsilon}}({\mathbb{P}}_{0})\end{subarray}}\bar{R}({\mathbb{P}}_{0}^{\prime},{\mathbb{P}}_{1}^{\prime}).

Furthermore, equality is attained at some Borel measurable $A$ , ${\mathbb{P}}_{0}^{*}$ , and ${\mathbb{P}}_{1}^{*}$ with ${\mathbb{P}}_{0}^{*}\in{{\mathcal{B}}^{\infty}_{\epsilon}}({\mathbb{P}}_{0})$ and ${\mathbb{P}}_{1}^{*}\in{{\mathcal{B}}^{\infty}_{\epsilon}}({\mathbb{P}}_{1})$ .

See (Frank & Niles-Weed, 2024a, Theorem 1) for a proof of this statement. The maximizers ${\mathbb{P}}_{0}^{*}$ , ${\mathbb{P}}_{1}^{*}$ can be interpreted as optimal adversarial attacks (see discussion following (Frank & Niles-Weed, 2024b, Theorem 7)). Frank (2024, Theorem 3.4) provide a criterion for uniqueness up to degeneracy in terms of dual maximizers.

Theorem 8.

The following are equivalent:

A)

The adversarial Bayes classifier is unique up to degeneracy
B)

There are maximizers ${\mathbb{P}}_{0}^{*}$ , ${\mathbb{P}}_{1}^{*}$ of $\bar{R}$ for which ${\mathbb{P}}^{*}(\eta^{*}=1/2)=0$ , where ${\mathbb{P}}^{*}={\mathbb{P}}_{0}^{*}+{\mathbb{P}}_{1}^{*}$ and $\eta^{*}=d{\mathbb{P}}_{1}^{*}/d{\mathbb{P}}^{*}$

In other words, the adversarial Bayes classifier is unique up to degeneracy iff the region where both classes are equally probable has measure zero under some optimal adversarial attack. Theorems 6 and 8 relate adversarial consistency and the dual problem, suggesting that these optimal adversarial attacks ${\mathbb{P}}_{0}^{*}$ , ${\mathbb{P}}_{1}^{*}$ may appear in adversarial surrogate bounds.

Frank & Niles-Weed (2024b) prove an minimax principle analogous to Theorem 7 for the adversarial surrogate risk. Let $C_{\phi}^{*}(\eta)$ be as defined in Equation 6 and let

\bar{R}_{\phi}({\mathbb{P}}_{0}^{\prime},{\mathbb{P}}_{1}^{\prime})=\inf_{f\text{ Borel}}\int\phi(f)d{\mathbb{P}}_{1}^{\prime}+\int\phi(-f)d{\mathbb{P}}_{0}^{\prime}=\int C_{\phi}^{*}\left(\frac{d{\mathbb{P}}_{1}^{\prime}}{d\left({\mathbb{P}}_{1}^{\prime}+{\mathbb{P}}_{0}^{\prime}\right)}\right)d\left({\mathbb{P}}_{0}^{\prime}+{\mathbb{P}}_{1}^{\prime}\right)

(17)

Theorem 9.

Let $\bar{R}_{\phi}$ be defined as in Equation 17. Then

\inf_{\begin{subarray}{c}f\text{ Borel,}\\ \overline{\mathbb{R}}\text{-valued}\end{subarray}}R_{\phi}^{\epsilon}(f)=\sup_{\begin{subarray}{c}{\mathbb{P}}_{1}^{\prime}\in{{\mathcal{B}}^{\infty}_{\epsilon}}({\mathbb{P}}_{1})\\ {\mathbb{P}}_{0}^{\prime}\in{{\mathcal{B}}^{\infty}_{\epsilon}}({\mathbb{P}}_{0})\end{subarray}}\bar{R}_{\phi}({\mathbb{P}}_{0}^{\prime},{\mathbb{P}}_{1}^{\prime}).

Furthermore, equality is attained at some Borel measurable $f^{*}$ , ${\mathbb{P}}_{0}^{*}$ , and ${\mathbb{P}}_{1}^{*}$ with ${\mathbb{P}}_{0}^{*}\in{{\mathcal{B}}^{\infty}_{\epsilon}}({\mathbb{P}}_{0})$ and ${\mathbb{P}}_{1}^{*}\in{{\mathcal{B}}^{\infty}_{\epsilon}}({\mathbb{P}}_{1})$ .

Just as in the non-adversarial scenario, $R_{\phi}^{\epsilon}$ may not assume its infimum at an $\mathbb{R}$ -valued function. However, (Frank & Niles-Weed, 2024a, Lemma 8) show that

\inf_{f\text{ $\overline{\mathbb{R}}$-valued}}R_{\phi}^{\epsilon}(f)=\inf_{f\text{ $\mathbb{R}$-valued }}R_{\phi}^{\epsilon}(f).

Lastly, one can show that maximizers of $\bar{R}_{\phi}$ are always maximizers of $\bar{R}$ as well. In other words— optimal attacks on minimizers of the adversarial surrogate $R_{\phi}^{\epsilon}$ are always optimal attacks on minimizers of the adversarial classification risk $R^{\epsilon}$ as well.

Theorem 10.

Consider maximizing the dual objectives $\bar{R}_{\phi}$ and $\bar{R}$ over ${{\mathcal{B}}^{\infty}_{\epsilon}}({\mathbb{P}}_{0})\times{{\mathcal{B}}^{\infty}_{\epsilon}}({\mathbb{P}}_{1})$ .

1)

Any maximizer $({\mathbb{P}}_{0}^{*},{\mathbb{P}}_{1}^{*})$ of $\bar{R}_{\phi}$ over ${{\mathcal{B}}^{\infty}_{\epsilon}}({\mathbb{P}}_{0})\times{{\mathcal{B}}^{\infty}_{\epsilon}}({\mathbb{P}}_{1})$ must maximize $\bar{R}$ as well.
2)

If the adversarial Bayes classifier is unique up to degeneracy, then there are maximizers $({\mathbb{P}}_{0}^{*},{\mathbb{P}}_{1}^{*})$ of $\bar{R}_{\phi}$ where ${\mathbb{P}}^{*}(\eta^{*}=1/2)=0$ , with ${\mathbb{P}}^{*}={\mathbb{P}}_{0}^{*}+{\mathbb{P}}_{1}^{*}$ and $\eta^{*}=d{\mathbb{P}}_{1}^{*}/d{\mathbb{P}}^{*}$ .

See Appendix D for a proof of Item 1), Item 2) is shown in Theorems 5 and 7 of (Frank, 2025).

3 Main Results

Prior work has characterized when a loss $\phi$ is adversarially consistent with respect to a distribution ${\mathbb{P}}_{0}$ , ${\mathbb{P}}_{1}$ : Theorem 5 proves that a distribution independent surrogate risk bound is only possible when $C_{\phi}^{*}(1/2)<\phi(0)$ while Theorem 6 suggests that a surrogate bound must depend on the marginal distribution of $\eta^{*}$ under ${\mathbb{P}}^{*}$ , and furthermore, such a bound is only possible when ${\mathbb{P}}^{*}(\eta^{*}=1/2)=0$ .

Compare these statements to Proposition 1: Theorems 5 and 6 imply that $\phi$ is adversarially consistent for ${\mathbb{P}}_{0}$ , ${\mathbb{P}}_{1}$ if $C_{\phi}^{*}(1/2)<\phi(0)$ or if there exist some maximizers of $\bar{R}$ that satisfy Massart’s noise condition. Alternatively, due to Theorem 10, one can equivalently assume that there are maximizers of $\bar{R}_{\phi}$ satisfying Massart’s noise condition. Our first bound extends Proposition 1 to the adversarial scenario, with the data distribution ${\mathbb{P}}_{0}$ , ${\mathbb{P}}_{1}$ replaced with the distribution of optimal adversarial attacks.

Theorem 11.

Let ${\mathbb{P}}_{0}$ , ${\mathbb{P}}_{1}$ be a distribution for which there are maximizers ${\mathbb{P}}_{0}^{*}$ , ${\mathbb{P}}_{1}^{*}$ of the dual problem $\bar{R}_{\phi}$ that satisfy $|\eta^{*}-1/2|\geq\alpha$ ${\mathbb{P}}^{*}$ -a.e. for some constant $\alpha\in[0,1/2]$ with $C_{\phi}^{*}(1/2-\alpha)<\phi(0)$ , where ${\mathbb{P}}^{*}={\mathbb{P}}_{0}^{*}+{\mathbb{P}}_{1}^{*}$ , $\eta^{*}=d{\mathbb{P}}_{1}^{*}/d{\mathbb{P}}^{*}$ . Then

R^{\epsilon}(f)-R_{*}^{\epsilon}\leq\frac{3+\sqrt{5}}{2}\frac{1}{\phi(0)-C_{\phi}^{*}(1/2-\alpha)}(R_{\phi}^{\epsilon}(f)-R_{\phi,*}^{\epsilon}).

When $C_{\phi}^{*}(1/2)<\phi(0)$ , one can select $\alpha=0$ in Theorem 11 to produce a distribution-independent bound. The constant $(3+\sqrt{5})/2$ may be sub-optimal; in fact Theorem 4 of Frank (2025) proves that $R^{\epsilon}(f)-R^{\epsilon}_{*}\leq 1/(2(\phi(0)-C_{\phi_{\rho}}^{*}(1/2)))(R_{\phi_{\rho}}^{\epsilon}(f)-R^{\epsilon}_{\phi_{\rho},*})$ where $\phi_{\rho}(\alpha)=\min(1,\max(0,1-\alpha/\rho))$ is the $\rho$ -margin loss. Furthermore, the bound in Equation 10 extends directly to the adversarial setting.

Theorem 12.

Let $\phi$ be any loss with $\phi(0)>0$ satisfying Assumption 1. Then if $R_{*}^{\epsilon}=0$ ,

R^{\epsilon}(f)-R^{\epsilon}_{*}\leq\frac{1}{\phi(0)}\left(R_{\phi}^{\epsilon}(f)-R_{\phi,*}^{\epsilon}\right)

A distribution will have zero adversarial risk whenever the supports of ${\mathbb{P}}_{0}$ and ${\mathbb{P}}_{1}$ are separated by at least $2\epsilon$ , see Examples 1 and 2(a) for and example. Zero adversarial classification risk corresponds to $\alpha=1/2$ in Massart’s noise condition.

In contrast, Theorem 11 states that if some distribution of optimal adversarial attacks satisfies Massart’s noise condition, then the excess adversarial surrogate risk is at worst a linear upper bound on the excess adversarial classification risk. However, if $C_{\phi}^{*}(1/2)=\phi(0)$ , this constant approaches infinity as $\alpha\to 0$ , reflecting the fact that adversarial consistency fails when the adversarial Bayes classifier is not unique up to degeneracy. When $\alpha\neq 1/2$ , understanding what assumptions on ${\mathbb{P}}_{0}$ , ${\mathbb{P}}_{1}$ guarantee Massart’s noise condition for ${\mathbb{P}}_{0}^{*}$ , ${\mathbb{P}}_{1}^{*}$ is an open question. Example 4.6 of (Frank, 2024) demonstrates a distribution that satisfies Massart’s noise condition and yet the adversarial Bayes classifier is not unique up to degeneracy. Thus Massart’s noise condition for ${\mathbb{P}}_{0},{\mathbb{P}}_{1}$ does not guarantee Massart’s noise condition for ${\mathbb{P}}_{0}^{*}$ , ${\mathbb{P}}_{1}^{*}$ . See Examples 2 and 2(b) for an example where Theorem 11 applies with $\alpha>0$ .

Finally, by averaging bounds of the form Theorem 11 over all values of $\eta^{*}$ produces a distribution-dependent surrogate bound, valid whenever the adversarial Bayes classifier is unique up to degeneracy. For a given function $f$ , let the concave envelope of $f$ be the smallest concave function larger than $f$ :

\operatorname{conc}(f)=\inf\{g:\geq f\text{ on }\operatorname{dom}(h),g\text{ concave and upper semi-continuous}\}

(18)

Theorem 13.

Assume that $C_{\phi}^{*}(1/2)=\phi(0)$ and that the adversarial Bayes classifier is unique up to degeneracy. Let ${\mathbb{P}}_{0}^{*}$ , ${\mathbb{P}}_{1}^{*}$ be maximizers of $\bar{R}_{\phi}$ for which ${\mathbb{P}}^{*}(\eta^{*}=1/2)=0$ , with ${\mathbb{P}}^{*}={\mathbb{P}}_{0}^{*}+{\mathbb{P}}_{1}^{*}$ and $\eta^{*}=d{\mathbb{P}}_{1}^{*}/d{\mathbb{P}}^{*}$ . Let $H(z)=\operatorname{conc}({\mathbb{P}}^{*}(|\eta^{*}-1/2|\leq z))$ . Let $\Psi$ be the function defined by Theorem 4 and let $\tilde{\Lambda}(z)=\Psi^{-1}(\min(\frac{z}{6},\phi(0)))$ . Then

R^{\epsilon}(f)-R^{\epsilon}_{*}\leq\tilde{\Phi}(R_{\phi}^{\epsilon}(f)-R_{\phi,*}^{\epsilon})

with

\tilde{\Phi}(z)=6\left(\operatorname{id}+\min(1,\sqrt{-2{e}H\ln 2H})\right)\circ\tilde{\Lambda}

See Examples 3 and 2(c) for an example of calculating a distribution-dependent surrogate risk bound.

One can prove that the function $H$ is always continuous and satisfies $H(0)=0$ , proving that this bound is non-vacuous (see Lemma 2 below). Further notice that $H\ln H$ approaches zero as $H\to 0$ .

The map $\tilde{\Phi}$ combines two components: $\tilde{\Lambda}$ , a modified version of $\Psi^{-1}$ , and $H$ , a modification of the cdf of $|\eta^{*}-1/2|$ . The function $\tilde{\Lambda}$ is a scaled version of $\Psi^{-1}$ , where $\Psi$ is the surrogate risk bound in the non-adversarial case of Theorem 4. The domain of $\Psi^{-1}$ is $[0,\phi(0)]$ , and thus the role of the $\min$ in the definition of $\tilde{\Lambda}$ is to truncate the argument so that it fits into this domain. The factor of $1/6$ in this function appears to be an artifact of our proof, see Section 5 for further discussion. In contrast, the map $H$ translates the distribution of $\eta^{*}$ into a surrogate risk transformation. Compare with Theorem 6, which states that consistency fails if ${\mathbb{P}}^{*}(\eta^{*}=1/2)>0$ ; accordingly, the function $H$ becomes a poorer bound when more mass of $\eta^{*}$ is near $1/2$ .

Examples

Below are three examples for which each of our three main theorems apply. These examples are all one-dimensional distributions, and we denote the pdfs of ${\mathbb{P}}_{0}$ , and ${\mathbb{P}}_{1}$ by $p_{0}$ and $p_{1}$ .

To start, a distribution for which the supports of ${\mathbb{P}}_{0}$ , ${\mathbb{P}}_{1}$ are more than $2\epsilon$ apart must have zero risk. Furthermore, if ${\mathbb{P}}$ is absolutely continuous with respect to Lebesgue measure and the supports of ${\mathbb{P}}_{0}$ , ${\mathbb{P}}_{1}$ are exactly $2\epsilon$ apart, then the adversarial classification risk will be zero (see for instance (Awasthi et al., 2023a, Lemma 4) or (Pydi & Jog, 2021, Lemma 4.3)).

Example 1 (When $R_{*}^{\epsilon}=0$ ).

Let $p_{0}$ and $p_{1}$ be defined by

p_{0}(x)=\begin{cases}1&\text{if }x\in[-1-\delta,-\delta]\\ 0&\text{otherwise}\end{cases}\quad p_{1}(x)=\begin{cases}1&\text{if }x\in[\delta,1+\delta]\\ 0&\text{otherwise}\end{cases}

for some $\delta>0$ . See Figure 2(a) for a depiction of $p_{0}$ and $p_{1}$ . This distribution satisfies $R^{\epsilon}_{*}=0$ for all $\epsilon\leq\delta$ and thus the surrogate bound of Theorem 12 applies.

Examples 2 and 3 require computing maximizers to the dual $\bar{R}_{\phi}$ . See Sections J.2 and J.1 for these calculations. The following example illustrates a distribution for which Massart’s noise condition can be verified for a distribution of optimal attacks.

Example 2 (Massart’s noise condition).

Let $\delta>0$ and let $p$ be the uniform density on $[-1-\delta,-\delta]\cup[\delta,1+\delta]$ . Define $\eta$ by

\eta(x)=\begin{cases}\frac{1}{4}&\text{if }x\in[-1-\delta,-\delta]\\ \frac{3}{4}&\text{if }x\in[\delta,1+\delta]\end{cases}

(19)

see Figure 2(b) for a depiction of $p_{0}$ and $p_{1}$ . For this distribution and $\epsilon\leq\delta$ , the minimal surrogate and adversarial surrogate risks are always equal ( $R_{\phi,*}=R^{\epsilon}_{\phi,*}$ ). This fact together with Theorem 9 imply that optimal attacks on this distribution are ${\mathbb{P}}_{1}^{*}={\mathbb{P}}_{1}$ and ${\mathbb{P}}_{0}^{*}={\mathbb{P}}_{0}$ , see Section J.1 for details. Consequently: the distribution of optimal attacks ${\mathbb{P}}_{0}^{*}$ , ${\mathbb{P}}_{1}^{*}$ satisfies Massart’s noise condition with $\alpha=1/4$ and as a result the bounds of Theorem 11 apply.

Finally, the next example presents a case in which Massart’s noise condition fails for the distribution of optimal adversarial attacks, yet the adversarial Bayes classifier remains unique up to degeneracy. Theorem 13 continues to yield a valid surrogate bound.

Example 3 (Gaussian example).

Consider an equal Gaussian mixture with equal variances and differing means, with $\mu_{1}>\mu_{0}$ :

p_{0}(x)=\frac{1}{2}\cdot\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(x-\mu_{0})^{2}}{2\sigma^{2}}},\quad p_{1}(x)=\frac{1}{2}\cdot\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(x-\mu_{1})^{2}}{2\sigma^{2}}}

We further assume $\mu_{1}-\mu_{0}\leq\sqrt{2}\sigma$ ; see Figure 2(c) for a depiction. We will show that when $\mu_{1}-\mu_{0}<2\epsilon$ , the optimal attacks ${\mathbb{P}}_{0}^{*}$ , ${\mathbb{P}}_{1}^{*}$ are gaussians centered at $\mu_{0}+\epsilon$ and $\mu_{1}-\epsilon$ — explicitly the pdfs of ${\mathbb{P}}_{0}^{*}$ and ${\mathbb{P}}_{1}^{*}$ are given by

p_{0}^{*}(x)=\frac{1}{2}\cdot\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(x-(\mu_{0}+\epsilon))^{2}}{2\sigma^{2}}},\quad p_{1}^{*}(x)=\frac{1}{2}\cdot\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(x-(\mu_{1}-\epsilon))^{2}}{2\sigma^{2}}},

(20)

see Section J.2 for details. We verify that ${\mathbb{P}}_{0}^{*}$ and ${\mathbb{P}}_{1}^{*}$ are in fact optimal by finding a function $f^{*}$ for which $R_{\phi}^{\epsilon}(f)=\bar{R}_{\phi}({\mathbb{P}}_{0}^{*},{\mathbb{P}}_{1}^{*})$ , the strong duality result in Theorem 9 will then imply that ${\mathbb{P}}_{0}^{*}$ and ${\mathbb{P}}_{1}^{*}$ must maximize the dual $\bar{R}_{\phi}$ , see Section J.2 for details.

However, when $\mu_{1}-\mu_{0}\leq\sqrt{2}\sigma$ , then the function $h(z)={\mathbb{P}}^{*}(|\eta^{*}-1/2|\leq z)$ is concave in $z$ for all $\epsilon<(\mu_{1}-\mu_{0})/2$ and consequently $h=H$ , see Section J.3 for details. Unfortunately, $h$ is a rather unweildy function. By comparing to the linear approximation at zero, one can show the following upper bound on $H$ :

H(z)\leq\min\left(\frac{16\sigma^{2}}{\mu_{1}-\mu_{0}-2\epsilon}z,1\right).

(21)

Again, see Section J.3 for details.

When $\epsilon\geq(\mu_{1}-\mu_{0})/2$ , (Frank, 2024, Example 4.1) shows that the adversarial Bayes classifier is not unique up to degeneracy. Notably, the bound in preceding example deteriorates as $(\mu_{1}-\mu_{0})/2\to\epsilon$ , and then fails entirely when $\epsilon=(\mu_{1}-\mu_{0})/2$ .

4 Linear Surrogate Bounds— Proof of Theorems 12 and 11

The proof of Theorem 12 simply involves bounding the indicator functions $S_{\epsilon}({\mathbf{1}}_{f>0})$ , $S_{\epsilon}({\mathbf{1}}_{f\leq 0})$ in terms of the functions $S_{\epsilon}(\phi\circ f)$ and $S_{\epsilon}(\phi\circ-f)$ . This strategy is entirely analogous to that the argument for the (non-adversarial) surrogate bound Equation 10 in Section B.1. A similar argument is also an essential intermediate step of the proof of Theorem 11.

Proof of Theorem 12.

If $R_{*}^{\epsilon}=0$ , then the duality result Theorem 7 implies that for any measures ${\mathbb{P}}_{0}^{\prime}\in{{\mathcal{B}}^{\infty}_{\epsilon}}({\mathbb{P}}_{0})$ , ${\mathbb{P}}_{1}^{\prime}\in{{\mathcal{B}}^{\infty}_{\epsilon}}({\mathbb{P}}_{1})$ then ${\mathbb{P}}^{\prime}(\eta^{\prime}=0\text{ or }1)=1$ , where ${\mathbb{P}}^{\prime}={\mathbb{P}}_{0}^{\prime}+{\mathbb{P}}_{1}^{\prime}$ and $\eta^{\prime}=d{\mathbb{P}}_{1}^{\prime}/d{\mathbb{P}}^{\prime}$ . Consequently, $\bar{R}_{\phi}({\mathbb{P}}_{0}^{\prime},{\mathbb{P}}_{1}^{\prime})=0$ for any $({\mathbb{P}}_{0}^{\prime},{\mathbb{P}}_{1}^{\prime})\in{{\mathcal{B}}^{\infty}_{\epsilon}}({\mathbb{P}}_{0})\times{{\mathcal{B}}^{\infty}_{\epsilon}}({\mathbb{P}}_{1})$ and consequently $R_{\phi,*}^{\epsilon}=0$ . Thus it remains to show that $R^{\epsilon}(f)\leq\frac{1}{\phi(0)}R_{\phi}^{\epsilon}(f)$ for all functions $f$ . We will prove the bound

S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})\leq\frac{1}{\phi(0)}S_{\epsilon}(\phi(f))({\mathbf{x}}).

(22)

The inequality Equation 22 trivially holds when $S_{\epsilon}({\mathbf{1}}_{f\leq 0})=0$ . Alternatively, the relation $S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})=1$ implies $f({\mathbf{x}}^{\prime})\leq 0$ for some ${\mathbf{x}}^{\prime}\in\overline{B_{\epsilon}({\mathbf{x}})}$ and consequently $S_{\epsilon}(\phi(f))({\mathbf{x}})\geq\phi(0)$ . Thus whenever $S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})=1$ ,

S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})=\frac{\phi(0)}{\phi(0)}\leq\frac{1}{\phi(0)}S_{\epsilon}(\phi(f))({\mathbf{x}}).

(23)

An analogous argument implies that whenever $S_{\epsilon}({\mathbf{1}}_{f>0})({\mathbf{x}})=1$ ,

S_{\epsilon}({\mathbf{1}}_{f>0})({\mathbf{x}})=S_{\epsilon}({\mathbf{1}}_{-f<0})({\mathbf{x}})\leq\frac{1}{\phi(0)}S_{\epsilon}(\phi(-f))({\mathbf{x}}).

As a result:

	$\displaystyle R^{\epsilon}(f)$	$\displaystyle=\int S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})d{\mathbb{P}}_{1}+\int S_{\epsilon}({\mathbf{1}}_{f>0})({\mathbf{x}})d{\mathbb{P}}_{0}\leq\frac{1}{\phi(0)}\left(\int S_{\epsilon}(\phi(f))({\mathbf{x}})d{\mathbb{P}}_{1}+\int S_{\epsilon}(\phi(-f)({\mathbf{x}})d{\mathbb{P}}_{0}\right)$
		$\displaystyle=\frac{1}{\phi(0)}R_{\phi}^{\epsilon}(f)$

∎

In contrast, when the optimal surrogate risk $R^{\epsilon}_{\phi,*}$ is non-zero, the bound in Theorem 11 necessitates a more sophisticated argument. Below, we decompose both the adversarial classification risk and the adversarial surrogate risk as the sum of four terms positive terms.

Let ${\mathbb{P}}_{0}^{*},{\mathbb{P}}_{1}^{*}$ be any maximizers of $\bar{R}_{\phi}$ that also maximize $\bar{R}$ by Theorem 10. Set ${\mathbb{P}}^{*}={\mathbb{P}}_{0}^{*}+{\mathbb{P}}_{1}^{*}$ , $\eta^{*}=d{\mathbb{P}}_{1}^{*}/d{\mathbb{P}}^{*}$ . Let $\gamma_{0}^{*}$ , $\gamma_{1}^{*}$ be the couplings between ${\mathbb{P}}_{0}$ , ${\mathbb{P}}_{0}^{*}$ and ${\mathbb{P}}_{1}$ , ${\mathbb{P}}_{1}^{*}$ respectively that achieve the infimum in Equation 15. Then due to the strong duality in Theorem 7, one can decompose the excess classification risk as

\displaystyle R^{\epsilon}(f)-R_{*}^{\epsilon}=R^{\epsilon}(f)-\bar{R}({\mathbb{P}}_{0}^{*},{\mathbb{P}}_{1}^{*})=I_{1}(f)+I_{0}(f)

(24)

with

I_{1}(f)=\left(\int S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})-{\mathbf{1}}_{f\leq 0}({\mathbf{x}}^{\prime})d\gamma_{1}^{*}\right)+\left(\int C(\eta^{*},f)-C^{*}(\eta^{*})d{\mathbb{P}}_{1}^{*}\right)

I_{0}(f)=\left(\int S_{\epsilon}({\mathbf{1}}_{f>0})({\mathbf{x}})-{\mathbf{1}}_{f>0}({\mathbf{x}}^{\prime})d\gamma_{0}^{*}\right)+\left(\int C(\eta^{*},f)-C^{*}(\eta^{*})d{\mathbb{P}}_{0}^{*}\right)

Lemma 1 implies that $S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})-{\mathbf{1}}_{f\leq 0}({\mathbf{x}}^{\prime})$ must be positive, while the definition of $C^{*}$ implies that $C(\eta^{*},f)-C^{*}(\eta^{*})\geq 0$ .

Similarly, one can express the excess surrogate risk as

\displaystyle R_{\phi}^{\epsilon}(f)-R_{\phi,*}^{\epsilon}

\displaystyle=I_{1}^{\phi}(f)+I_{0}^{\phi}(f)

(25)

with

I_{1}^{\phi}(f)=\left(\int S_{\epsilon}(\phi(f))({\mathbf{x}})-\phi(f)({\mathbf{x}}^{\prime})d\gamma_{1}^{*}\right)+\left(\int C_{\phi}(\eta^{*},f)-C_{\phi}^{*}(\eta^{*})d{\mathbb{P}}_{1}^{*}\right)

I_{0}^{\phi}(f)=\left(\int S_{\epsilon}(\phi(-f))({\mathbf{x}})-\phi(-f)({\mathbf{x}}^{\prime})d\gamma_{0}^{*}\right)+\left(\int C_{\phi}(\eta^{*},f)-C_{\phi}^{*}(\eta^{*})d{\mathbb{P}}_{0}^{*}\right)

Define $K_{\phi}=\frac{3+\sqrt{5}}{2}\cdot\frac{1}{\phi(0)-C_{\phi}^{*}(1/2-\alpha)}$ . We will argue that:

I_{0}(f)\leq K_{\phi}I_{0}^{\phi}(f).

(26)

I_{1}(f)\leq K_{\phi}I_{1}^{\phi}(f).

(27)

Below, we discuss the proof of Equation 27 and an analogous argument will imply Equation 26.

The proof proceeds by splitting the domain $\mathbb{R}^{d}\times\mathbb{R}^{d}$ into three different regions $D_{1}$ , $E_{1}$ $F_{1}$ and proving the inequality in each case with a slightly different argument. These three cases will also appear in the proof of theorem Theorem 13. Define the sets $D_{1}$ , $E_{1}$ , $F_{1}$ by

	$\displaystyle D_{1}=\{({\mathbf{x}},{\mathbf{x}}^{\prime}):S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})-{\mathbf{1}}_{f({\mathbf{x}}^{\prime})\leq 0}=0\}$		(28)
	$\displaystyle E_{1}=\{({\mathbf{x}},{\mathbf{x}}^{\prime}):S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})-{\mathbf{1}}_{f({\mathbf{x}}^{\prime})\leq 0}=1,f({\mathbf{x}}^{\prime})\geq\beta\}$		(29)
	$\displaystyle F_{1}=\{({\mathbf{x}},{\mathbf{x}}^{\prime}):S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})-{\mathbf{1}}_{f({\mathbf{x}}^{\prime})\leq 0}=1,f({\mathbf{x}}^{\prime})<\beta\}$		(30)

where $\beta>0$ is some constant, to be specified later (see Equations 32 and 33). On the set $D_{1}$ the adversarial error matches the non-adversarial error with respect to the distribution ${\mathbb{P}}_{0}^{*}$ , ${\mathbb{P}}_{1}^{*}$ , and thus the bound in Equation 12 implies a linear surrogate bound. On $E_{1}$ , the same argument as Equation 23 together with Equation 12 proves a linear surrogate bound for adversarial risks. In short: this argument uses the first term in $I_{1}^{\phi}(f)$ to bound the first term in $I_{1}(f)$ and the second term of $I_{1}^{\phi}(f)$ to bound the second term of $I_{1}(f)$ .

In contrast, The counterexample discussed in Equation 14 demonstrates that when $f$ is near 0, the quantity $S_{\epsilon}(\phi(f))({\mathbf{x}})-\phi(f)({\mathbf{x}}^{\prime})$ can be small even though $S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})-{\mathbf{1}}_{f\leq 0}({\mathbf{x}}^{\prime})$ can be large. Consequently, a different strategy is required to establish a linear surrogate bound on the set $F_{1}$ . The key observation is that under the assumptions of Proposition 1, the function $f$ must be bounded away from zero whenever it misclassifies a point. As a result, the excess conditional risk $C_{\phi}(\eta,f)-C_{\phi}^{*}(\eta)$ is bounded below by a positive constant and thus can be used to bound terms comprising $I_{1}^{\phi}(f)$ . The constant $\beta$ is then specifically chosen to balance the contribution of the risks over the sets $E_{1}$ and $F_{1}$ .

Proof of Theorem 11.

We will will prove Equation 27, the argument for Equation 26 is analogous. Due due to Equations 24 and 25, these inequalities prove the desired result. First, notice that Equation 12 implies that

C(\eta^{*}({\mathbf{x}}^{\prime}),f({\mathbf{x}}^{\prime}))-C^{*}(\eta^{*}({\mathbf{x}}^{\prime}))\leq\frac{1}{\phi(0)-C_{\phi}^{*}(1/2-\alpha)}\left(C_{\phi}(\eta^{*}({\mathbf{x}}^{\prime}),f({\mathbf{x}}^{\prime}))-C_{\phi}^{*}(\eta^{*}({\mathbf{x}}^{\prime}))\right)\quad{\mathbb{P}}^{*}\text{-a.e.}

(31)

Choose the constant $\beta$ to satisfy

\phi(\beta)=tC_{\phi}^{*}(1/2-\alpha)+(1-t)\phi(0)

(32)

with $t=(3-\sqrt{5})/2$ . The parameter $t$ is specifically selected to balance the contributions of the errors on $E_{1}$ and $F_{1}$ , specifically it should satisfy

\frac{1}{t}=1+\frac{1}{1-t}=\frac{3+\sqrt{5}}{2}=K_{\phi}(\phi(0)-C_{\phi}^{*}(1/2-\alpha))

(33)

Next, we prove the relation Equation 27 on each of the sets $D_{1}$ , $E_{1}$ , $F_{1}$ separately.

1.

On the set $D_{1}$ :

Lemma 1 implies that $S_{\epsilon}(\phi(f))({\mathbf{x}})-\phi(f({\mathbf{x}}^{\prime}))\geq 0$ . This fact together with Equation 31 implies Equation 27.

On the set $E_{1}$ :

If $S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})-{\mathbf{1}}_{f({\mathbf{x}}^{\prime})\leq 0}=1$ but $f({\mathbf{x}}^{\prime})\geq\beta$ , then $S_{\epsilon}(\phi\circ f)({\mathbf{x}})\geq\phi(0)$ while $\phi(f({\mathbf{x}}^{\prime}))\leq\phi(\beta)$ and thus $S_{\epsilon}(\phi\circ f)({\mathbf{x}})-\phi(f({\mathbf{x}}^{\prime}))\geq\phi(0)-\phi(\beta)=t(\phi(0)-C_{\phi}^{*}(1/2-\alpha))=1/K_{\phi}$ by Equation 33. Thus

S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})-{\mathbf{1}}_{f\leq 0}({\mathbf{x}}^{\prime})=1=\frac{K_{\phi}}{K_{\phi}}\leq{K_{\phi}}(S_{\epsilon}(\phi\circ f)({\mathbf{x}})-\phi(f({\mathbf{x}}^{\prime}))

(34)

This relation together with Equation 31 implies Equation 27.

On the set $F_{1}$ :

First, $S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})-{\mathbf{1}}_{f({\mathbf{x}}^{\prime})\leq 0}=1$ implies that $f({\mathbf{x}}^{\prime})>0$ . If additionally $f({\mathbf{x}}^{\prime})<\beta$ , then both $f({\mathbf{x}}^{\prime})<\beta$ and $-f({\mathbf{x}}^{\prime})<\beta$ and consequently $C_{\phi}(\eta^{*},f({\mathbf{x}}^{\prime}))\geq\phi(\beta)$ . Furthermore, as $C_{\phi}^{*}$ is increasing on $[0,1/2]$ and decreasing on $[1/2,1]$ (see Lemma 5 in Section B.2), $\sup_{|\eta-1/2|\geq\alpha}C_{\phi}^{*}(\eta)=C_{\phi}^{*}(1/2-\alpha)$ . Thus due to the choice of $\beta$ in Equation 32:

C_{\phi}(\eta^{*},f({\mathbf{x}}^{\prime}))-C_{\phi}^{*}(\eta^{*})\geq\phi(\beta)-C_{\phi}^{*}(1/2-\alpha)=(1-t)(\phi(0)-C_{\phi}^{*}(1/2-\alpha)).

The same argument as Equation 34 then implies

S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})-{\mathbf{1}}_{f\leq 0}({\mathbf{x}}^{\prime})\leq\frac{1}{(1-t)(\phi(0)-C_{\phi}^{*}(1/2-\alpha))}(C_{\phi}^{*}(\eta^{*},f({\mathbf{x}}^{\prime}))-C_{\phi}^{*}(\eta^{*}))

This relation together with Equations 31, 33 and 1 imply Equation 27.

∎

5 Proof of Theorem 13

Before proving Theorem 13, we will show that this bound is non-vacuous when the adversarial Bayes classifier is unique up to degeneracy. The function $h(z)={\mathbb{P}}(|\eta^{*}-1/2|\leq z)$ is a cdf, and is thus right-continuous in $z$ . Furthermore, if the adversarial Bayes classifier is unique up to degeneracy, then $h(0)=0$ . The following lemma implies that if $H=\operatorname{conc}(h)$ then $H$ is continuous at 0 and $H(0)=0$ . See Appendix E for a proof.

Lemma 2.

Let $h:[0,1/2]\to\mathbb{R}$ be a non-decreasing function with $h(0)=0$ and $h(1/2)=1$ that is right-continuous at $0$ . Then $\operatorname{conc}(h)$ is non-decreasing, $\operatorname{conc}(h)(0)=0$ , and continuous on $[0,1/2]$ .

The first step in proving Theorem 13 is showing an analog of Theorem 11 with $\alpha=0$ for which the linear function is replaced by an $\eta$ -dependent concave function.

Proposition 2.

Let $\Phi$ be a concave non-decreasing function for which $C(\eta,\alpha)-C^{*}(\eta)\leq\Phi(C_{\phi}(\eta,\alpha)-C_{\phi}^{*}(\eta))$ for any $\eta\in[0,1]$ and $\alpha\in\overline{\mathbb{R}}$ . Let ${\mathbb{P}}_{0}^{*}$ , ${\mathbb{P}}_{1}^{*}$ be any two maximzers of $\bar{R}_{\phi}$ for which ${\mathbb{P}}^{*}(\eta^{*}=1/2)=0$ for ${\mathbb{P}}^{*}={\mathbb{P}}_{0}^{*}+{\mathbb{P}}_{1}^{*}$ and $\eta^{*}=d{\mathbb{P}}_{1}^{*}/d{\mathbb{P}}^{*}$ . Let $G:[0,\infty)\to\mathbb{R}$ be any non-decreasing concave function for which the quantity

K=\int\frac{1}{G((\phi(0)-C_{\phi}^{*}(\eta^{*}))/2)}d{\mathbb{P}}^{*}

is finite. Then $R^{\epsilon}(f)-R_{*}^{\epsilon}\leq\tilde{\Phi}(R_{\phi}^{\epsilon}(f)-R_{\phi,*}^{\epsilon})$ , where

\tilde{\Phi}(z)=6\sqrt{KG\left(\frac{1}{6}z\right)}+2\Phi\left(\frac{1}{2}z\right)

(35)

The function $\Psi^{-1}$ in Theorem 4 and the surrogate bounds of Zhang (2004) provide examples of candidate functions for $\Phi$ . As before, this result is proved by dividing the risks $R_{\phi}^{\epsilon}$ , $R^{\epsilon}$ as the sum of four terms as in Equation 24, Equation 25 and then bounding these quantities over the sets $D_{1}$ , $E_{1}$ , and $F_{1}$ defined in Equation 28,Equation 29, and Equation 30 separately. The key observation is that when $f$ is bounded away from $\operatorname*{argmin}C_{\phi}(\eta,\cdot)$ , the excess conditional risk $C_{\phi}(\eta,f)-C_{\phi}^{*}(\eta)$ must be strictly positive. This quantity again serves to bound both components of $I_{1}^{\phi}(f)$ , even if iti s not uniformly bounded away from zero. As before, the constant $\beta$ is selected to balance the contributions of the risk on the sets $E_{1}$ and $F_{1}$ . This time, the value $\beta$ is function of $\eta^{*}({\mathbf{x}}^{\prime})$ , where $\beta:[0,1]\to\overline{\mathbb{R}}$ is a monotonic function for which

\phi(\beta(\eta))=tC_{\phi}^{*}(\eta)+(1-t)\phi(0)=\frac{1}{2}\left(\phi(0)+C_{\phi}^{*}(\eta)\right)

(36)

with $t=1/2$ . In Appendix F, we show that there exists such a function $\beta$ . An argument like the proof of Theorem 11 mixed with applications of the Cauchy-Schwartz and Jensen’s inequality then proves Proposition 2, see Appendix G for details. Again, the function $\beta$ is chosen to balance the contributions of the upper bounds on the risk on $E_{1}$ and $F_{1}$ .

The factor of $1/6$ in Equation 35 arises as an artifact of the proof technique. Specifically, the constant $6=2\cdot 3$ reflects two distinct components of the argument: the factor of 3 results from averaging over three sets $D_{1}$ , $E_{1}$ , $F_{1}$ , (see Equation 68 in Appendix G), the factor of 2 arises from combining the bounds associated with the two integrals $I_{1}(f)$ and $I_{0}(f)$ (see Equations 66 and 68 in Appendix G).

We now turn to the problem of identifying functions $G$ for which the constant $K$ in the preceding proposition is guaranteed to be finite. Observe that $\phi(0)-C_{\phi}^{*}(\eta^{*})\geq\phi(0)-C_{\phi}^{*}(1/2)$ and so if $\phi(0)>C_{\phi}^{*}(1/2)$ , the identity function is a possible choice for $G$ . This option results in

\tilde{\Phi}(z)=\frac{2}{\phi(0)-C_{\phi}^{*}(1/2)}z+2\Phi\left(\frac{1}{2}z\right),

which may improve the convergence rate relative to the bound in Theorem 11. The results developed here extend the classical analysis of Bartlett et al. (2006) to the adversarial setting. Moreover, Proposition 2 points to a pathway for generalizing the framework of Zhang (2004) to robust classification.

Alternatively, we consider constructing a function $G$ for which the constant $K$ in Proposition 2 is always finite when the adversarial Bayes classifier is unique, but distribution dependent. Observe that if $h$ is the cdf of $|\eta-1/2|$ and $h$ is continuous, then $\int 1/h^{r}dh$ is always finite. This calculation suggests $\Phi=h\circ\Psi^{-1}$ , with $\Psi$ defined in Theorem 4. To ensure the concavity of $G$ , we instead select $G=H\circ\Psi^{-1}$ with $H=\operatorname{conc}(h)$ .

Lemma 3.

Assume $C_{\phi}^{*}(1/2)=\phi(0)$ . Let ${\mathbb{P}}_{1}$ , ${\mathbb{P}}_{0}$ , ${\mathbb{P}}_{1}^{*}$ , ${\mathbb{P}}_{0}^{*}$ , $\phi$ , $H$ , and $\Psi$ be as in Theorem 13. Let $\Lambda(z)=\Psi^{-1}(\min(z,\phi(0)))$ . Then for any $r\in(0,1)$ , Then

R^{\epsilon}(f)-R^{\epsilon}_{*}\leq\tilde{\Phi}(R_{\phi}^{\epsilon}(f)-R_{\phi,*}^{\epsilon})

(37)

with

\tilde{\Phi}(z)=6\sqrt{\frac{1}{1-r}2^{r}H\left(\Lambda\left(\frac{1}{6}z\right)\right)^{r}}+2\Lambda\left(\frac{z}{2}\right).

Proof.

For convenience, let $\bar{\Lambda}(z)=\frac{1}{2}\Lambda(2z)$ . Let $G=(H\circ\bar{\Lambda})^{r}$ , where $h(z)={\mathbb{P}}^{*}(|\eta^{*}-1/2|\leq z)$ . Then $G$ is concave because it is the composition of an concave function and an increasing concave function. We will verify that

K=\int\frac{1}{G((\phi(0)-C_{\phi}^{*}(\eta^{*}))/2)}d{\mathbb{P}}^{*}\leq\frac{2^{r}}{1-r}

First,

\displaystyle\int\frac{1}{G((\phi(0)-C_{\phi}^{*}(\eta^{*}))/2)}d{\mathbb{P}}^{*}=\int\frac{1}{H(|\eta^{*}-1/2|)^{r}}d{\mathbb{P}}^{*}=\int_{[0,\frac{1}{2}]}\frac{1}{H(s)^{r}}d{\mathbb{P}}^{*}\sharp s=\int_{(0,\frac{1}{2}]}\frac{1}{H(s)^{r}}d{\mathbb{P}}^{*}\sharp s

with $s=|\eta^{*}-1/2|$ . The assumption ${\mathbb{P}}^{*}(|\eta^{*}=1/2|)=0$ allows us to drop 0 from the domain of integration. Because the function $H$ is continuous on $(0,1]$ by Lemma 2, this last expression can actually be evaluated as a Riemann-Stieltjes integral with respect to the function $h(s)={\mathbb{P}}(|\eta^{*}-1/2|\leq s)$ :

\displaystyle\int\frac{1}{H(s)^{r}}d{\mathbb{P}}^{*}\sharp s=\int\frac{1}{H(s)^{r}}dh

(38)

This result is standard when ${\mathbb{P}}^{*}$ is Lebesgue measure, (see for instance Theorem 5.46 of (Wheeden & Zygmund, 1977)). We prove equality in Equation 38 for strictly decreasing functions in Proposition 4 in Section H.1.

Finally, the integral in Equation 38 can be bounded as

\displaystyle\int\frac{1}{H(s)^{r}}dh\leq\frac{2^{r}}{1-r}

(39)

If $h$ were differentiable, then the chain rule would imply

\int\frac{1}{H(s)^{r}}dh\leq\int\frac{1}{h(s)^{r}}dh=\int_{0}^{1}\frac{1}{h(s)^{r}}h^{\prime}(s)dz=\frac{1}{1-r}h(s)^{1-r}\bigg{|}_{0}^{1}=\frac{1}{1-r}.

This calculation is more delicate for non-differentiable $H$ ; we formally prove inequality in Equation 39 in Section H.2.

This calculation proves the inequality Equation 37 with $\tilde{\Phi}$ as

6\sqrt{\frac{\cdot 2^{r}}{1-r}H\left(\frac{1}{2}\Lambda\left(\frac{2}{6}z\right)\right)^{r}}+\Lambda(z)

The concavity of $\Lambda$ together with the fact that $\Lambda(0)=0$ then proves the result. ∎

Minimizing this bound over $r$ then produces Theorem 13, see Appendix I for details.

6 Related Works

The most similar results to this paper are Li & Telgarsky (2023); Mao et al. (2023a). Li & Telgarsky (2023) prove a surrogate bound for convex losses, when one can minimize over the thresh-holding value in Equation 13 rather than just 0. (Mao et al., 2023a) proves an adversarial surrogate bound for a modified $\rho$ -margin loss.

Many papers study the statistical consistency of surrogate risks in the standard and adversarial context. Bartlett et al. (2006); Zhang (2004) prove surrogate risk bounds that apply to the class of all measurable functions Lin (2004); Steinwart (2007) prove further results on consistency in the standard setting, and Frongillo & Waggoner (2021) study the optimally of such surrogate risk bounds. (Bao, 2023) relies on the modulus of convexity of $C_{\phi}^{*}$ to construct surrogate risk bounds. Philip M. Long (2013); Mingyuan Zhang (2020); Awasthi et al. (2022); Mao et al. (2023a; b); Awasthi et al. (2023b) further study consistency restricted to a specific family of functions; a concepts called $\mathcal{H}$ -consistency. Prior work Mahdavi et al. (2014)also uses these surrogate risk bounds in conjunction with surrogate generalization bounds to study the generalization of the classification error.

In the adversarial setting, (Meunier et al., 2022; Frank & Niles-Weed, 2024a) identify which losses are adversarially consistent for all data distributions while (Frank, 2025) shows that under reasonable distributional assumptions, a consistent loss is adversarially consistent for a specific distribution iff the adversarial Bayes classifier is unique up to degeneracy. (Awasthi et al., 2021) study adversarial consistency for a well-motivated class of linear functions while some prior work also studies the approximation error caused by learning from a restricted function class $\mathcal{H}$ . Liu et al. (2024) study the approximation error of the surrogate risk. Complimenting this result, Awasthi et al. (2023b); Mao et al. (2023a) study $\mathcal{H}$ -consistency in the adversarial setting for specific surrogate risks. Standard and adversarial surrogate risk bounds are a tool central tool in the derivation of the $\mathcal{H}$ -consistency bounds in this line of research. Whether the adversarial surrogate bounds presented in this paper could result in improved adversarial $\mathcal{H}$ -consistency bounds remains an open problem.

Our proofs rely on prior results that study adversarial risks and adversarial Bayes classifiers. Notably, (Bungert et al., 2021; Pydi & Jog, 2021; 2020; Bhagoji et al., 2019; Awasthi et al., 2023a) establish the existence of the adversarial Bayes classifier while (Frank & Niles-Weed, 2024b; Pydi & Jog, 2020; 2021; Bhagoji et al., 2019; Frank, 2025) prove various minimax theorems for the adversarial surrogate and classification risks. Subsequently, (Pydi & Jog, 2020) uses a minimax theorem to study the adversarial Bayes classifier, and (Frank, 2024) uses minimax results to study the notion of uniqueness up to degeneracy.

7 Conclusion

In conclusion, we prove surrogate risk bounds for adversarial risks whenever $\phi$ is adversarially consistent for the distribution ${\mathbb{P}}_{0}$ , ${\mathbb{P}}_{1}$ . When $\phi$ is adversarially consistent or the distribution of optimal adversarial attacks satisfies Massart’s noise condition, we prove a linear surrogate risk bound. For the general case, we prove a concave distribution-dependent bound. Understanding the optimality of these bounds remains an open problem, as does understanding how these bounds interact with the sample complexity of estimating the surrogate quantity. These questions were partly addressed by (Frongillo & Waggoner, 2021) and (Mahdavi et al., 2014) in the standard setting, but remain unstudied in the adversarial scenario.

Acknowledgments

Natalie Frank was supported in part by the Research Training Group in Modeling and Simulation funded by the National Science Foundation via grant RTG/DMS – 1646339, NSF grant DMS-2210583, and NSF TRIPODS II - DMS 2023166.

References

Ambrosio et al. (2000) Luigi Ambrosio, Nicola Fosco, and Diego Pallara. Functions of Bounded Variation and Free Discontuity Problems. Oxford Mathematics Monographs. Oxford University Press, 2000.
Apostol (1974) Tom M. Apostol. Mathematical analysis, 1974.
Awasthi et al. (2021) Pranjal Awasthi, Natalie S. Frank, Anqui Mao, Mehryar Mohri, and Yutao Zhong. Calibration and consistency of adversarial surrogate losses. NeurIps, 2021.
Awasthi et al. (2022) Pranjal Awasthi, Anqi Mao, Mehryar Mohri, and Yutao Zhong. H-consistency bounds for surrogate loss minimizers. In Proceedings of the 39th International Conference on Machine Learning. PMLR, 2022.
Awasthi et al. (2023a) Pranjal Awasthi, Natalie S. Frank, and Mehryar Mohri. On the existence of the adversarial bayes classifier (extended version). arxiv, 2023a.
Awasthi et al. (2023b) Pranjal Awasthi, Anqi Mao, Mehryar Mohri, and Yutao Zhong. Theoretically grounded loss functions and algorithms for adversarial robustness. In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent (eds.), Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research. PMLR, 2023b.
Bao (2023) Han Bao. Proper losses, moduli of convexity, and surrogate regret bounds. In Proceedings of Thirty Sixth Conference on Learning Theory, Proceedings of Machine Learning Research. PMLR, 2023.
Bartlett et al. (2006) Peter L. Bartlett, Michael I. Jordan, and Jon D. McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473), 2006.
Bhagoji et al. (2019) Arjun Nitin Bhagoji, Daniel Cullina, and Prateek Mittal. Lower bounds on adversarial robustness from optimal transport, 2019.
Biggio et al. (2013) Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Joint European conference on machine learning and knowledge discovery in databases, pp. 387–402. Springer, 2013.
Buja et al. (2005) Andreas Buja, Werner Stuetzle, and Yi Shen. Loss functions for binary class probability estimation and classification: Structure and applications. Technical report, University of Pennslvania, 2005.
Bungert et al. (2021) Leon Bungert, Nicolás García Trillos, and Ryan Murray. The geometry of adversarial training in binary classification. arxiv, 2021.
Frank (2024) Natalie S. Frank. A notion of uniqueness for the adversarial bayes classifier, 2024.
Frank (2025) Natalie S. Frank. Adversarial consistency and the uniqueness of the adversarial bayes classifier. European Journal of Applied Mathematics, 2025.
Frank & Niles-Weed (2024a) Natalie S. Frank and Jonathan Niles-Weed. The adversarial consistency of surrogate risks for binary classification. NeurIps, 2024a.
Frank & Niles-Weed (2024b) Natalie S. Frank and Jonathan Niles-Weed. Existence and minimax theorems for adversarial surrogate risks in binary classification. Journal of Machine Learning Research, 2024b.
Frongillo & Waggoner (2021) Rafael Frongillo and Bo Waggoner. Surrogate regret bounds for polyhedral losses. In Advances in Neural Information Processing Systems, 2021.
Hiriart-Urruty & Lemaréchal (2001) Jean-Baptiste Hiriart-Urruty and Claude Lemaréchal. Fundamentals of convex analysis, 2001.
Jylhä (2014) Heikki Jylhä. The $l^{\infty}$ optimal transport: Infinite cyclical monotonicity and the existence of optimal transport maps. Calculus of Variations and Partial Differential Equations, 2014.
Li & Telgarsky (2023) Justin D. Li and Matus Telgarsky. On achieving optimal adversarial test error, 2023.
Lin (2004) Yi Lin. A note on margin-based loss functions in classification. Statistics & Probability Letters, 68(1):73–82, 2004.
Liu et al. (2024) Changyu Liu, Yuling Jiao, Junhui Wang, and Jian Huang. Nonasymptotic bounds for adversarial excess risk under misspecified models. SIAM Journal on Mathematics of Data Science, 6(4), 2024. URL https://doi.org/10.1137/23M1598210.
Mahdavi et al. (2014) Mehrdad Mahdavi, Lijun Zhang, and Rong Jin. Binary excess risk for smooth convex surrogates, 2014.
Mao et al. (2023a) Anqi Mao, Mehryar Mohri, and Yutao Zhong. Cross-entropy loss functions: Theoretical analysis and applications, 2023a.
Mao et al. (2023b) Anqi Mao, Mehryar Mohri, and Yutao Zhong. Structured prediction with stronger consistency guarantees. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems. Curran Associates, Inc., 2023b.
Massart & Nédélec (2006) Pascal Massart and Élodie Nédélec. Risk bounds for statistical learning. The Annals of Statistics, 34, 2006.
Meunier et al. (2022) Laurent Meunier, Raphaël Ettedgui, Rafael Pinot, Yann Chevaleyre, and Jamal Atif. Towards consistency in adversarial classification. arXiv, 2022.
Mingyuan Zhang (2020) Shivani Agarwal Mingyuan Zhang. Consistency vs. h-consistency: The interplay between surrogate loss functions and the scoring function class. NeurIps, 2020.
Paschali et al. (2018) Magdalini Paschali, Sailesh Conjeti, Fernando Navarro, and Nassir Navab. Generalizability vs. robustness: Adversarial examples for medical imaging. Springer, 2018.
Philip M. Long (2013) Rocco A. Servedio Philip M. Long. Consistency versus realizable h-consistency for multiclass classification. ICML, 2013.
Pydi & Jog (2020) Muni Sreenivas Pydi and Varun Jog. Adversarial risk via optimal transport and optimal couplings. ICML, 2020.
Pydi & Jog (2021) Muni Sreenivas Pydi and Varun Jog. The many faces of adversarial risk. Neural Information Processing Systems, 2021.
Raftery & Gneiting (2007) Adrian Raftery and Raftery Gneiting. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistcal Association, 2007.
Reid & Williamson (2009) Mark D. Reid and Robert C. Williamson. Surrogate regret bounds for proper losses. In Proceedings of the 26th Annual International Conference on Machine Learning, New York, NY, USA, 2009. Association for Computing Machinery.
Rudin (1976) Walter Rudin. Principles of Mathematical Analysis. Mathematics Series. McGraw-Hill International Editions, third edition, 1976.
Savage (1971) Leonard J. Savage. Elicitation of personal probabilities and expectations. Journal of the American Statistical Association, 66(336), 1971.
Schervish (1989) Mark J. Schervish. A general method for comparing probability assessors. The Annals of Statistics, 1989.
Stein & Shakarchi (2005) Elias Stein and Rami Shakarchi. Real analysis: Measure Theory, Integration, and Hilbert Spaces. Princeton University Press, 2005.
Steinwart (2007) Ingo Steinwart. How to compare different loss functions and their risks. Constructive Approximation, 2007.
Szegedy et al. (2013) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
Wheeden & Zygmund (1977) Richard L. Wheeden and Antoni Zygmund. Measure and Integral. Pure and Applied Mathematics. Marcel Dekker Inc., 1977.
Xu et al. (2022) Ying Xu, Kiran Raja, Raghavendra Ramachandra, and Christoph Busch. Adversarial Attacks on Face Recognition Systems, pp. 139–161. Springer International Publishing, Cham, 2022.
Zhang (2004) Tong Zhang. Statistical behavior and consistency of classification methods based on convex risk minimization. The Annals of Statistics, 2004.

Contents of Appendix

\startcontents

[appendix] \printcontents[appendix]l1

Appendix A Proof of Theorem 3

Lemma 4.

Assume $\phi$ is consistent. Then $C_{\phi}^{*}(\eta)=\phi(0)$ implies that $\eta=1/2$ .

This result appeared as Lemma 7 of (Frank, 2025).

Proof.

If $\phi$ is consistent and $0$ minimizes $C_{\phi}(\eta,\alpha)$ , then $0$ must also minimize $C(\eta,\alpha)=\eta{\mathbf{1}}_{\alpha\leq 0}+(1-\eta){\mathbf{1}}_{\alpha>0}$ and consequently $\eta\leq 1/2$ . However $C_{\phi}(\eta,\alpha)=C_{\phi}(1-\eta,-\alpha)$ so that $0$ must minimize $C(1-\eta,-\alpha)$ as well. Consequently, $1-\eta\leq 1/2$ and thus $\eta$ must actually equal $1/2$ . ∎

Proof of Theorem 3.

Forward direction: Assume that $\phi$ is consistent. Note that $C_{\phi}^{*}(\eta)\leq C_{\phi}(\eta,0)=\phi(0)$ for any $\eta$ . Thus Lemma 4 implies that $C_{\phi}^{*}(\eta)<\phi(0)$ for $\eta\neq 1/2$ .

Backward direction: Assume that $C_{\phi}^{*}(\eta)<\phi(0)$ for all $\eta\neq 1/2$ . Notice that if $\eta=1/2$ , $C(1/2,\alpha)$ is constant in $\alpha$ so any sequence $\alpha_{n}$ minimizes $C(1/2,\cdot)$ . We will show if $\eta>1/2$ and $\alpha_{n}$ is a minimizing sequence of $C_{\phi}(\eta,\cdot)$ , then $\alpha_{n}>0$ for sufficiently large $n$ , and thus must also minimize $C(\eta,\cdot)$ . An analogous argument will imply that if $\eta<1/2$ , any minimizing sequence of $C_{\phi}(\eta,\cdot)$ must also minimize $C(\eta,\cdot)$ as well.

Assume $\eta>1/2$ and let $\alpha_{n}$ be any minimizing sequence of $C_{\phi}(\eta,\cdot)$ . Let $\alpha^{*}$ be a limit point of the sequence $\alpha_{n}$ in the extended real number line $\overline{\mathbb{R}}$ . Then $\alpha^{*}$ is a minimizer of $C_{\phi}(\eta,\alpha)$ . The statement $C_{\phi}^{*}(\eta)=C_{\phi}(\eta,\alpha^{*})<\phi(0)$ implies that at least one of $\phi(\alpha^{*})$ and $\phi(-\alpha^{*})$ must be larger that or equal to $\phi(0)$ , and so the other must be strictly less than $\phi(0)$ . Because $\eta>1/2$ and $\alpha^{*}$ is a minimizer of $C_{\phi}(\eta,\cdot)$ , one can conclude that $\phi(\alpha^{*})<\phi(0)$ and consequently $\alpha^{*}>0$ .

Therefore, every limit point of the sequence $\{\alpha_{n}\}$ is strictly positive. Consequently, one can conclude that $\alpha_{n}>0$ for sufficiently large $n$ .

∎

Appendix B (Non-Adversarial) Surrogate Bounds

B.1 The Realizable Case— Proof of Equation 10

Proof of Equation 10.

If $R_{*}=0$ , then ${\mathbb{P}}(\eta=0\text{ or }1)=1$ and consequently $C_{\phi}^{*}(\eta)=0$ ${\mathbb{P}}$ -a.e. As a result, $R_{\phi,*}=0$ and thus it remains to show that $R(f)\leq\frac{1}{\phi(0)}R_{\phi}(f)$ for all functions $f$ . Next, we will prove the bound

{\mathbf{1}}_{f({\mathbf{x}})\leq 0}\leq\frac{1}{\phi(0)}\phi(f({\mathbf{x}})).

(40)

The inequality Equation 40 trivially holds when ${\mathbf{1}}_{f(x)\leq 0}=0$ . Alternatively, the relation ${\mathbf{1}}_{f({\mathbf{x}})\leq 0}=1$ implies $f({\mathbf{x}})\leq 0$ and consequently $\phi(f({\mathbf{x}}))\geq\phi(0)$ . Thus whenever ${\mathbf{1}}_{f({\mathbf{x}})\leq 0}=1$ ,

{\mathbf{1}}_{f({\mathbf{x}})\leq 0}=\frac{\phi(0)}{\phi(0)}\leq\frac{1}{\phi(0)}\phi(f({\mathbf{x}})).

An analogous argument implies

{\mathbf{1}}_{f({\mathbf{x}})>0}={\mathbf{1}}_{-f({\mathbf{x}})<0}\leq\frac{1}{\phi(0)}\phi(-f({\mathbf{x}})).

As a result:

	$\displaystyle R(f)$	$\displaystyle=\int{\mathbf{1}}_{f({\mathbf{x}})\leq 0}d{\mathbb{P}}_{1}+\int{\mathbf{1}}_{f({\mathbf{x}})>0}d{\mathbb{P}}_{0}\leq\frac{1}{\phi(0)}\left(\int\phi(f({\mathbf{x}}))d{\mathbb{P}}_{1}+\int\phi(-f({\mathbf{x}}))d{\mathbb{P}}_{0}\right)$
		$\displaystyle=\frac{1}{\phi(0)}R_{\phi}(f)$

∎

B.2 Linear Surrogate Risk Bounds—Proof of Proposition 1

In this appendix, we will find it useful to study the function

C_{\phi}^{-}(\eta)=\inf_{z(2\eta-1)\leq 0}C_{\phi}(\eta,z)

introduced by (Bartlett et al., 2006). This function maps $\eta$ to the smallest value of the conditional $\phi$ -risk assuming an incorrect classification. The symmetry $C_{\phi}(\eta,\alpha)=C_{\phi}(1-\eta,-\alpha)$ implies $C_{\phi}^{-}(\eta)=C_{\phi}^{-}(1-\eta)$ . Further, the function $C_{\phi}^{-}$ is concave on each of the intervals $[0,1/2]$ and $[1/2,1]$ , as it is an infimum of linear functions on each of these regions. The next result examines the monotonicity properties of $C_{\phi}^{*}$ and $C_{\phi}^{-}$ .

Lemma 5.

The function $C_{\phi}^{*}$ is non-decreasing on $[0,1/2]$ and non-increasing on $[1/2,1]$ . In contrast, $C_{\phi}^{-}$ is non-increasing on $[0,1/2]$ and non-decreasing on $[1/2,1]$

Proof.

The symmetry $C_{\phi}^{*}(\eta)=C_{\phi}^{*}(1-\eta)$ and $C_{\phi}^{-}(\eta)=C_{\phi}^{-}(1-\eta)$ implies that it suffices to check monotonicity on $[0,1/2]$ . Observe that

C_{\phi}(\eta,\alpha)-C_{\phi}(\eta,-\alpha)=\eta(\phi(\alpha)-\phi(-\alpha))+(1-\eta)(\phi(\alpha)-\phi(-\alpha))=(2\eta-1)(\phi(\alpha)-\phi(-\alpha)).

If $\eta\leq 1/2$ , then this quantity is non-negative when $\alpha\leq 0$ . Therefore, when computing $C_{\phi}^{*}$ over $[0,1/2]$ , it suffices to minimize $C_{\phi}(\eta,\alpha)$ over $\alpha\leq 0$ . In other words, for $\eta\leq 1/2$ ,

C_{\phi}^{*}(\eta)=\inf_{\alpha}C_{\phi}(\eta,\alpha)=\inf_{\alpha\leq 0}C_{\phi}(\eta,\alpha)

For any fixed $\alpha\leq 0$ , the quantity $C_{\phi}(\eta,\alpha)$ is non-increasing in $\eta$ and thus $C_{\phi}^{*}(\eta_{1})\leq C_{\phi}^{*}(\eta_{2})$ when $\eta_{1}\leq\eta_{2}\leq 12$ .

In contrast, for any $\alpha\geq 0$ , the quantity $C_{\phi}(\eta,\alpha)$ is non-decreasing in $\eta$ and thus $C_{\phi}^{-}(\eta_{1})\geq C_{\phi}^{-}(\eta_{2})$ when $\eta_{1}\leq\eta_{2}\leq 12$ .

∎

Next we’ll prove a useful lower bound on $C_{\phi}^{-}$ .

Lemma 6.

For all $\eta\in[0,1]$ ,

C_{\phi}^{-}(\eta)\geq(1-2\eta)\phi(0)+2\eta C_{\phi}^{*}(\eta)

(41)

Proof.

First, observe that $\eta$ is the convex combination $\eta=2\eta\cdot 1/2+(1-2\eta)\cdot 0$ . By the concavity of $C_{\phi}^{-}$ on $[0,1/2]$ ,

\displaystyle C_{\phi}^{-}(\eta)

\displaystyle=C_{\phi}^{-}\left(2\eta\cdot\frac{1}{2}+(1-2\eta)\cdot 0\right)\geq(1-2\eta)C_{\phi}^{-}(0)+2\eta C_{\phi}^{-}\left(\tfrac{1}{2}\right)

However, $C_{\phi}^{-}(0)=\phi(0)$ while $C_{\phi}^{-}(1/2)=C_{\phi}^{*}(1/2)$ . Further, Lemma 5 implies that $C_{\phi}^{*}(1/2)\geq C_{\phi}^{*}(\eta)$ , yielding the desired inequality.

∎

Proof of Proposition 1.

If $C(\eta,f)-C^{*}(\eta)=0$ then Equation 11 holds trivially. Otherwise, $C(\eta,f)-C^{*}(\eta)=|2\eta-1|$ . If $C(\eta,f)=|2\eta-1|$ , then

	$\displaystyle C(\eta,f)-C^{*}(\eta)$	$\displaystyle=\|2\eta-1\|=\|2\eta-1\|\cdot\frac{\phi(0)-C_{\phi}^{}(\eta)}{\phi(0)-C_{\phi}^{}(\eta)}$		(42)
		$\displaystyle\leq\frac{1}{\phi(0)-C_{\phi}^{}(\eta)}\left(\left(\|2\eta-1\|\phi(0)+(1-\|2\eta-1\|)C_{\phi}^{}(\eta)\right)-C_{\phi}^{*}(\eta)\right)$		(42)

At the same time, because $|\eta-1/2|\geq\alpha$ ${\mathbb{P}}$ -a.e. Lemma 5 implies that $C_{\phi}^{*}(\eta)\leq C_{\phi}^{*}(1/2-\alpha)$ ${\mathbb{P}}$ -a.e. Furthermore, applying Equation 41 at $1-\eta$ and observing $C_{\phi}^{*}(\eta)=C_{\phi}^{*}(1-\eta)$ shows that

|2\eta-1|\phi(0)+(1-|2\eta-1|)C_{\phi}^{*}(\eta)\leq C_{\phi}^{-}(\eta).

Therefore, Equation 42 is bounded above by

\displaystyle\leq\frac{1}{\phi(0)-C_{\phi}^{*}\left(\frac{1}{2}-\alpha\right)}\left(C_{\phi}^{-}(\eta)-C_{\phi}^{*}(\eta)\right)\leq\frac{1}{\phi(0)-C_{\phi}^{*}\left(\frac{1}{2}-\alpha\right)}\left(C_{\phi}(\eta,f)-C_{\phi}^{*}(\eta)\right).

(43)

The last equality follows from the assumption $C(\eta,f)-C^{*}(\eta)=|2\eta-1|$ , as it implies $(2\eta-1)f\leq 0$ , and thus $C_{\phi}(\eta,f)\geq C_{\phi}^{-}(\eta)$ . Consequently, Equation 43 implies Equation 11.

Integrating Equation 11 with respect to ${\mathbb{P}}$ then produces the surrogate bound Equation 12.

∎

Appendix C Proof of Lemma 1

Proof of Lemma 1.

If ${\mathbf{x}}^{\prime}\in\overline{B_{\epsilon}({\mathbf{x}})}$ then $S_{\epsilon}(g)({\mathbf{x}})\geq g({\mathbf{x}}^{\prime})$ . Thus if $\gamma$ is supported on $\Delta_{\epsilon}$ , then $S_{\epsilon}(g)({\mathbf{x}})\geq g({\mathbf{x}}^{\prime})$ $\gamma$ -a.e. Integrating this inequality in $\gamma$ produces

\int S_{\epsilon}(g)d{\mathbb{Q}}\geq\int gd{\mathbb{Q}}^{\prime}.

Taking the supreumum over all ${\mathbb{Q}}\in{{\mathcal{B}}^{\infty}_{\epsilon}}({\mathbb{Q}})$ then proves the result. ∎

Appendix D Proof of Theorem 10 Item 1)

In this appendix, we relax the regularity assumption of the loss functions: specifically we require loss functions to be lower semi-continuous rather than continuous.

Assumption 2.

The loss $\phi$ is lower semi-continuous, non-increasing, and $\lim_{\alpha\to\infty}\phi(\alpha)=0$ .

Frank & Niles-Weed (2024b) establish their result under this weaker condition, so Theorem 9 continues to hold for lower semi-continuous losses. Moreover, Theorem 7 of (Frank & Niles-Weed, 2024b) finds two conditions that characterize minimizers and maximizers of $R_{\phi}^{\epsilon}$ and $\bar{R}_{\phi}$ .

Theorem 14 (Complementary Slackness).

The function $f^{*}$ minimizes $R_{\phi}^{\epsilon}$ and the measures $({\mathbb{P}}_{0}^{*},{\mathbb{P}}_{1}^{*})$ maximize $\bar{R}_{\phi}$ over ${{\mathcal{B}}^{\infty}_{\epsilon}}({\mathbb{P}}_{0})\times{{\mathcal{B}}^{\infty}_{\epsilon}}({\mathbb{P}}_{1})$ iff the following two conditions hold:

\int S_{\epsilon}(\phi(f^{*}))d{\mathbb{P}}_{1}=\int\phi(f^{*})d{\mathbb{P}}_{1}^{*}\quad\text{and}\quad\int S_{\epsilon}(\phi(-f^{*}))d{\mathbb{P}}_{0}=\int\phi(-f^{*})d{\mathbb{P}}_{0}^{*}

2)

$C_{\phi}(\eta^{*},f^{*})=C_{\phi}^{*}(\eta^{*})\quad{\mathbb{P}}^{*}\text{-a.e.}$

To prove Item 1) of Theorem 10, one uses the properties above to show that $R^{\epsilon}(f^{*})\leq\bar{R}({\mathbb{P}}_{0}^{*},{\mathbb{P}}_{1}^{*})$ for some function $f^{*}$ and any pair of measures ${\mathbb{P}}_{0}^{*}$ , ${\mathbb{P}}_{1}^{*}$ that maximize $\bar{R}_{\phi}$ . When $\phi$ is strictly decreasing at 0, the identity $\{{\mathbf{1}}_{\phi(f({\mathbf{x}}))\geq\phi(0)}={\mathbf{1}}_{f({\mathbf{x}})\leq 0}\}$ provides a means to relate $\bar{R}_{\phi}$ and $\bar{R}$ . In Sections D.1 and D.2, we show that for any loss $\phi$ , there is another loss $\psi$ with $C_{\phi}^{*}=C_{\psi}^{*}$ that satisfies this property.

Lemma 7.

Let $\phi$ be a consistent loss function that satisfies Assumption 1. Then there is a non-increasing, lower semi-continuous loss function $\psi$ with $\lim_{\alpha\to\infty}\phi(\alpha)=0$ for which $C_{\phi}^{*}(\eta)=C_{\psi}^{*}(\eta)$ and

\psi(\alpha)>\psi(0)\quad\quad\text{for all }\alpha<0.

(44)

The function $\psi$ may fail to be continuous.

Proof of Item 1) of Theorem 10 .

Let $\psi$ be the loss function constructed in Lemma 7 for which $C_{\psi}^{*}=C_{\phi}^{*}$ and let $\gamma_{1}^{*}$ be the coupling between ${\mathbb{P}}_{1}$ and ${\mathbb{P}}_{1}^{*}$ that achieves the minimum $W_{\infty}$ distance. We aim to show that

\int S_{\epsilon}({\mathbf{1}}_{f^{*}\leq 0})d{\mathbb{P}}_{1}=\int{\mathbf{1}}_{f^{*}\leq 0}d{\mathbb{P}}_{1}^{*}

(45)

and similarly,

\int S_{\epsilon}({\mathbf{1}}_{f^{*}>0})d{\mathbb{P}}_{0}=\int{\mathbf{1}}_{f^{*}>0}d{\mathbb{P}}_{0}^{*}.

(46)

Combining Equations 45 and 46 yields $R^{\epsilon}(f^{*})=\bar{R}({\mathbb{P}}_{0}^{*},{\mathbb{P}}_{1}^{*})$ , and thus ${\mathbb{P}}_{0}^{*}$ , ${\mathbb{P}}_{1}^{*}$ are optimal by Theorem 7.

We now prove Equation 45; the argument for Equation 46 is analogous, and we briefly outline the necessary modifications at the end of the proof.

First observe that the property Equation 44 implies that ${\mathbf{1}}_{f({\mathbf{y}})\leq 0}={\mathbf{1}}_{\psi(f({\mathbf{y}}))\geq\psi(0)}$ pointwise and thus $S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})=S_{\epsilon}({\mathbf{1}}_{\psi(f)\geq\psi(0)})({\mathbf{x}})$ . We will apply the complimentary slackness conditions of Theorem 14 to argue that in fact $S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})={\mathbf{1}}_{f({\mathbf{x}}^{\prime})\leq 0}$ $\gamma_{1}^{*}$ -a.e.

Item 1) of Theorem 14 implies that

S_{\epsilon}(\psi\circ f^{*})({\mathbf{x}})=\psi\circ f^{*}({\mathbf{x}}^{\prime})\quad\gamma_{1}^{*}\text{-a.e.}

(47)

and thus $\psi\circ f^{*}$ assumes its maximum over closed $\epsilon$ -balls ${\mathbb{P}}_{1}$ -a.e. Now because $f^{*}$ assumes its maximum over closed $\epsilon$ -balls ${\mathbb{P}}_{1}$ -a.e., one can further conclude that $S_{\epsilon}({\mathbf{1}}_{\psi\circ f^{*}\geq\psi(0)})={\mathbf{1}}_{S_{\epsilon}(\psi\circ f^{*})\geq\psi(0)}$ ${\mathbb{P}}_{1}$ -a.e. and therefore

S_{\epsilon}({\mathbf{1}}_{\psi\circ f^{*}\geq\psi(0)})({\mathbf{x}})={\mathbf{1}}_{S_{\epsilon}(\psi\circ f^{*})({\mathbf{x}})\geq\psi(0)}={\mathbf{1}}_{\psi(f^{*}({\mathbf{x}}^{\prime}))\geq\psi(0)}\quad\gamma_{1}^{*}\text{-a.e.}

(48)

Next, the property Equation 44 implies that ${\mathbf{1}}_{f^{*}\leq 0}={\mathbf{1}}_{\psi\circ f^{*}\geq\psi(0)}$ . Consequently, Equation 48 implies that

S_{\epsilon}({\mathbf{1}}_{f^{*}\leq 0})={\mathbf{1}}_{\psi(f^{*}({\mathbf{x}}^{\prime}))\geq\psi(0)}\quad\gamma_{1}^{*}\text{-a.e.}

Now, Equation 44 again implies that ${\mathbf{1}}_{\psi(f^{*}({\mathbf{x}}^{\prime}))\geq\psi(0)}={\mathbf{1}}_{f^{*}({\mathbf{x}}^{\prime})\leq 0}$ . Combining with Equation 48 proves that $S_{\epsilon}({\mathbf{1}}_{\psi\circ f^{*}\geq\psi(0)})={\mathbf{1}}_{f^{*}({\mathbf{x}}^{\prime})\leq 0}$ and integrating this statement results in Equation 45.

Similarly, if $\gamma_{0}^{*}$ is the coupling between ${\mathbb{P}}_{0}$ and ${\mathbb{P}}_{0}^{*}$ that achieves the minimum $W_{\infty}$ distance, then

S_{\epsilon}(\psi\circ-f^{*})({\mathbf{x}})=\psi\circ-f^{*}({\mathbf{x}}^{\prime})\quad\gamma_{0}^{*}\text{-a.e.}

and thus $\psi\circ-f^{*}$ assumes its maximum over closed $\epsilon$ -balls ${\mathbb{P}}_{0}$ -a.e. and analogous reasoning to Equation 48 implies that

S_{\epsilon}({\mathbf{1}}_{\psi\circ-f^{*}({\mathbf{x}})\geq\psi(0)})={\mathbf{1}}_{S_{\epsilon}(\psi\circ-f^{*}({\mathbf{x}}))\geq\psi(0)}={\mathbf{1}}_{\psi(-f^{*}({\mathbf{x}}^{\prime}))\geq\psi(0)}\quad\gamma_{0}^{*}\text{-a.e.}

(49)

Finally, Equation 44 implies that ${\mathbf{1}}_{f^{*}>0}={\mathbf{1}}_{\psi(-f^{*})<\psi(0)}$ . Consequently, the same argument as for Equation 45 implies Equation 46.

∎

D.1 Proof of Lemma 7

Proper losses are loss functions that can be used to learn a probability value rather than a binary classification decision. These losses are typically studied as functions on $[0,1]$ rather than $\mathbb{R}$ . Prior work (Bao, 2023; Buja et al., 2005; Reid & Williamson, 2009) considers losses of the form $\ell(c,\hat{\eta})$ where $\hat{\eta}\in[0,1]$ is the estimated probability value of the binary class $c\in\{0,1\}$ . One can define the minimal conditional risk just as before via

C_{\ell}^{*}(\eta)=\inf_{\hat{\eta}\in[0,1]}\eta\ell(1,\hat{\eta})+(1-\eta)\ell(0,\hat{\eta})

Savage (1971) first studied how to reconstruct the loss $\ell$ given a concave function $H$ . We adapt their argument to construct a proper loss $\ell$ on $[0,1]$ , and then compose this loss with a link to obtain a loss on $\mathbb{R}$ with the desired properties. This reconstruction involves technical properties of concave functions.

Lemma 8.

For a continuous concave function $H:[0,1]\to\mathbb{R}$ , the left-hand ( $D^{-}H$ ) and right-hand ( $D^{+}H$ ) derivatives always exist. The right-hand derivative $D^{+}H$ is non-increasing and right-continuous.

Furthermore, if $H$ has a strict maximum at $1/2$ , then the function $H_{+}^{\prime}:[0,1]\to\partial H(\eta)$ defined by

H_{+}^{\prime}(\eta)=\begin{cases}D^{+}H(\eta)&\eta\neq 1/2\\ 0&\eta=1/2\end{cases}

(50)

is non-increasing and right-continuous.

See Section D.2 for a proof.

Proposition 3.

Let $H:[0,1]\twoheadrightarrow[0,K]$ be a concave function for which the only global maximum is at $1/2$ . Define $\ell$ by

\ell_{H}(1,\hat{\eta})=H(\hat{\eta})+(1-\hat{\eta})H_{+}^{\prime}(\hat{\eta}),\quad\ell_{H}(0,\hat{\eta})=H(\hat{\eta})-\hat{\eta}H_{+}^{\prime}(\hat{\eta})

Then $C_{\ell_{H}}^{*}=H$ .

Notice that Lemma 8 implies that $\ell_{H}$ is lower semi-continuous. Furthermore, when $H(\eta)=H(1-\eta)$ , the definitions of $\ell_{H}(1,\hat{\eta})$ and $\ell_{H}(0,\hat{\eta})$ imply that

\ell_{H}(1,1-\hat{\eta})=\ell_{H}(0,\hat{\eta}).

(51)

Indeed, Equation 51 suggests that $\ell_{H}(1,\hat{\eta})$ corresponds to $\phi(\alpha)$ while $\ell_{H}(0,\hat{\eta})$ corresponds to $\phi(-\alpha)$ . Fixing the value of $H_{+}^{\prime}$ at $1/2$ to zero ensures that $\ell_{H}(1,\cdot)$ , $\ell_{H}(0,\cdot)$ coincide $\hat{\eta}=1/2$ , which corresponds to $\alpha=0$ for the losses $\phi(\alpha)$ , $\phi(-\alpha)$ . This correspondence is formalized in the proof of proof of Lemma 7.

Proof of Proposition 3.

Calculating $C_{\ell_{H}}^{*}$ for the loss $\ell_{H}$ defined above results in

C_{\ell_{H}}^{*}(\eta)=\inf_{\hat{\eta}\in[0,1]}H(\hat{\eta})+(\eta-\hat{\eta})H_{+}^{\prime}(\hat{\eta})

The choice $\hat{\eta}=\eta$ results in $C_{\ell_{H}}^{*}(\eta)\leq H(\eta)$ .

However, the concavity of $H$ implies that

H(\hat{\eta})+H_{+}^{\prime}(\hat{\eta})(\eta-\hat{\eta})\geq H(\eta)

and as a result:

C_{\ell_{H}}^{*}(\eta)\geq H(\eta).

Therefore, $C_{\ell_{H}}^{*}(\eta)=H(\eta)$ .

∎

Next, an integral representation of $\ell_{H}$ proves that $\ell_{H}(1,\hat{\eta})$ is non-decreasing. Let $\beta(c)=-H_{+}^{\prime}(c)$ , then

\ell_{H}(1,\hat{\eta})=\int_{\hat{\eta}}^{1}(1-c)d\beta(c),\quad\ell_{H}(0,\hat{\eta})=\int_{0}^{\hat{\eta}}cd\beta(c),

(52)

where the integrals in Equation 52 are Reimann-Stieltjes integrals. Lemma 8 implies that $\beta$ is right-continuous and non-decreasing, and thus these integrals are in fact well defined.

Such a representation was first given in (Schervish, 1989, Theorem 4.2) for left-continuous losses and Lebesgue integrals, see also (Raftery & Gneiting, 2007, Theorem 3) for a discussion of this result. (Reid & Williamson, 2009, Theorem 2) offer an alternative proof of Equation 52 terms of generalized derivatives.

Writing these integrals as Riemann-Stieltjes integrals provides a more streamlined proof— integration by parts for Riemann-Stieltjes integrals (see for instance (Apostol, 1974, Theorem 7.6, Chapter 7)) implies that the first integral in Equation 52 evaluates to $\ell_{H}(1,\hat{\eta})$ and the second evaluates to $\ell_{H}(0,\hat{\eta})$ .

Corollary 1.

The function $\ell_{H}(0,\hat{\eta})$ is upper semi-continuous while the function $\ell_{H}(1,\hat{\eta})$ is lower semi-continuous. Furthermore, $\ell_{H}(1,\hat{\eta})$ is non-increasing and $\ell_{H}(0,\hat{\eta})$ is non-decreasing.

Proof.

The representation Equation 52 implies that $\ell_{H}(1,\hat{\eta})$ is non-increasing and $\ell_{H}(0,\hat{\eta})$ is non-decreasing in $\hat{\eta}$ .

Lemma 8 implies that $D^{+}H(\hat{\eta})$ is both non-increasing and right-continuous. For non-increasing functions, right-continuous and lower semi-continuous is equivalent. Consequently, $\ell_{H}(0,\hat{\eta})$ is upper semi-continuous while the function $\ell_{H}(1,\hat{\eta})$ is lower semi-continuous.

∎

Finally, we will use the $\ell_{H}$ defined in the previous result to construct the $\psi$ of Lemma 7.

Proof of Lemma 7.

Define a function $\sigma:\overline{\mathbb{R}}\to[0,1]$ via $\sigma(\alpha)=\frac{1}{1+e^{-\alpha}}$ for $\alpha\in\mathbb{R}$ and extend via continuity to $\pm\infty$ . Notice that this function satisfies

\sigma(-\alpha)=1-\sigma(\alpha).

(53)

To simplify, notation, we let $H(\eta)=C_{\phi}^{*}(\eta)$ and let $\ell_{H}$ be the loss function defined by Proposition 3. Define the loss $\psi$ by

\psi(\alpha)=\ell_{H}(1,\sigma(\alpha)).

This composition is non-increasing and lower semi-continuous.

Next, the function $C_{\phi}^{*}$ computes to $H$ :

$\displaystyle C_{\phi}^{*}(\eta)$	$\displaystyle=\inf_{\alpha\in\mathbb{R}}\eta\ell_{H}(1,\sigma(\alpha))+(1-\eta)\ell_{H}(1,\sigma(-\alpha))$
	$\displaystyle=\inf_{\alpha\in\mathbb{R}}\eta\ell_{H}(1,\sigma(\alpha))+(1-\eta)\ell_{H}(1,1-\sigma(\alpha))$	(Equation 53)
	$\displaystyle=\inf_{\hat{\eta}\in[0,1]}\eta\ell_{H}(1,\hat{\eta})+(1-\eta)\ell_{H}(1,1-\hat{\eta})$
	$\displaystyle=\inf_{\hat{\eta}\in[0,1]}\eta\ell_{H}(1,\hat{\eta})+(1-\eta)\ell_{H}(0,\hat{\eta})$	(Equation 51)
	$\displaystyle=H(\eta)$	(Proposition 3)

Finally, we’ll argue that $\psi(\alpha)>\psi(0)$ for all $\alpha<0$ . To start, note that $\psi(0)=H(1/2)$ . The assumption that $\phi$ is consistent together with Theorem 3 implies that $D^{+}H(\eta)$ must be strictly positive on $(0,1/2)$ and strictly negative on $(1/2,1]$ . Next, the concavity of $H$ implies that

\psi(0)=H(1/2)\leq H(\eta)+D^{+}H(\eta)\cdot(1/2-\eta).

If furthermore $\eta<1/2$ , then both $D^{+}H(\eta)$ and $(1/2-\eta)$ are strictly positive and consequently one can conclude that

\psi(0)<H(\eta)+H_{+}^{\prime}(\eta)\cdot(1-\eta)=\ell_{H}(1,\eta).

(54)

However, $\alpha<0$ implies $\sigma(\alpha)<1/2$ and consequently Equation 54 implies that when $\alpha<0$ ,

\psi(0)<\ell_{H}(1,\sigma(\alpha))=\psi(\alpha).

It remains to compute $\lim_{\alpha\to\infty}\psi(\alpha)$ :

\lim_{\alpha\to\infty}\psi(\alpha)=\ell_{H}(1,1)=H(1)=0

∎

D.2 Proof of Lemma 8

Proof of Lemma 8.

We’ll begin by proving existence and subsequently we’ll show that this function is non-increasing and right-continuous.

Problem 23 of Chapter 4, (Rudin, 1976) implies that for a concave function $H$ , if $0<s<t<u<1$ , then

\frac{H(t)-H(s)}{t-s}\geq\frac{H(u)-H(s)}{u-s}\geq\frac{H(u)-H(t)}{u-t}.

(55)

The continuity of $H$ then implies that this inequality holds for $0\leq s<t<u\leq 1$ . The first inequality implies the the right-hand limit $D^{+}H(s)$ exists for $s\in[0,1)$ while the second inequality implies the left-hand limit $D^{-}H(s)$ exists for $s\in(0,1]$ .

Next, we’ll prove that $D^{+}H$ is non-increasing. First, Equation 55 implies that if $0\leq s<t<u<v\leq 1$ then

\frac{H(t)-H(s)}{t-s}\geq\frac{H(v)-H(u)}{v-u}

(56)

and consequently taking the limits $t\downarrow s$ and $v\downarrow u$ proves that the function $D^{+}H$ is non-increasing.

Next, we prove that $D^{+}H$ is right-continuous. Fix a point $s$ and define $K=\lim_{x\downarrow s}D^{+}H(x)$ . We will argue that $D^{+}H(s)\leq K$ , and consequently, $\lim_{x\downarrow s}D^{+}H(x)=D^{+}H(s)$ . Equation 56 implies that for any $x,u,v$ satisfying $s<x<u<v$ ,

K\geq D^{+}H(x)\geq\frac{H(v)-H(u)}{v-u}

and thus

K(v-u)\geq H(v)-H(u)

for any $s<u<v$ . Thus by the continuity of $H$ , this inequality must extend to $u=s$ ,

K(v-s)\geq H(v)-H(s)

and taking the limit $v\downarrow s$ then proves

K\geq D^{+}H(s)

and thus the monotonicity of $H$ implies $D^{+}H(s)=K$ .

Finally, if $H$ has a strict maximum at $1/2$ , then the super-differential $\partial H(\eta)$ can include 0 only at $\eta=1/2$ . Thus, as $D^{+}H$ is non-increasing, $D^{+}H(\eta)>0$ when $\eta<1/2$ and $D^{+}H(\eta)<0$ when $\eta>1/2$ . Thus the function $H_{+}^{\prime}$ defined in Equation 50 is non-increasing and right-continuous.

∎

Appendix E Proof of Lemma 2

We define the concave conjugate of a function $h$ as

h_{*}(y)=\inf_{x\in\operatorname{dom}(h)}yx-h(x)

Recall that $\operatorname{conc}(h)$ as defined in Equation 18 is the biconjugate $h_{**}$ . Consequently, $\operatorname{conc}(h)$ can be expressed as

\operatorname{conc}(h)(x)=\inf\{\ell(x):\ell\text{ linear, and }\ell\geq h\text{ on }\operatorname{dom}(h)\}

(57)

One can prove Lemma 2 by studying properties of concave conjugates.

Lemma 9.

Let $h:[a,b]\to\mathbb{R}$ be a non-decreasing function. Then $\operatorname{conc}(h)$ is non-decreasing as well.

Proof.

We will argue that if $h$ is non-decreasing, then it suffices to consider the infimum in Equation 57 over non-decreasing linear functions. Observe that if $\ell$ is a decreasing linear function with $\ell(x)\geq h(x)$ then the constant function $\ell(b)$ satisfies $\ell(b)\geq h(x)$ and $\ell(b)\leq\ell(x)$ . Therefore,

\operatorname{conc}(h)(x)=\inf\{\ell(x):\ell\text{ linear, non-decreasing, and }\ell\geq h\}

∎

Lemma 10.

Let $h:[0,b]\to\mathbb{R}$ be a non-decreasing function that is right-continuous at zero with $h(0)=0$ . Then $\sup_{y}h_{*}(y)=0$ . Furthermore, there is a sequence $y_{n}$ with $y_{n}\to\infty$ and $\lim_{n\to\infty}h_{*}(y_{n})=0$ .

Proof.

First, notice that

h_{*}(y)=\inf_{x\in[0,b]}yx-h(x)\leq y\cdot 0-h(0)=0

(58)

for any $y\in\mathbb{R}$ . It remains to show a sequence $y_{n}$ for which $\lim_{n\to\infty}h_{*}(y_{n})=0$ .

We will argue than any sequence $y_{n}$ with

y_{n}>nh(b)\geq\sup_{x\in[1/n,b]}\frac{h(x)}{x}

(59)

satisfies this property.

If $x\in[1/n,b]$ and $y_{n}$ satisfies Equation 59 then

xy_{n}-h(x)=x\left(y_{n}-\frac{h(x)}{x}\right)>0

and thus Equation 58 implies that

h_{*}(y_{n})=\inf_{x\in[0,1/n)}xy_{n}-h(x)

The monononicity of $h$ then implies that

h_{*}(y_{n})\geq-h(1/n)

and

\lim_{n\to\infty}h_{*}(y_{n})\geq 0

because $h$ is right-continuous at zero. This relation together with Equation 58 implies the result.

∎

Proof of Lemma 2.

Lemma 9 implies that $\operatorname{conc}(h)$ is non-decreasing. Standard results in convex analysis imply that $\operatorname{conc}(h)$ is continuous on $(0,1/2)$ (Hiriart-Urruty & Lemaréchal, 2001, Lemma 3.1.1) and upper semi-continuous on $[0,1/2]$ (Hiriart-Urruty & Lemaréchal, 2001, Theorem 1.3.5). Thus monotonicity implies that for all $x\in[0,1/2]$ , $\operatorname{conc}(h)(x)\leq\operatorname{conc}(h)(1/2)$ and thus $\lim_{x\to 1/2}\operatorname{conc}(h)(x)\leq\operatorname{conc}(h)(1/2)$ . We will show the opposite inequality, implying that $\operatorname{conc}h$ is continuous at $1/2$ .

First, as the constant function $h(1/2)$ is an upper bound on $h$ , one can conclude that $\operatorname{conc}(h)(1/2)=h(1/2)=1$ . Next, recall that $\operatorname{conc}(h)$ can be expressed as an infimum of linear functions as in Equation 57. If $\ell\geq h$ , then $\ell(0)\geq 0$ and $\ell(1/2)\geq 1$ . Therefore,

\ell(\tfrac{1}{2}-\delta)=\ell((1-2\delta)\cdot\tfrac{1}{2}+2\delta\cdot 0)=(1-2\delta)\ell(\tfrac{1}{2})+2\delta\ell(0)\geq 1-2\delta.

Therefore, the representation Equation 57 implies that $\operatorname{conc}(h)(1/2-\delta)\geq 1-2\delta$ . Taking $\delta\to 0$ proves that $\lim_{x\to 1/2}\operatorname{conc}(h)(x)\geq 1$ . Thus, $\operatorname{conc}(h)$ is continuous at $1/2$ , if viewed as a function on $[0,1/2]$ .

Next, Lemma 10 implies that $h_{**}(0)=0$ :

h_{**}(0)=\inf_{y\in\mathbb{R}}-h_{*}(y)=-\sup_{y\in\mathbb{R}}h_{*}(y)=0.

Finally, it remains to show that $h_{**}$ is continuous at 0. The monotonicity of $h_{**}$ implies that $\lim_{y\to 0^{+}}h_{**}(y)=\inf_{y\in(0,1/2]}h_{**}(y)$ and consequently

	$\displaystyle\lim_{y\to 0^{+}}h_{*}(y)=\inf_{y\in(0,1/2]}\inf_{x\in\mathbb{R}}yx-h_{}(x)=\inf_{x\in_{\mathbb{R}}}\inf_{y\in(0,1/2]}yx-h_{}(x)=\inf_{x\in\mathbb{R}}-h_{}(x)+\begin{cases}0&\text{if }x\geq 0\\ \frac{x}{2}&\text{if }x<0\end{cases}$
	$\displaystyle=\min\left(\inf_{x\geq 0}-h_{}(x),\inf_{x<0}\frac{x}{2}-h_{}(x)\right)$		(60)

However, Lemma 10 implies that

\inf_{x\geq 0}-h_{*}(x)=\inf_{x\in\mathbb{R}}-h_{*}(x)=0

(61)

Notice that if $x\leq 0$ ,

h_{*}(x)=\inf_{z\in[0,1/2]}xz-h(z)=\frac{x}{2}-h\left(\frac{1}{2}\right)=\frac{x}{2}-1

(62)

Consequently, Equation 61 and Equation 62 implies that Equation 60 evaluates to 0.

∎

Appendix F Defining the Function $\beta$

Lemma 11.

Let $\beta:[0,1]\to\overline{\mathbb{R}}$ be a function defined by

\beta(\eta)=\inf\{\alpha:\phi(\alpha)=\frac{1}{2}(\phi(0)+C_{\phi}^{*}(\eta))\}

Then $C_{\phi}^{*}(\beta(\eta))=\frac{1}{2}(\phi(0)+C_{\phi}^{*}(\eta))$ , and $\beta$ is monotonic on $[0,1/2]$ and $[1/2,1]$ .

Proof.

The statement $C_{\phi}^{*}(\beta(\eta))=\frac{1}{2}(\phi(0)+C_{\phi}^{*}(\eta))$ is a consequence of the continuity of $\phi$ . Recall that $\phi$ is non-increasing, and $C_{\phi}^{*}$ is non-decreasing on $[0,1/2]$ . Thus, $\beta(\eta)$ must be non-increasing as well on $[0,1/2]$ . Analogously, $\beta$ must be non-decreasing on $[1/2,1]$ .

∎

Appendix G Proof of Proposition 2

A modified version of Jensen’s inequality will be used at several points in the proof of Proposition 2.

Lemma 12.

Let $G$ be a concave function with $G(0)=0$ and let $\nu$ be a measure with $\nu(\mathbb{R}^{d})\leq 1$ . Then

\int G(f)d\nu\leq G\left(\int fd\nu\right)

Proof.

The inequality trivially holds if $\nu(\mathbb{R}^{d})=0$ , so we assume $\nu(\mathbb{R}^{d})>0$ . Jensen’s inequality implies that

\int G(f)d\nu=\nu(\mathbb{R}^{d})\left(\frac{1}{\nu(\mathbb{R}^{d})}\int G(f)d\nu\right)\leq\nu(\mathbb{R}^{d})G\left(\frac{1}{\nu(\mathbb{R}^{d})}\int fd\nu\right).

As $G(0)=0$ , concavity implies that

\nu(\mathbb{R}^{d})G\left(\frac{1}{\nu(\mathbb{R}^{d})}\int fd\nu\right)+(1-\nu(\mathbb{R}^{d})G(0)\leq G\left(\int fd\nu\right)

∎

Proof of Proposition 2.

Define $I_{1}(f)$ , $I_{0}(f)$ , $I_{1}^{\phi}(f)$ , and $I_{0}^{\phi}(f)$ as in Section 4. We will prove

I_{1}(f)\leq\frac{1}{2}\tilde{\Phi}\big{(}2I_{1}^{\phi}(f)\big{)}

(63)

and an analogous argument will imply

I_{0}(f)\leq\frac{1}{2}\tilde{\Phi}\big{(}2I_{0}^{\phi}(f)\big{)}.

(64)

The concavity of $\tilde{\Phi}$ then implies that

R^{\epsilon}(f)-R_{*}^{\epsilon}=I_{1}(f)+I_{0}(f)\leq\frac{1}{2}\tilde{\Phi}\big{(}2I_{1}^{\phi}(f)\big{)}+\frac{1}{2}\tilde{\Phi}\big{(}2I_{0}^{\phi}(f)\big{)}\leq\tilde{\Phi}\Big{(}\frac{1}{2}2I_{1}^{\phi}(f)+\frac{1}{2}2I_{0}^{\phi}(f)\Big{)}=\tilde{\Phi}\big{(}R_{\phi}^{\epsilon}(f)-R_{\phi,*}^{\epsilon}\big{)}.

We will prove Equation 63, the argument for Equation 64 is analogous. Next, let $\gamma_{1}^{*}$ be the coupling between ${\mathbb{P}}_{1}$ and ${\mathbb{P}}_{1}^{*}$ supported on $\Delta_{\epsilon}$ . The assumption on $\Phi$ implies that

C(\eta^{*}({\mathbf{x}}^{\prime}),f({\mathbf{x}}^{\prime}))-C^{*}(\eta^{*}({\mathbf{x}}^{\prime}))\leq\Phi\big{(}C_{\phi}(\eta^{*}({\mathbf{x}}^{\prime}),f({\mathbf{x}}^{\prime}))-C_{\phi}^{*}(\eta^{*}({\mathbf{x}}^{\prime}))\big{)}

(65)

and consequently,

\int C(\eta^{*}({\mathbf{x}}^{\prime}),f({\mathbf{x}}^{\prime}))-C^{*}(\eta^{*}({\mathbf{x}}^{\prime}))d\gamma_{1}^{*}\leq\Phi\left(\int C_{\phi}(\eta^{*}({\mathbf{x}}^{\prime}),f({\mathbf{x}}^{\prime}))-C_{\phi}^{*}(\eta^{*}({\mathbf{x}}^{\prime}))d\gamma_{1}^{*}\right)\leq\Phi(I_{1}^{\phi}(f)).

(66)

To bound the term $S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})-{\mathbf{1}}_{f({\mathbf{x}}^{\prime})\leq 0}$ , we consider three different cases for $({\mathbf{x}},{\mathbf{x}}^{\prime})$ . Define the sets $D_{1}$ , $E_{1}$ , $F_{1}$ by

D_{1}=\{({\mathbf{x}},{\mathbf{x}}^{\prime}):S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})-{\mathbf{1}}_{f({\mathbf{x}}^{\prime})\leq 0}=0\}

E_{1}=\{({\mathbf{x}},{\mathbf{x}}^{\prime}):S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})-{\mathbf{1}}_{f({\mathbf{x}}^{\prime})\leq 0}=1,f({\mathbf{x}}^{\prime})\geq\beta(\eta^{*}({\mathbf{x}}^{\prime}))\}

F_{1}=\{({\mathbf{x}},{\mathbf{x}}^{\prime}):S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})-{\mathbf{1}}_{f({\mathbf{x}}^{\prime})\leq 0}=1,f({\mathbf{x}}^{\prime})<\beta(\eta^{*}({\mathbf{x}}^{\prime}))\}

with the function $\beta$ as in Equation 36. The composition $\beta\circ\eta^{*}$ is measurable because $\beta$ is piecewise monotonic, see Lemma 11 in Appendix F. We will show that if $T_{1}$ is any of the three sets $D_{1}$ , $E_{1}$ , $F_{1}$ , then

		$\displaystyle\int_{T_{1}}S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})-{\mathbf{1}}_{f({\mathbf{x}}^{\prime})\leq 0}d\gamma_{1}^{*}$		(67)
		$\displaystyle\leq\left(\int\frac{1}{G\Big{(}\big{(}\phi(0)-C_{\phi}^{}(\eta^{}({\mathbf{x}}^{\prime}))\big{)}/2\Big{)}}d\gamma_{1}^{}\right)^{\frac{1}{2}}G\left(\int_{T_{1}}\big{(}(S_{\epsilon}(\phi\circ f)({\mathbf{x}})-\phi(f({\mathbf{x}}^{\prime}))\big{)}+\left(C_{\phi}(\eta^{}({\mathbf{x}}^{\prime}),f({\mathbf{x}}^{\prime}))-C_{\phi}^{}(\eta^{}({\mathbf{x}}^{\prime}))\right)d\gamma_{1}^{*}\right)^{\frac{1}{2}}$		(67)

Thus because $G$ is concave and non-decreasing, the composition $\sqrt{G}$ is as well. Thus summing the inequality Equation 67 over $T_{1}\in\{D_{1},E_{1},F_{1}\}$ results in

\displaystyle\int S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})-{\mathbf{1}}_{f({\mathbf{x}}^{\prime})\leq 0}d\gamma_{1}^{*}

\displaystyle\leq 3\left(\int\frac{1}{G\Big{(}\big{(}\phi(0)-C_{\phi}^{*}(\eta^{*}({\mathbf{x}}^{\prime}))\big{)}/2\Big{)}}d{\mathbb{P}}^{*}\right)^{\frac{1}{2}}G\left(\frac{1}{3}I_{1}^{\phi}(f)\right)^{\frac{1}{2}}

(68)

Summing Equation 66 and Equation 68 results in Equation 63.

It remains to show the inequality Equation 67 for the three sets $D_{1}$ , $E_{1}$ , $F_{1}$ .

A)

On the set $D_{1}$ :

If $S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})={\mathbf{1}}_{f({\mathbf{x}}^{\prime})\leq 0}$ , then $\int_{D_{1}}S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})-{\mathbf{1}}_{f({\mathbf{x}}^{\prime})\leq 0}d\gamma_{1}^{*}=0$ while the left-hand side of Equation 67 is non-negative by Lemma 1, which implies Equation 67 for $T_{1}=D_{1}$ .

On the set $E_{1}$ :

If $S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})-{\mathbf{1}}_{f({\mathbf{x}}^{\prime})\leq 0}=1$ but $f({\mathbf{x}}^{\prime})\geq\beta(\eta^{*}({\mathbf{x}}^{\prime}))$ , then $S_{\epsilon}(\phi\circ f)({\mathbf{x}})\geq\phi(0)$ while $\phi(f({\mathbf{x}}^{\prime}))\leq\phi(\beta(\eta^{*}({\mathbf{x}}^{\prime})))$ and thus $S_{\epsilon}(\phi\circ f)({\mathbf{x}})-\phi(f({\mathbf{x}}^{\prime}))\geq\phi(0)-\phi(\beta(\eta^{*}({\mathbf{x}}^{\prime})))=(\phi(0)-C_{\phi}^{*}(\eta^{*}({\mathbf{x}}^{\prime})))/2$ . Lemma 1 then implies that $S_{\epsilon}(\phi\circ f)({\mathbf{x}})-\phi(f({\mathbf{x}}^{\prime}))\geq 0$ $\gamma_{1}^{*}$ -a.e. and thus

S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})-{\mathbf{1}}_{f({\mathbf{x}}^{\prime})\leq 0}=1=\frac{\sqrt{G\left(\left(\phi(0)-C_{\phi}^{*}(\eta^{*}({\mathbf{x}}^{\prime}))\right)/2\right)}}{\sqrt{G\left(\left(\phi(0)-C_{\phi}^{*}(\eta^{*}({\mathbf{x}}^{\prime}))\right)/2\right)}}\leq\frac{\sqrt{G\big{(}S_{\epsilon}(\phi\circ f)({\mathbf{x}})-\phi(f({\mathbf{x}}^{\prime}))\big{)}}}{\sqrt{G\left(\left(\phi(0)-C_{\phi}^{*}(\eta^{*}({\mathbf{x}}^{\prime}))\right)/2\right)}}\quad\gamma_{1}^{*}\text{-a.e.}

(69)

Now the Cauchy-Schwartz inequality and the variant of Jensen’s inequality on Lemma 12 imply

		$\displaystyle\int_{E_{1}}S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})-{\mathbf{1}}_{f({\mathbf{x}}^{\prime})\leq 0}d\gamma_{1}^{*}$		(70)
		$\displaystyle\leq\left(\int_{E_{1}}\frac{1}{G\left(\left(\phi(0)-C_{\phi}^{}(\eta^{}({\mathbf{x}}^{\prime}))\right)/2\right)}d\gamma_{1}^{}\right)^{\frac{1}{2}}\left(\int_{E_{1}}G\big{(}S_{\epsilon}(\phi\circ f)({\mathbf{x}})-\phi(f({\mathbf{x}}^{\prime}))\big{)}d\gamma_{1}^{}\right)^{\frac{1}{2}}$
		$\displaystyle\leq\left(\int\frac{1}{G\left(\left(\phi(0)-C_{\phi}^{}(\eta^{}({\mathbf{x}}^{\prime}))\right)/2\right)}d\gamma_{1}^{}\right)^{\frac{1}{2}}G\left(\int_{E_{1}}S_{\epsilon}(\phi\circ f)({\mathbf{x}})-\phi(f({\mathbf{x}}^{\prime}))d\gamma_{1}^{}\right)^{\frac{1}{2}}$

On the set $F_{1}$ :

First, $S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})-{\mathbf{1}}_{f({\mathbf{x}}^{\prime})\leq 0}=1$ implies that $f({\mathbf{x}}^{\prime})>0$ . If furthermore $f({\mathbf{x}}^{\prime})<\beta(\eta^{*}({\mathbf{x}}^{\prime}))$ , then both $f({\mathbf{x}}^{\prime})<\beta(\eta^{*}({\mathbf{x}}^{\prime}))$ and $-f({\mathbf{x}}^{\prime})<\beta(\eta^{*}({\mathbf{x}}^{\prime}))$ and consequently $C_{\phi}(\eta^{*}({\mathbf{x}}^{\prime}),f({\mathbf{x}}^{\prime}))\geq\phi(\beta(\eta^{*}({\mathbf{x}}^{\prime})))$ , and so due to the definition of $\beta$ in Equation 36:

C_{\phi}(\eta^{*},f({\mathbf{x}}^{\prime}))-C_{\phi}^{*}(\eta^{*})\geq\frac{1}{2}(\phi(0)-C_{\phi}^{*}(\eta^{*})).

The same argument as Equation 69 then implies

S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})-{\mathbf{1}}_{f\leq 0}({\mathbf{x}}^{\prime})\leq\frac{\sqrt{G\big{(}C_{\phi}^{*}(\eta^{*},f({\mathbf{x}}^{\prime}))-C_{\phi}^{*}(\eta^{*}({\mathbf{x}}^{\prime}))\big{)}}}{\sqrt{G\left(\left(\phi(0)-C_{\phi}^{*}(\eta^{*}({\mathbf{x}}^{\prime}))\right)/2\right)}}\quad\gamma_{0}^{*}\text{-a.e.}

Now the Cauchy-Schwartz inequality and Lemma 12 imply

		$\displaystyle\int_{F_{1}}S_{\epsilon}({\mathbf{1}}_{f\leq 0})({\mathbf{x}})-{\mathbf{1}}_{f({\mathbf{x}}^{\prime})\leq 0}d\gamma_{1}^{*}\leq$
		$\displaystyle\left(\int_{F_{1}}\frac{1}{G\left(\left(\phi(0)-C_{\phi}^{}(\eta^{}({\mathbf{x}}^{\prime}))\right)/2\right)}d\gamma_{1}^{}\right)^{\frac{1}{2}}\left(\int_{F_{1}}G\big{(}C_{\phi}^{}(\eta^{}({\mathbf{x}}^{\prime}),f({\mathbf{x}}^{\prime}))-C_{\phi}^{}(\eta^{}({\mathbf{x}}^{\prime}))\big{)}d\gamma_{1}^{}\right)^{\frac{1}{2}}\leq$
		$\displaystyle\left(\int\frac{1}{G\left(\left(\phi(0)-C_{\phi}^{}(\eta^{}({\mathbf{x}}^{\prime}))\right)/2\right)}d{\mathbb{P}}_{1}^{}\right)^{\frac{1}{2}}G\left(\int_{F_{1}}C_{\phi}^{}(\eta^{},f({\mathbf{x}}^{\prime}))-C_{\phi}^{}(\eta^{})d\gamma_{1}^{}\right)^{\frac{1}{2}}$

∎

Appendix H Technical Integral Lemmas

H.1 The Lebesgue and Riemann–Stieltjes integral of an increasing function

The goal of this section is to prove Equation 38, or namely:

Proposition 4.

Let $f$ be a non-increasing, non-negative, continuous function on an interval $[a,b]$ and let ${\mathbb{Q}}$ be a finite positive measure. Let $z$ be a random variable distributed according to ${\mathbb{Q}}$ and define $h(\alpha)={\mathbb{Q}}(z\leq\alpha)$ then

\int_{(a,b]}f(z)d{\mathbb{Q}}(z)=\int_{a}^{b}f(\alpha)dh(\alpha)

where the integral on the left is defined as the Lebesgue integral in terms of the measure ${\mathbb{Q}}$ while the integral on the right is defined as a Riemann–Stieltjes integral.

Proof.

Recall that the Lebesgue integral $\int fd{\mathbb{Q}}$ is defined as

\int fd{\mathbb{Q}}=\sup\left\{\int gd{\mathbb{Q}}:g\leq f,g\text{ simple function, }\right\}

while the Riemann-Stieltjes integral is defined as the value of the limits

\int fdh=\lim_{\Delta\alpha_{i}\to 0}\sum_{i=0}^{I-1}f(\alpha_{i})(h(\alpha_{i+1})-h(\alpha_{i}))=\lim_{\Delta\alpha_{i}\to 0}\sum_{i=0}^{I-1}f(\alpha_{i+1})(h(\alpha_{i+1})-h(\alpha_{i})),

(71)

where these limits are evaluated as the size of the partition $\Delta\alpha_{i}=\alpha_{i+1}-\alpha_{i}$ approaches 0. A standard analysis result states that these limits exist and are equal whenever $f$ is continuous (see for instance Theorem 2.24 of (Wheeden & Zygmund, 1977)). Let $\delta>0$ be arbitrary and choose a partition $\{\alpha_{i}\}_{i=0}^{I}$ for which each of the sums in Equation 71 is within $\delta$ of $\int fdh$ , and $f(\alpha_{i})-f(\alpha_{i+1})<\delta$ for all $i$ . Such a partition exists because every continuous function on a compact set is uniformly continuous.

Next, consider two simple functions $g_{1}$ , $g_{2}$ defined according to

g_{1}(z)=\sum_{i=0}^{I-1}f(\alpha_{i+1})\chi_{z\in(\alpha_{i},\alpha_{i+1}]},\quad g_{2}(z)=\sum_{i=0}^{I-1}f(\alpha_{i})\chi_{z\in(\alpha_{i},\alpha_{i+1}]}.

By construction, $g_{1}(x)\leq f(x)$ for all $x\in(a,b]$ . Moreover, since $f(\alpha_{i})-f(\alpha_{i+1})<\delta$ , it follows that $f(x)\leq g_{2}(x)+\delta$ when $x\in(a,b]$ . Now applying the definition of the integral of a simply function, we obtain:

	$\displaystyle\int fdh-\delta\leq\sum_{i=0}^{I-1}f(\alpha_{i+1})\big{(}h(\alpha_{i+1})-h(\alpha_{i})\big{)}=\int_{(a,b]}g_{1}d{\mathbb{Q}}\leq\int_{(a,b]}fd{\mathbb{Q}}\leq\int_{(a,b]}g_{2}+\delta d{\mathbb{Q}}$
	$\displaystyle=\sum_{i=0}^{I-1}f(\alpha_{i})\big{(}h(\alpha_{i+1})-h(\alpha_{i})\big{)}+\delta{\mathbb{Q}}((a,b])\leq\int fdh+\delta+\delta{\mathbb{Q}}((a,b])$

As $\delta$ is arbitrary, it follows that $\int fdh=\int fd{\mathbb{Q}}$ . ∎

Notice that because $H(0)=0$ , the integral in the right-hand side of Equation 38 is technically an improper integral. Thus to show Equation 38, one can conclude that

\int_{z\in(\delta,1/2]}\frac{1}{H(z)}d{\mathbb{Q}}(z)=\int_{\delta}^{1/2}\frac{1}{H(\alpha)}dh(\alpha)

from Proposition 4 and then take the limit $\delta\to 0$ .

H.2 Proof of the last equality in Equation 39

Even though $H=\operatorname{conc}(h)$ is always continuous by Lemma 2, $h$ may be discontinuous and thus $\int h^{-r}dh$ may not exist as a Riemann-Stieltjes integral. Thus one must avoid this quantity in proofs.

The proof of Equation 39 relies on summation by parts and on a well-behaved decomposition of $h$ that is a consequence of the Lebesgue decomposition of a function of bounded variation. This result states that $h$ can be decomposed into a continuous portion and a right-continuous “jump” portion.

See Corollary 3.33 of (Ambrosio et al., 2000) for a statement that implies this result or (Stein & Shakarchi, 2005, Exercise 24,Chapter 3).

Proposition 5.

Let $h:[0,1/2]\to[0,1]$ be a non-decreasing, and right-continuous function with $h(0)=0$ and $h(1/2)=1$ . Then one can decompose $h$ as

h=wh_{C}+(1-w)h_{J}

where $0\leq w\leq 1$ , $h_{C}$ , $h_{J}$ are non-decreasing functions mapping $[0,1/2]$ into $[0,1]$ with $h_{C}(0)=h_{J}(0)=0$ , $h_{C}(1)=h_{J}(1)=1$ , $h_{C}$ is continuous, and $h_{J}$ is a right-continuous step-function.

The goal of this appendix is to prove the following inequality:

Lemma 13.

Let $h:[0,1/2]\to[0,1]$ be an increasing and right-continuous function with $h(0)=0$ and $h(1/2)=1$ . Let $H$ be any continuous function with $H\geq h$ and let $r\in[0,1)$ . Then one can bound the Riemann–Stieltjes integral $\int 1/H(z)^{r}dh$ by

\int\frac{1}{H(z)^{r}}dh\leq\frac{2^{r}}{1-r}

Proof.

Let $h=wh_{C}+(1-w)h_{J}$ be the decomposition in Proposition 5. Thus

\int\frac{1}{H(z)^{r}}dh=w\int\frac{1}{H(z)^{r}}dh_{C}+(1-w)\int\frac{1}{H(z)^{r}}dh_{J}

(72)

We will bound each of the two integrals above separately. Notice that the portions of this decomposition bound $H$ below: $wh_{c}\leq H$ and $(1-w)h_{J}\leq H$ . First,

w\int\frac{1}{H(z)^{r}}dh_{C}=w\int_{\operatorname{supp}h_{C}}\frac{1}{H(z)^{r}}dh_{C}\leq w\int_{\operatorname{supp}h_{C}}\frac{1}{(wh_{C}(z))^{r}}dh_{C}=w^{1-r}\int_{\operatorname{supp}h_{C}}h_{C}^{-r}dh_{C}

Now by comparing the limiting sums that define the Riemman-Stieltjes integral $\int h_{C}^{-r}dh_{C}$ with the limit of the Riemann sums for the integral $\int y^{-r}dy$ , one can conclude that $\int h_{C}^{-r}dh_{C}$ can be evaluated as a Riemann integral in the variable $h_{C}$ . This argument relies on the continuity of the function $h_{C}$ . Thus $\int_{\operatorname{supp}h_{C}}h_{C}^{-r}dh_{C}\leq 1/(1-r)$ . Consequently:

w\int\frac{1}{H(z)^{r}}dh_{C}\leq\frac{w^{1-r}}{1-r}

(73)

Next, because $h_{J}$ is right-continuous and $h(z)=0$ , one can write

h_{J}(z)=\sum_{k=0}^{K-1}a_{k}{\mathbf{1}}_{[z_{k},z_{k+1})}(z)+a_{K}{\mathbf{1}}_{z_{K}}

(74)

with $K\in\mathbb{N}\cup\{\infty\}$ . Furthermore, because $h(0)=0$ , $h(1/2)=1$ and $h$ is non-decreasing, one can conclude that $z_{0}=0$ , $z_{K}=1/2$ , $a_{0}=0$ , $a_{K}=1$ , and $a_{k}\leq a_{k+1}$ . Lastly, one can require that $a_{k}<a_{k+1}$ in this representation, since otherwise one could express $h_{J}$ as in Equation 74 but with a smaller value of $K$ . Thus the second integral in Equation 72 evaluates to

\displaystyle(1-w)\int\frac{1}{H(z)^{r}}dh_{J}=(1-w)\sum_{k=0}^{K-1}\frac{1}{H(z_{k+1})^{r}}(a_{k+1}-a_{k}).

Recalling that $H(z)\geq(1-w)h_{J}$ , this quantity is bounded above by

(1-w)\sum_{k=0}^{K-1}\frac{1}{H(z_{k+1})^{r}}(a_{k+1}-a_{k})\leq(1-w)\sum_{k=0}^{K-1}\frac{1}{(1-w)^{r}h_{J}(z_{k+1})^{r}}(a_{k+1}-a_{k})=(1-w)^{1-r}\sum_{k=0}^{K-1}\frac{1}{a_{k+1}^{r}}(a_{k+1}-a_{k})

(75)

Because the function $y\mapsto y^{-r}$ is decreasing in $y$ , one can bound $\sum_{k=0}^{K-1}a_{k+1}^{-r}(a_{k+1}-a_{k})\leq\int_{a_{k}}^{a_{k+1}}y^{-r}dy$ and consequently Equation 75 is bounded above by

(1-w)^{1-r}\sum_{k=0}^{K-1}\int_{a_{k}}^{a_{k+1}}y^{-r}dy=(1-w)^{1-r}\int_{0}^{1}y^{-r}dy=\frac{(1-w)^{r}}{1-r}

Combining this bound with Equation 73 shows that Equation 72 is bounded above by

\frac{1}{1-r}(w^{1-r}+(1-w)^{1-r}).

Maximizing this quantity over $w\in[0,1]$ proves the result. ∎

Appendix I Optimizing the Bound of Lemma 3 over $r$

Proof of Theorem 13.

Let

f(r)=\frac{1}{1-r}a^{r}

Then

f^{\prime}(r)=\frac{1}{(1-r)^{2}}a^{r}+\frac{1}{1-r}\ln aa^{r}

solving $f^{\prime}(r^{*})=0$ produces $r^{*}=1+\frac{1}{\ln a}$ , and

f\left(1+\frac{1}{\ln a}\right)=-\ln aa^{1+\ln a}=-ea\ln a

One can verify that this point is a minimum via the second derivative test:

f^{\prime}(r)=\left(\frac{1}{1-r}+\ln a\right)f(r)

and thus

f^{\prime\prime}(r)=\left(\frac{1}{1-r}+\ln a\right)f^{\prime}(r)+\frac{1}{(1-r)^{2}}f(r).

Consequently, $f^{\prime\prime}(r^{*})=\ln(a)^{2}f(1+\frac{1}{\ln a})>0$ .

However, the point $r^{*}$ is in the interval $[0,1]$ only when $a\in[0,e^{-1}]$ . When $a>e^{-1}$ , $f$ is minimized over $[0,1]$ at $r=0$ . Because $r^{*}$ is a minimizer when $a\in[0,e^{-1}]$ , one can bound $f(0)\geq f(r^{*})$ over this set and thus

f(r)\leq\min\left(1,-ea\ln a\right)

Next, let $\Lambda$ be defined as in Lemma 3. Then the concavity of $\Lambda$ and the fact that $\Lambda(0)=0$ implies that $6\Lambda(z/6)\geq 2\Lambda(z/2)$ .

∎

Appendix J Further details from Examples 3 and 2

In Sections J.1 and J.2, we use an operation analogous to $S_{\epsilon}$ . Let $I_{\epsilon}$ be an operation on functions that computes the infimum over an $\epsilon$ -ball. Formally, we define:

I_{\epsilon}(g)({\mathbf{x}})=\inf_{\|{\mathbf{x}}^{\prime}-{\mathbf{x}}\|\leq\epsilon}g({\mathbf{x}}^{\prime}).

(76)

Nest, we define a mapping $\alpha_{\phi}$ from $\eta\in[0,1]$ to minimizers of $C_{\phi}(\eta,\cdot)$ by

\alpha_{\phi}(\eta)=\inf\{\alpha:\alpha\text{ is a minimizer of }C_{\phi}(\eta,\cdot)\}.

(77)

Lemma 25 of (Frank & Niles-Weed, 2024b) shows that the function $\alpha_{\phi}$ defined in Equation 77 maps $\eta$ to the smallest minimizer of $C_{\phi}(\eta,\cdot)$ and is non-decreasing. This property will be instrumental in constructing minimizers for $R_{\phi}^{\epsilon}$ .

J.1 Calculating the optimal ${\mathbb{P}}_{0}^{}$ , ${\mathbb{P}}_{1}^{}$ for Example 2

First, notice that a minimizer of $R_{\phi}$ is given by $f(x)=\alpha_{\phi}(\eta(x))$ with $\eta(x)$ as defined in Equation 19.

Below, we construct a minimizer $f^{*}$ for $R_{\phi}^{\epsilon}$ . We’ll verify that this function is a minimizer by showing that $R_{\phi}^{\epsilon}(f^{*})=R_{\phi}(f)$ . As the minimal possible adversarial risk is bounded below by $R_{\phi,*}$ , one can conclude that $f^{*}$ minimizes $R_{\phi}^{\epsilon}$ . Consequently, $\bar{R}_{\phi}({\mathbb{P}}_{0},{\mathbb{P}}_{1})=R_{\phi}^{\epsilon}(f)$ and thus the strong duality result in Theorem 9 would imply that ${\mathbb{P}}_{0}$ , ${\mathbb{P}}_{1}$ must maximize the dual problem.

Define a function $\tilde{\eta}:[-\delta-\epsilon-1,1+\delta+\epsilon]\to[0,1]$ by

\tilde{\eta}(x)=\begin{cases}\frac{1}{4}&\text{if }x\in[-1-\delta-\epsilon,0)\\ \frac{1}{2}&\text{if }x=0\\ \frac{3}{4}&\text{if }x\in(0,1+\delta+\epsilon]\end{cases}

and a function $f^{*}$ by $f^{*}(x)=\alpha_{\phi}(\tilde{\eta}(x))$ . Both $\tilde{\eta}$ and $\alpha_{\phi}$ are non-decreasing, and so $f^{*}$ must be non-decreasing as well. Consequently, $S_{\epsilon}(\phi(f^{*}))(x)=\phi(I_{\epsilon}(f^{*})(x))=\phi(f^{*}(x-\epsilon))$ and similarly, $S_{\epsilon}(\phi(-f^{*}))(x)=\phi(-S_{\epsilon}(f^{*})(x))=\phi(-f^{*}(x+\epsilon))$ . (Recall the $I_{\epsilon}$ operation was defined in Equation 76.)

Therefore,

	$\displaystyle R_{\phi}^{\epsilon}(f^{*})$	$\displaystyle=\int S_{\epsilon}(\phi(f^{}))(x)p_{1}(x)dx+\int S_{\epsilon}(\phi(-f^{}))(x)p_{0}(x)dx=\int\phi(f^{}(x-\epsilon))p_{1}(x)dx+\int\phi(-f^{}(x+\epsilon))p_{0}(x)dx$
		$\displaystyle=\int\phi(f^{}(x))p_{1}(x+\epsilon)dx+\int\phi(-f^{}(x))p_{0}(x-\epsilon)dx$

However, because the point 0 has Lebesgue measure zero if it is contained in $\operatorname{supp}p_{1}(\cdot+\epsilon)$ :

\int\phi(f^{*}(x))p_{1}(x+\epsilon)dx=\int_{-1-\delta-\epsilon}^{-\delta-\epsilon}\frac{1}{8}\phi\Big{(}\alpha_{\phi}\Big{(}\frac{1}{4}\Big{)}\Big{)}dx+\int_{\delta-\epsilon}^{1+\delta-\epsilon}\frac{1}{8}\phi\Big{(}\alpha_{\phi}\Big{(}\frac{3}{4}\Big{)}\Big{)}dx=\int\phi(f(x))p_{1}(x)dx

Analogously, one can show that

\int\phi(-f^{*}(x))p_{0}(x-\epsilon)dx=\int\phi(-f(x))p_{0}(x)dx

and consequently $R_{\phi}^{\epsilon}(f^{*})=R_{\phi}(f)$ .

J.2 Calculating the optimal ${\mathbb{P}}_{0}^{}$ and ${\mathbb{P}}_{1}^{}$ for Example 3

We will show that the densities in Equation 20 are dual optimal by finding a function $f^{*}$ for which $R_{\phi}^{\epsilon}(f^{*})=\bar{R}_{\phi}({\mathbb{P}}_{0}^{*},{\mathbb{P}}_{1}^{*})$ . Theorem 9 will then imply that ${\mathbb{P}}_{0}^{*}$ , ${\mathbb{P}}_{1}^{*}$ must maximize the dual.

and let $\eta^{*}(x)$ is defined by

\eta^{*}(x)=\frac{p_{1}^{*}(x)}{p_{1}^{*}(x)+p_{0}^{*}(x)},

with $p_{0}^{*}(x)$ and $p_{1}^{*}(x)$ as in Equation 20. For a given loss $\phi$ we will prove that the optimal function $f^{*}$ is given by

f^{*}(x)=\alpha_{\phi}(\eta^{*}(x)).

The function $\eta^{*}$ computes to

\eta^{*}(x)=\frac{1}{1+e^{\frac{\mu_{1}-\mu_{0}-2\epsilon}{\sigma^{2}}(\frac{\mu_{1}+\mu_{0}}{2}-x)}}

If $\mu_{1}-\mu_{0}>2\epsilon$ , the function $\eta^{*}(x)$ is increasing in $x$ and consequently $f^{*}$ is non-decreasing. Therefore, $S_{\epsilon}(\phi(f^{*}))(x)=\phi(I_{\epsilon}(f^{*})(x))=\phi(f^{*}(x-\epsilon))$ (recall $I_{\epsilon}$ was defined in Equation 76). Similarly, one can argue that $S_{\epsilon}(\phi(-f^{*}))(x)=\phi(-f^{*}(x+\epsilon))$ . Therefore,

	$\displaystyle R_{\phi}^{\epsilon}(f^{*})$	$\displaystyle=\int S_{\epsilon}(\phi(f^{}))(x)p_{1}(x)dx+\int S_{\epsilon}(\phi(-f^{}))(x)p_{0}(x)dx=\int\phi(f^{}(x-\epsilon))p_{1}(x)dx+\int\phi(-f^{}(x+\epsilon))p_{0}(x)dx$
		$\displaystyle=\int\phi(f^{}(x))p_{1}(x+\epsilon)dx+\int\phi(-f^{}(x))p_{0}(x-\epsilon)dx$

Next, notice that $p_{1}(x+\epsilon)=p_{1}^{*}(x)$ and $p_{0}(x-\epsilon)=p_{0}^{*}(x)$ . Define ${\mathbb{P}}^{*}={\mathbb{P}}_{0}^{*}+{\mathbb{P}}_{1}^{*}$ . Then

\displaystyle R_{\phi}^{\epsilon}(f^{*})

\displaystyle=\int\eta^{*}\phi(\alpha_{\phi}(\eta^{*}))+(1-\eta^{*})\phi(-\alpha_{\phi}(\eta^{*}))d{\mathbb{P}}^{*}=\int C_{\phi}^{*}(\eta^{*})d{\mathbb{P}}^{*}=\bar{R}_{\phi}({\mathbb{P}}_{0}^{*},{\mathbb{P}}_{1}^{*})

Consequently, the strong duality result in Theorem 9 implies that ${\mathbb{P}}_{0}^{*}$ ${\mathbb{P}}_{1}^{*}$ must maximize the dual $\bar{R}_{\phi}$ .

J.3 Showing Equation 21

Lemma 14.

Consider an equal gaussian mixture with variance $\sigma$ and means $\mu_{0}<\mu_{1}$ , with pdfs given by

p_{0}(x)=\frac{1}{2}\cdot\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(x-\mu_{0})^{2}}{2\sigma^{2}}},\quad p_{1}(x)=\frac{1}{2}\cdot\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(x-\mu_{1})^{2}}{2\sigma^{2}}}

Let $\eta(x)=p_{1}(x)/(p_{0}(x)+p_{1}(x))$ . Then $|\eta(x)-1/2|\leq z$ is equivalent to $x\in[\frac{\mu_{0}+\mu_{1}}{2}-\Delta(z),\frac{\mu_{0}+\mu_{1}}{2}+\Delta(z)]$ , where $\Delta(z)$ is defined by

\Delta(z)=\frac{\sigma^{2}}{\mu_{1}-\mu_{0}}\ln\left(\frac{\frac{1}{2}+z}{\frac{1}{2}-z}\right).

(78)

Proof.

The function $\eta$ can be rewritten as $\eta(x)=1/(1+p_{0}/p_{1})$ while

\frac{p_{0}(x)}{p_{1}(x)}=\exp\left(-\frac{(x-\mu_{0})^{2}}{2\sigma^{2}}+\frac{(x-\mu_{1})^{2}}{2\sigma^{2}}\right)=\exp\left(\frac{\mu_{1}-\mu_{0}}{\sigma^{2}}\left(\frac{\mu_{1}+\mu_{0}}{2}-x\right)\right)

Consequently, $|\eta(x)-1/2|\leq z$ is equivalent to

\frac{1}{2}-z\leq\frac{1}{\exp\left(\frac{\mu_{1}-\mu_{0}}{\sigma^{2}}(\frac{\mu_{1}+\mu_{0}}{2}-x)\right)+1}\leq\frac{1}{2}+z

which is equivalent to

\frac{\mu_{1}+\mu_{0}}{2}-\frac{\sigma^{2}}{\mu_{1}-\mu_{0}}\ln\left(\frac{1}{\frac{1}{2}-z}-1\right)\leq x\leq\frac{\mu_{1}+\mu_{0}}{2}-\frac{\sigma^{2}}{\mu_{1}-\mu_{0}}\ln\left(\frac{1}{z+\frac{1}{2}}-1\right)

Finally, notice that

\Delta(z)=\frac{\sigma^{2}}{\mu_{1}-\mu_{0}}\ln\left(\frac{1}{\frac{1}{2}-z}-1\right)

(79)

while

-\Delta(z)=\frac{\sigma^{2}}{\mu_{1}-\mu_{0}}\ln\left(\frac{1}{z+\frac{1}{2}}-1\right)

∎

Lemma 15.

Let $p_{0},p_{1}$ , and $\eta$ be as in Lemma 14 and let $h(z)={\mathbb{P}}(|\eta-1/2|\leq z)$ . Then if $\mu_{1}-\mu_{0}\leq 4\sqrt{2}\sigma$ , then $h$ is concave.

Proof.

To start, we calculate the second derivative of $\Delta(z)$ and the first derivative of $p_{0}$ .

The first derivative of $\Delta$ is

\Delta^{\prime}(z)=\frac{\sigma^{2}}{\mu_{1}-\mu_{0}}\cdot\frac{1}{\frac{1}{4}-z^{2}}.

and the second derivative of $\Delta(z)$ is

\Delta^{\prime\prime}(z)=\frac{\sigma^{2}}{\mu_{1}-\mu_{0}}\cdot\frac{2z}{(\frac{1}{4}-z^{2})^{2}}

(80)

Next, one can calculate the derivative of $p_{0}$ as

p_{0}^{\prime}(x)=\frac{1}{2}\cdot\frac{1}{\sqrt{2\pi}\sigma}\cdot\frac{-(x-\mu_{0})}{\sigma^{2}}e^{-\frac{(x-\mu_{0})^{2}}{2\sigma^{2}}}=-\frac{(x-\mu_{0})}{\sigma^{2}}p_{0}(x)

(81)

and similarly

p_{1}^{\prime}(x)=-\frac{(x-\mu_{1})}{\sigma^{2}}p_{1}(x)

(82)

Let $p(x)=p_{0}+p_{1}$ . Lemma 14 implies that the function $h$ is given by $h(z)=\int_{\tfrac{1}{2}-\Delta(z)}^{\tfrac{1}{2}+\Delta(z)}p(z)dz$ . The first derivative of $h$ is then

h^{\prime}(z)=\left(p\Big{(}\frac{\mu_{1}+\mu_{0}}{2}+\Delta(z)\Big{)}+p\Big{(}\frac{\mu_{1}+\mu_{0}}{2}-\Delta(z)\Big{)}\right)\Delta^{\prime}(z).

Differentiating $h$ twice results in

$\displaystyle h^{\prime\prime}(z)$	$\displaystyle=\left(p\Big{(}\frac{\mu_{1}+\mu_{0}}{2}+\Delta(z)\Big{)}+p\Big{(}\frac{\mu_{1}+\mu_{0}}{2}-\Delta(z)\Big{)}\right)\Delta^{\prime\prime}(z)$
	$\displaystyle+\left(p^{\prime}\Big{(}\frac{\mu_{1}+\mu_{0}}{2}+\Delta(z)\Big{)}-p^{\prime}\Big{(}\frac{\mu_{1}+\mu_{0}}{2}-\Delta(z)\Big{)}\right)(\Delta^{\prime}(z))^{2}$
	$\displaystyle=\left(p\Big{(}\frac{\mu_{1}+\mu_{0}}{2}+\Delta(z)\Big{)}+p\Big{(}\frac{\mu_{1}+\mu_{0}}{2}-\Delta(z)\Big{)}\right)\left(\Delta^{\prime\prime}(z)-\frac{\Delta(z)\Delta^{\prime}(z)^{2}}{\sigma^{2}}\right)$	(83)
	$\displaystyle+\bigg{(}p_{1}\Big{(}\frac{\mu_{1}+\mu_{0}}{2}+\Delta(z)\Big{)}+p_{1}\Big{(}\frac{\mu_{1}+\mu_{0}}{2}-\Delta(z)\Big{)}\bigg{)}\frac{\mu_{1}-\mu_{0}}{2\sigma^{2}}(\Delta^{\prime}(z))^{2}$	(84)
	$\displaystyle-\bigg{(}p_{0}\Big{(}\frac{\mu_{1}+\mu_{0}}{2}+\Delta(z)\Big{)}+p_{0}\Big{(}\frac{\mu_{1}+\mu_{0}}{2}-\Delta(z)\Big{)}\bigg{)}\frac{\mu_{1}-\mu_{0}}{2\sigma^{2}}(\Delta^{\prime}(z))^{2}.$	(85)

where the final equality is a consequence of Equations 81 and 82. Next, we’ll argue that the sum of the terms in Equations 84 and 85 is zero:

	$\displaystyle\bigg{(}p_{1}\Big{(}\frac{\mu_{1}+\mu_{0}}{2}+\Delta(z)\Big{)}+p_{1}\Big{(}\frac{\mu_{1}+\mu_{0}}{2}-\Delta(z)\Big{)}\bigg{)}-\bigg{(}p_{0}\Big{(}\frac{\mu_{1}+\mu_{0}}{2}+\Delta(z)\Big{)}+p_{0}\Big{(}\frac{\mu_{1}+\mu_{0}}{2}-\Delta(z)\Big{)}\bigg{)}$
	$\displaystyle=\frac{1}{2\sqrt{2\pi}\sigma}\left(\left(e^{-\frac{\left(\frac{\mu_{0}-\mu_{1}}{2}+\Delta(z)\right)^{2}}{2\sigma^{2}}}+e^{-\frac{\left(\frac{\mu_{0}-\mu_{1}}{2}-\Delta(z)\right)^{2}}{2\sigma^{2}}}\right)-\left(e^{-\frac{\left(\frac{\mu_{1}-\mu_{0}}{2}+\Delta(z)\right)^{2}}{2\sigma^{2}}}+e^{-\frac{\left(\frac{\mu_{1}-\mu_{0}}{2}-\Delta(z)\right)^{2}}{2\sigma^{2}}}\right)\right)$
	$\displaystyle=0$

Next, we’ll show that under the assumption $\mu_{1}-\mu_{0}\leq\sqrt{2}\sigma$ , the term Equation 83 is always negative. Define $k=sigma^{2}/(\mu_{1}-\mu_{0})$ . Then

\Delta^{\prime\prime}(z)-\Delta(z)\frac{\Delta^{\prime}(z)^{2}}{\sigma^{2}}=\frac{2k}{(\frac{1}{4}-z^{2})^{2}}\left(z-\frac{k^{2}}{2\sigma^{2}}\ln\left(\frac{1}{\frac{1}{2}-z}-1\right)\right)

(86)

The fact that $\Delta^{\prime\prime}(z)>0$ for all $z$ implies that $\ln(1/(1/2-z)-1)$ is convex, and this function has derivative $4$ at zero. Consequently, $\ln(1/(1/2-z)-1)\geq 4z$ and Equation 86 implies

\Delta^{\prime\prime}(z)\leq\frac{2k}{(\frac{1}{4}-z^{2})^{2}}(z-\frac{k^{2}}{2\sigma^{2}}\cdot 4z)=\frac{2kz}{(\frac{1}{4}-z^{2})^{2}}\left(1-\frac{2k^{2}}{\sigma^{2}}\right)

The condition $\mu_{1}-\mu_{0}\leq\sqrt{2}\sigma$ is equivalent to $1-2k^{2}/\sigma^{2}<0$ .

∎

This lemma implies that $h(z)\leq h^{\prime}(0)z$ . Noting also that $h(z)\leq 1$ for all $z$ produces the bound

h(z)\leq\min\left(\frac{16\sigma^{2}}{\mu_{1}-\mu_{0}}z,1\right)

applying this bound to the gaussians with densities $p_{0}^{*}$ and $p_{1}^{*}$ results in Equation 21.

Adversarial Surrogate Risk Bounds for Binary Classification

Abstract

1 Introduction

2 Background and Preliminaries

2.1 Surrogate Risks

Assumption 1.

Definition 1.

Theorem 1.

Theorem 2.

Theorem 3.

Theorem 4.

Proposition 1.

2.2 Adversarial Risks

Definition 2.

Theorem 5.

Definition 3.

Theorem 6.

2.3 Minimax Theorems

Lemma 1.

Theorem 7.

Theorem 8.

Theorem 9.

Theorem 10.

3 Main Results

Theorem 11.

Theorem 12.

Theorem 13.

Examples

Example 1 (When R∗ϵ=0R_{*}^{\epsilon}=0).

Example 2 (Massart’s noise condition).

Example 3 (Gaussian example).

4 Linear Surrogate Bounds— Proof of Theorems 12 and 11

Proof of Theorem 12.

Proof of Theorem 11.

5 Proof of Theorem 13

Lemma 2.

Proposition 2.

Lemma 3.

Proof.

6 Related Works

7 Conclusion

Acknowledgments

References

Contents of Appendix

Appendix A Proof of Theorem 3

Lemma 4.

Proof.

Proof of Theorem 3.

Appendix B (Non-Adversarial) Surrogate Bounds

B.1 The Realizable Case— Proof of Equation 10

Proof of Equation 10.

B.2 Linear Surrogate Risk Bounds—Proof of Proposition 1

Lemma 5.

Proof.

Lemma 6.

Proof.

Proof of Proposition 1.

Appendix C Proof of Lemma 1

Proof of Lemma 1.

Appendix D Proof of Theorem 10 Item 1)

Assumption 2.

Theorem 14 (Complementary Slackness).

Lemma 7.

Proof of Item 1) of Theorem 10 .

D.1 Proof of Lemma 7

Lemma 8.

Proposition 3.

Proof of Proposition 3.

Corollary 1.

Proof.

Proof of Lemma 7.

D.2 Proof of Lemma 8

Proof of Lemma 8.

Appendix E Proof of Lemma 2

Lemma 9.

Proof.

Lemma 10.

Proof.

Proof of Lemma 2.

Appendix F Defining the Function β\beta

Adversarial Surrogate Risk Bounds
for Binary Classification

Example 1 (When $R_{*}^{\epsilon}=0$ ).

Appendix F Defining the Function $\beta$

Appendix I Optimizing the Bound of Lemma 3 over $r$

J.1 Calculating the optimal ${\mathbb{P}}_{0}^{}$ , ${\mathbb{P}}_{1}^{}$ for Example 2

J.2 Calculating the optimal ${\mathbb{P}}_{0}^{}$ and ${\mathbb{P}}_{1}^{}$ for Example 3