Adapting to Function Difficulty and
Growth Conditions in Private Optimization

Hilal Asi Daniel Levy^∗ John C. Duchi
{asi,danilevy,jduchi}@stanford.edu Equal contribution. Author order determined by coin toss.

Abstract

We develop algorithms for private stochastic convex optimization that adapt to the hardness of the specific function we wish to optimize. While previous work provide worst-case bounds for arbitrary convex functions, it is often the case that the function at hand belongs to a smaller class that enjoys faster rates. Concretely, we show that for functions exhibiting $\kappa$ -growth around the optimum, i.e., $f(x)\geq f(x^{\star})+\lambda\kappa^{-1}\|{x-x^{\star}}\|_{2}^{\kappa}$ for $\kappa>1$ , our algorithms improve upon the standard ${\sqrt{d}}/{n\varepsilon}$ privacy rate to the faster $({\sqrt{d}}/{n\varepsilon})^{\tfrac{\kappa}{\kappa-1}}$ . Crucially, they achieve these rates without knowledge of the growth constant $\kappa$ of the function. Our algorithms build upon the inverse sensitivity mechanism, which adapts to instance difficulty [2], and recent localization techniques in private optimization [25]. We complement our algorithms with matching lower bounds for these function classes and demonstrate that our adaptive algorithm is simultaneously (minimax) optimal over all $\kappa\geq 1+c$ whenever $c=\Theta(1)$ .

1 Introduction

Stochastic convex optimization (SCO) is a central problem in machine learning and statistics, where for a sample space $\mathbb{S}$ , parameter space $\mathcal{X}\subset\mathbb{R}^{d}$ , and a collection of convex losses $\{F(\cdot;s):s\in\mathbb{S}\}$ , one wishes to solve

\mathop{\rm minimize}_{x\in\mathcal{X}}f(x)\coloneqq\mathbb{E}_{S\sim P}\left[F(x;S)\right]=\int_{\mathbb{S}}F(x;s)\mathrm{d}P(s)

(1)

using an observed dataset $\mathcal{S}=S_{1}^{n}\stackrel{{\scriptstyle\rm iid}}{{\sim}}P$ . While as formulated, the problem is by now fairly well-understood [12, 38, 29, 10, 37], it is becoming clear that, because of considerations beyond pure statistical accuracy—memory or communication costs [45, 26, 13], fairness [23, 28], personalization or distributed learning [35]—problem (1) is simply insufficient to address modern learning problems. To that end, researchers have revisited SCO under the additional constraint that the solution preserves the privacy of the provided sample [22, 21, 1, 16, 19]. A waypoint is Bassily et al. [7], who provide a private method with optimal convergence rates for the related empirical risk minimization problem, with recent papers focus on SCO providing (worst-case) optimal rates in various settings: smooth convex functions [8, 25], non-smooth functions [9], non-Euclidean geometry [5, 4] and under more stringent privacy constraints [34].

Yet these works ground their analyses in worst-case scenarios and provide guarantees for the hardest instance of the class of problems they consider. Conversely, they argue that their algorithms are optimal in a minimax sense: for any algorithm, there exists a hard instance on which the error achieved by the algorithm is equal to the upper bound. While valuable, these results are pessimistic—the exhibited hard instances are typically pathological—and fail to reflect achievable performance.

In this work, we consider the problem of adaptivity when solving (1) under privacy constraints. Importantly, we wish to provide private algorithms that adapt to the hardness of the objective $f$ . A loss function $f$ may belong to multiple problem classes, each exhibiting different achievable rates, so a natural desideratum is to attain the error rate of the easiest sub-class. As a simple vignette, if one gets an arbitrary $1$ -Lipschitz convex loss function $f$ , the worst-case guarantee of any $\varepsilon$ -DP algorithm is $\Theta(1/\sqrt{n}+d/(n\varepsilon))$ . However, if one learns that $f$ exhibits some growth property—say $f$ is $1$ -strongly convex—the regret guarantee improves to the faster $\Theta(1/n+(d/(n\varepsilon))^{2})$ rate with the appropriate algorithm. It is thus important to provide algorithms that achieves the rates of the “easiest” class to which the function belongs [32, 46, 18].

To that end, consider the nested classes of functions $\mathcal{F}^{\kappa}$ for $\kappa\in[1,\infty]$ such that, if $f\in\mathcal{F}^{\kappa}$ then there exists $\lambda>0$ such that for all $x\in\mathcal{X}$ ,

f(x)-\inf_{x^{\prime}\in\mathcal{X}}f(x^{\prime})\geq\frac{\lambda}{\kappa}\|x-x^{\star}\|_{2}^{\kappa}.

For example, strong convexity implies growth with parameter $\kappa=2$ . This growth assumption closely relates to uniform convexity [32] and the Polyak-Kurdyka-Łojasiewicz inequality [11], and we make these connections precise in Section 2. Intuitively, smaller $\kappa$ makes the function much easier to optimize: the error around the optimal point grows quickly. Objectives with growth are widespread in machine learning applications: among others, the $\ell_{1}$ -regularized hinge loss exhibits sharp growth (i.e. $\kappa=1$ ) while $\ell_{1}$ - or $\ell_{\infty}$ -constrained $\kappa$ -norm regression —i.e. $s=(a,b)\in\mathbb{R}^{d}\times\mathbb{R}$ and $F(x;s)=\lvert b-\langle a,x\rangle\rvert^{\kappa}$ —has $\kappa$ -growth for any $\kappa$ integer greater than $2$ [43]. In this work, we provide private adaptive algorithms that adapt to the actual growth of the function at hand.

We begin our analysis by examining Asi and Duchi’s inverse sensitivity mechanism [2] on ERM as a motivation. While not a practical algorithm, it achieves instance-optimal rates for any one-dimensional function under mild assumptions, quantifying the best bound one could hope to achieve with an adaptive algorithm, and showing (in principle) that adaptive private algorithms can exist. We first show that for any function with $\kappa$ -growth, the inverse sensitivity mechanism achieves privacy cost $(d/(n\varepsilon))^{\kappa/(\kappa-1)}$ ; importantly, without knowledge of the function class $\mathcal{F}^{\kappa}$ , that $f$ belongs to. This constitutes grounding and motivation for our work in three ways: (i) it validates our choice of sub-classes $\mathcal{F}^{\kappa}$ as the privacy rate is effectively controlled by the value of $\kappa$ , (ii) it exhibits the rate we wish to achieve with efficient algorithms on $\mathcal{F}^{\kappa}$ and (iii) it showcases that for easier functions, privacy costs shrink significantly—to illustrate, for $\kappa=5/4$ the privacy rate becomes $(d/(n\varepsilon))^{5}$ .

We continue our treatment of problem (1) under growth in Section 4 and develop practical algorithms that achieve the rates of the inverse sensitivity mechanism. Moreover, for approximate $(\varepsilon,\delta)$ -differential privacy, our algorithms improve the rates, achieving roughly $(\sqrt{d}/(n\varepsilon))^{\kappa/(\kappa-1)}$ . Our algorithms hinge on a reduction to SCO: we show that by solving a sequence of increasingly constrained SCO problems, one achieves the right rate whenever the function exhibits growth at the optimum. Importantly, our algorithm only requires a lower bound $\underline{\kappa}\leq\kappa$ (where $\kappa$ is the actual growth of $f$ ).

We provide optimality guarantees for our algorithms in Section 5 and show that both the inverse sensitivity and the efficient algorithms of Section 4 are simultaneously minimax optimal over all classes $\mathcal{F}^{\kappa}$ whenever $\kappa=1+\Theta(1)$ and $d=1$ for $\varepsilon$ -DP algorithms. Finally, we prove that in arbitrary dimension, for both pure- and approximate-DP constraints, our algorithms are also simultaneously optimal for all classes $\mathcal{F}^{\kappa}$ with $\kappa\geq 2$ .

On the way, we provide results that may be of independent interest to the community. First, we develop optimal algorithms for SCO under pure differential privacy constraints, which, to the best of our knowledge, do not exist in the literature. Secondly, our algorithms and analysis provide high-probability bounds on the loss, whereas existing results only provide (weaker) bounds on the expected loss. Finally, we complete the results of Ramdas and Singh [40] on (non-private) optimization lower bounds for functions with $\kappa$ -growth by providing information-theoretic lower bounds (in contrast to oracle-based lower bounds that rely on observing only gradient information) and capturing the optimal dependence on all problem parameters (namely $d,L$ and $\lambda$ ).

1.1 Related work

Convex optimization is one of the best studied problems in private data analysis [16, 19, 41, 7]. The first papers in this line of work mainly study minimizing the empirical loss, and readily establish that the (minimax) optimal privacy rates are ${d}/{n\varepsilon}$ for pure $\varepsilon$ -DP and ${\sqrt{d\log(1/\delta)}}/{n\varepsilon}$ for $(\varepsilon,\delta)$ -DP [16, 7]. More recently, several works instead consider the harder problem of privately minimizing the population loss [8, 25]. These papers introduce new algorithmic techniques to obtain the worst-case optimal rates of $1/\sqrt{n}+{\sqrt{d\log(1/\delta)}}/{n\varepsilon}$ for $(\varepsilon,\delta)$ -DP. They also show how to improve this rate to the faster $1/n+{{d\log(1/\delta)}}/{(n\varepsilon)^{2}}$ in the case of $1$ -strongly convex functions. Our work subsumes both of these results as they correspond to $\kappa=\infty$ and $\kappa=2$ respectively. To the best of our knowledge, there has been no work in private optimization that investigates the rates under general $\kappa$ -growth assumptions or adaptivity to such conditions.

In contrast, the optimization community has extensively studied growth assumptions [40, 32, 15] and show that on these problems, carefully crafted algorithms improves upon the standard $1/\sqrt{n}$ for convex functions to the faster $(1/\sqrt{n})^{\kappa/(\kappa-1)}$ . [32] derives worst-case optimal (in the first-order oracle model) gradient algorithms in the uniformly convex case (i.e. $\kappa\geq 2$ ) and provides technique to adapt to the growth $\kappa$ , while [40], drawing connections between growth conditions and active learning, provides upper and lower bounds in the first-order stochastic oracle model. We complete the results of the latter and provide information-theoretic lower bounds that have optimal dependence on $d,\lambda$ and $n$ —their lower bound only holding for $\lambda$ inversely proportional to $d^{1/2-1/\kappa}$ , when $\kappa\geq 2$ . Closest to our work is [15] who studies instance-optimality via local minimax complexity [14]. For one-dimensional functions, they develop a bisection-based instance-optimal algorithm and show that on individual functions of the form $t\mapsto\kappa^{-1}\lvert t\rvert^{\kappa}$ , the local minimax rate is $(1/\sqrt{n})^{\kappa/(\kappa-1)}$ .

2 Preliminaries

We first provide notation that we use throughout this paper, define useful assumptions and present key definitions in convex analysis and differential privacy.

Notation.

$n$ typically denotes the sample size and $d$ the dimension. Throughout this work, $x$ refers to the optimization variable, $\mathcal{X}\subset\mathbb{R}^{d}$ to the constraint set and $s$ to elements ( $S$ when random) of the sample space $\mathbb{S}$ . We usually denote by $F:\mathcal{X}\times\mathbb{S}\to\mathbb{R}$ the (convex) loss function and for a dataset $\mathcal{S}=(s_{1},\ldots,s_{n})\subset\mathbb{S}$ , we define the empirical and population losses

f_{\mathcal{S}}(x)\coloneqq\frac{1}{n}\sum_{i\leq n}F(x;s_{i})\mbox{~{}~{}and~{}~{}}f(x)\coloneqq\mathbb{E}_{S\sim P}\left[F(x;S)\right].

We omit the dependence on $P$ as it is often clear from context. We reserve $\varepsilon,\delta\geq 0$ for the privacy parameters of Definition 2.1. We always take gradients with respect to the optimization variable $x$ . In the case that $F(\cdot;s)$ is not differentiable at $x$ , we override notation and define $\nabla F(x;s)=\mathop{\rm argmin}_{g\in\partial F(x;s)}\|g\|_{2}$ , where $\partial F(x;s)$ is the subdifferential of $F(\cdot;s)$ at $x$ . We use $\mathsf{A}$ for (potentially random) mechanism and $S_{1}^{n}$ as a shorthand for $\left(S_{1},\ldots,S_{n}\right)$ . For $p\geq 1$ , $\|\cdot\|_{p}$ is the standard $\ell_{p}$ -norm, $\mathbb{B}_{p}^{d}(R)$ is the corresponding $d$ -dimensional $p$ -ball of radius $R$ and $p^{\star}$ is the dual of $p$ , i.e. such that $1/p^{\star}+1/p=1$ . Finally, we define the Hamming distance between datasets $d_{\rm Ham}(\mathcal{S},\mathcal{S}^{\prime})\coloneqq\inf_{\sigma\in\mathfrak{S}_{n}}\mathbf{1}\{s_{i}\neq s^{\prime}_{\sigma(i)}\}$ , where $\mathfrak{S}_{n}$ is the set of permutations over sets of size $n$ .

Assumptions.

We first state standard assumptions for solving (1). We assume that $\mathcal{X}$ is a closed, convex domain such that $\mathsf{diam}_{2}(\mathcal{X})=\sup_{x,y\in\mathcal{X}}\|x-y\|_{2}\leq D<\infty$ . Furthermore, we assume that for any $s\in\mathbb{S}$ , $F(\cdot;s)$ is convex and $L$ -Lipschitz with respect to $\|\cdot\|_{2}$ . Central to our work, we define the following $\kappa$ -growth assumption.

Assumption 1 ( $\kappa$ -growth).

Let $x^{\star}=\mathop{\rm argmin}_{x\in\mathcal{X}}f(x)$ . For a loss $F$ and distribution $P$ , we say that $(F,P)$ has $(\lambda,\kappa)$ growth for $\kappa\in[1,\infty]$ and $\lambda>0$ , if the population function satisfies

\mbox{for all~{}}x\in\mathcal{X},\quad f(x)-f(x^{\star})\geq\frac{\lambda}{\kappa}\|x-x^{\star}\|_{2}^{\kappa}.

In the case where $\widehat{P}$ is the empirical distribution on a finite dataset $\mathcal{S}$ , we refer to $(\lambda,\kappa)$ -growth of $(F,\widehat{P})$ as $\kappa$ -growth of the empirical function $f_{\mathcal{S}}$ .

Uniform convexity and Kurdyka-Łojasiewicz inequality.

Assumption 1 is closely related to two fundamental notions in convex analysis: uniform convexity and the Kurdyka-Łojasiewicz inequality. Following [39], we say that $h:\mathcal{Z}\subset\mathbb{R}^{d}\to\mathbb{R}$ is $(\sigma,\kappa)$ -uniformly convex with $\sigma>0$ and $\kappa\geq 2$ if

\mbox{for all~{}~{}}x,y\in\mathcal{Z},\quad h(y)\geq h(x)+\langle\nabla h(x),y-x\rangle+\frac{\sigma}{\kappa}\|x-y\|_{2}^{\kappa}.

This immediately implies that (i) sums (and expectations) preserve uniform convexity (ii) if $f$ is uniformly convex with $\lambda$ and $\kappa$ , then it has $(\lambda,\kappa)$ -growth. This will be useful when constructing hard instances as it will suffice to consider $(\lambda,\kappa)$ -uniformly convex functions which are generally more convenient to manipulate. Finally, we point out that, in the general case that $\kappa\geq 1$ , the literature refers to Assumption 1 as the Kurdyka-Łojasiewicz inequality [11] with, in their notation, $\varphi(s)=(\kappa/\lambda)^{1/\kappa}s^{1/\kappa}$ . Theorem 5-(ii) in [11] says that, under mild conditions, Assumption 1 implies the following inequality between the error and the gradient norm for all $x\in\mathcal{X}$

f(x)-\inf_{x^{\prime}\in\mathcal{X}}f(x^{\prime})\leq\frac{e}{\lambda^{\tfrac{1}{\kappa-1}}}\left\|\nabla f(x)\right\|_{2}^{\tfrac{\kappa}{\kappa-1}},

(2)

This is a key result in our analysis of the inverse sensitivity mechanism of Section 3.

Differential privacy.

We begin by recalling the definition of $(\varepsilon,\delta)$ -differential privacy.

Definition 2.1 ([22, 21]).

A randomized algorithm $\mathsf{A}$ is $(\varepsilon,\delta)$ -differentially private ( $(\varepsilon,\delta)$ -DP) if, for all datasets $\mathcal{S},\mathcal{S}^{\prime}\in\mathbb{S}^{n}$ that differ in a single data element and for all events $\mathcal{O}$ in the output space of $\mathsf{A}$ , we have

\Pr\left(\mathsf{A}(\mathcal{S})\in\mathcal{O}\right)\leq e^{\epsilon}\Pr\left(\mathsf{A}(\mathcal{S}^{\prime})\in\mathcal{O}\right)+\delta.

We use the following standard results in differential privacy.

Lemma 2.1 (Composition [20, Thm. 3.16]).

If $\mathsf{A}_{1},\dots,\mathsf{A}_{k}$ are randomized algorithms that each is $(\varepsilon,\delta)$ -DP, then their composition $(\mathsf{A}_{1}(\mathcal{S}),\dots,\mathsf{A}_{k}(\mathcal{S}))$ is $(k\varepsilon,k\delta)$ -DP.

Next, we consider the Laplace mechanism. We will let $Z\sim\mathsf{Lap}_{d}(\sigma)$ denote a $d$ -dimensional vector $Z\in\mathbb{R}^{d}$ such that $Z_{i}\stackrel{{\scriptstyle\rm iid}}{{\sim}}\mathsf{Lap}(\sigma)$ for $1\leq i\leq d$ .

Lemma 2.2 (Laplace mechanism [20, Thm. 3.6]).

Let $h:\mathbb{S}^{n}\to\mathbb{R}^{d}$ have $\ell_{1}$ -sensitivity $\Delta$ , that is, $\sup_{\mathcal{S},\mathcal{S}^{\prime}\in\mathbb{S}^{n}:d_{\rm Ham}(\mathcal{S},\mathcal{S}^{\prime})\leq 1}\|h(\mathcal{S})-h(\mathcal{S}^{\prime})\|_{1}\leq\Delta$ . Then the Laplace mechanism $\mathsf{A}(\mathcal{S})=h(\mathcal{S})+\mathsf{Lap}_{d}(\sigma)$ with $\sigma=\Delta/\varepsilon$ is $\varepsilon$ -DP.

Finally, we need the Gaussian mechanism for $(\varepsilon,\delta)$ -DP.

Lemma 2.3 (Gaussian mechanism [20, Thm. A.1]).

Let $h:\mathbb{S}^{n}\to\mathbb{R}^{d}$ have $\ell_{2}$ -sensitivity $\Delta$ , that is, $\sup_{\mathcal{S},\mathcal{S}^{\prime}\in\mathbb{S}^{n}:d_{\rm Ham}(\mathcal{S},\mathcal{S}^{\prime})\leq 1}\|h(\mathcal{S})-h(\mathcal{S}^{\prime})\|_{2}\leq\Delta$ . Then the Gaussian mechanism $\mathsf{A}(\mathcal{S})=h(\mathcal{S})+\mathsf{N}(0,\sigma^{2}I_{d})$ with $\sigma=2\Delta\log(2/\delta)/\varepsilon$ is $(\varepsilon,\delta)$ -DP.

Inverse sensitivity mechanism.

Our goal is to design private optimization algorithms that adapt to the difficulty of the underlying function. As a reference point, we turn to the inverse sensitivity mechanism of [2] as it enjoys general instance-optimality guarantees. For a given function $h:\mathbb{S}^{n}\to\mathcal{T}\subset\mathbb{R}^{d}$ that we wish to estimate privately, define the inverse sensitivity at $x\in\mathcal{T}$

\mathsf{len}_{h}(\mathcal{S};x)=\inf_{\mathcal{S}^{\prime}}\{d_{\rm Ham}(\mathcal{S}^{\prime},\mathcal{S}):h(\mathcal{S}^{\prime})=x\},

(3)

that is, the inverse sensitivity of a target parameter $y\in\mathcal{T}$ at instance $\mathcal{S}$ is the minimal number of samples one needs to change to reach a new instance $\mathcal{S}^{\prime}$ such that $h(\mathcal{S}^{\prime})=y$ . Having this quantity, the inverse sensitivity mechanism samples an output from the following probability density

\pi_{\mathsf{A}_{\mathrm{inv}}(\mathcal{S})}(x)\propto e^{-\varepsilon\mathsf{len}_{h}(\mathcal{S};x)}.

(4)

The inverse sensitivity mechanism preserves $\varepsilon$ -DP and enjoys instance-optimality guarantees in general settings [2]. In contrast to (worst-case) minimax optimality guarantees which measure the performance of the algorithm on the hardest instance, these notions of instance-optimality provide stronger per-instance optimality guarantees.

3 Adaptive rates through inverse sensitivity for $\varepsilon$ -DP

To understand the achievable rates when privately optimizing functions with growth, we begin our theoretical investigation by examining the inverse sensitivity mechanism in our setting. We show that, for instances that exhibit $\kappa$ -growth of the empirical function, the inverse sensitivity mechanism privately solves ERM with excess loss roughly $({d}/{n\varepsilon})^{\frac{\kappa}{\kappa-1}}$ .

In our setting, we use a gradient-based approximation of the inverse sensitivity mechanism to simplify the analysis, while attaining similar rates. Following [3] with our function of interest $h(\mathcal{S})\coloneqq\mathop{\rm argmin}_{x\in\mathcal{X}}f_{\mathcal{S}}(x)$ , we can lower bound the inverse sensitivity $\mathsf{len}_{h}(\mathcal{S};x)\geq n\|\nabla f_{\mathcal{S}}(x)\|_{2}/2L$ under natural assumptions. We define a $\rho$ -smoothed version of this quantity which is more suitable to continuous domains

G^{\rho}_{\mathcal{S}}(x)=\inf_{y\in\mathcal{X}:\|y-x\|_{2}\leq\rho}\|\nabla f_{\mathcal{S}}(y)\|_{2},

and define the $\rho$ -smooth gradient-based inverse sensitivity mechanism

\pi_{\mathsf{A}_{\mathrm{gr{-}inv}}(\mathcal{S})}(x)\propto e^{-\varepsilon nG^{\rho}_{\mathcal{S}}(x)/2L}.

(5)

Note that while exactly sampling from the un-normalized density $\pi_{\mathsf{A}_{\mathrm{gr{-}inv}}(\mathcal{S})}$ is computationally intractable, analyzing its performance is an important step towards understanding the optimal rates for the family of functions with growth that we study in this work. The following theorem demonstrates the adaptivity of the inverse sensitivity mechanism to the growth of the underlying instance. We defer the proof to Appendix A.

Theorem 1.

Let $\mathcal{S}=(s_{1},\ldots,s_{n})\in\mathbb{S}^{n}$ , $F(x;s)$ be convex, $L$ -Lipschitz for all $s\in\mathbb{S}$ . Let $x^{\star}=\mathop{\rm argmin}_{x\in\mathcal{X}}f_{\mathcal{S}}(x)$ and assume $x^{\star}$ is in the interior of $\mathcal{X}$ . Assume that $f_{S}(x)$ has $\kappa$ -growth (Assumption 1) with $\kappa\geq\underline{\kappa}>1$ . For $\rho>0$ , the $\rho$ -smooth inverse sensitivity mechanism $\mathsf{A}_{\mathrm{gr{-}inv}}$ (5) is $\varepsilon$ -DP, and with probability at least $1-\beta$ the output $\hat{x}=\mathsf{A}_{\mathrm{gr{-}inv}}(\mathcal{S})$ has

f_{\mathcal{S}}(\hat{x})-\min_{x\in\mathcal{X}}f_{\mathcal{S}}(x)\leq\frac{1}{\lambda^{\frac{1}{\kappa-1}}}\left(\frac{2L(\log(1/\beta)+d\log(D/\rho))}{n\varepsilon}\right)^{\frac{\kappa}{\kappa-1}}+L\rho.

Moreover, setting $\rho=({L}/{\lambda})^{\frac{1}{\underline{\kappa}-1}}({d}/{n\varepsilon})^{\frac{\underline{\kappa}}{\underline{\kappa}-1}}$ , we have

\displaystyle f_{\mathcal{S}}(\hat{x})-\min_{x\in\mathcal{X}}f_{\mathcal{S}}(x)

\displaystyle\leq\frac{1}{\lambda^{\frac{1}{\kappa-1}}}\widetilde{O}\left(\frac{Ld}{n\varepsilon}\right)^{\frac{\kappa}{\kappa-1}}.

The rates of the inverse sensitivity in Theorem 1 provide two main insights regarding the landscape of the problem with growth conditions. First, these conditions allow to improve the worst-case rate $d/n\varepsilon$ to $({d}/{n\varepsilon})^{\frac{\kappa}{\kappa-1}}$ for pure $\varepsilon$ -DP and therefore suggest a better rate $({\sqrt{d\log(1/\delta)}}/{n\varepsilon})^{\frac{\kappa}{\kappa-1}}$ is possible for approximate $(\varepsilon,\delta)$ -DP. Moreover, the general instance-optimality guarantees of this mechanism [2] hint that these are the optimal rates for our class of functions. In the sections to come, we validate the correctness of these predictions by developing efficient algorithms that achieve these rates (for pure and approximate privacy), and prove matching lower bounds which demonstrate the optimality of these algorithms.

4 Efficient algorithms with optimal rates

While the previous section demonstrates that there exists algorithms that improve the rates for functions with growth, we pointed out that $\mathsf{A}_{\mathrm{gr{-}inv}}$ was computationally intractable in the general case. In this section, we develop efficient algorithms—e.g. that are implementable with gradient-based methods—that achieve the same convergence rates. Our algorithms build on the recent localization techniques that Feldman et al. [25] used to obtain optimal rates for DP-SCO with general convex functions. In Section 4.1, we use these techniques to develop private algorithms that achieve the optimal rates for (pure) DP-SCO with high probability, in contrast to existing results which bound the expected excess loss. These results are of independent interest.

In Section 4.2, we translate these results into convergence guarantees on privately optimizing convex functions with growth by solving a sequence of increasingly constrained SCO problems—the high-probability guarantees of Section 4.1 being crucial to our convergence analysis of these algorithms.

4.1 High-probability guarantees for convex DP-SCO

We first describe our algorithm (Algorithm 1) then analyze its performance under pure-DP (Proposition 1) and approximate-DP constraints (Proposition 2). Our analysis builds on novel tight generalization bounds for uniformly-stable algorithms with high probability [24]. We defer the proofs to Appendix B.

Algorithm 1 Localization-based Algorithm

0: Dataset

\mathcal{S}=(s_{1},\ldots,s_{n})\in\mathbb{S}^{n}

, constraint set

\mathcal{X}

, step size

\eta

, initial point

x_{0}

, privacy parameters

(\varepsilon,\delta)

;

1: Set

k=\lceil\log n\rceil

and

n_{0}=n/k

2: for

i=1

k

3: Set

\eta_{i}=2^{-4i}\eta

4: Solve the following ERM over

\mathcal{X}_{i}=\{x\in\mathcal{X}:\|x-x_{i-1}\|_{2}\leq{2L\eta_{i}n_{0}}\}

\displaystyle\quad\quad\quad\quad F_{i}(x)=\frac{1}{n_{0}}\sum_{j=1+(i-1)n_{0}}^{in_{0}}F(x;s_{j})+\frac{1}{\eta_{i}n_{0}}\|x-x_{i-1}\|_{2}^{2}

5: Let

\hat{x}_{i}

be the output of the optimization algorithm

6: if

\delta=0

then

7: Set

\zeta_{i}\sim\mathsf{Lap}_{d}(\sigma_{i})

where

\sigma_{i}=4L\eta_{i}\sqrt{d}/\varepsilon_{i}

8: else if

\delta>0

then

9: Set

\zeta_{i}\sim\mathsf{N}(0,\sigma_{i}^{2})

where

\sigma_{i}=4L\eta_{i}\sqrt{\log(1/\delta)}/\varepsilon

10: Set

x_{i}=\hat{x}_{i}+\zeta_{i}

11: return the final iterate

x_{k}

Proposition 1.

Let $\beta\leq 1/(n+d)$ , $\mathsf{diam}_{2}(\mathcal{X})\leq D$ and $F(x;s)$ be convex, $L$ -Lipschitz for all $s\in\mathbb{S}$ . Setting

\eta=\frac{D}{L}\min\left(\frac{1}{\sqrt{n\log(1/\beta)}},\frac{\varepsilon}{d\log(1/\beta)}\right)

then for $\delta=0$ , Algorithm 1 is $\varepsilon$ -DP and has with probability $1-\beta$

f(x)-f(x^{\star})\leq LD\cdot O\left(\frac{\sqrt{\log(1/\beta)}\log^{3/2}n}{\sqrt{n}}+\frac{d\log(1/\beta)\log n}{n\varepsilon}\right).

Similarly, by using a different choice for the parameters and noise distribution, we have the following guarantees for approximate $(\varepsilon,\delta)$ -DP.

Proposition 2.

Let $\beta\leq 1/(n+d)$ , $\mathsf{diam}_{2}(\mathcal{X})\leq D$ and $F(x;s)$ be convex, $L$ -Lipschitz for all $s\in\mathbb{S}$ . Setting

\eta=\frac{D}{L}\min\left(\frac{1}{\sqrt{n\log(1/\beta)}},\frac{\varepsilon}{\sqrt{d\log(1/\delta)}\log(1/\beta)}\right),

then for $\delta>0$ , Algorithm 1 is $(\varepsilon,\delta)$ -DP and has with probability $1-\beta$

f(x)-f(x^{\star})\leq LD\cdot O\left(\frac{\sqrt{\log(1/\beta)}\log^{3/2}n}{\sqrt{n}}+\frac{\sqrt{d\log(1/\delta)}\log(1/\beta)\log n}{n\varepsilon}\right).

4.2 Algorithms for DP-SCO with growth

Building on the algorithms of the previous section, we design algorithms that recover the rates of the inverse sensitivity mechanism for functions with growth, importantly without knowledge of the value of $\kappa$ . Inspired by epoch-based algorithms from the optimization literature [31, 29], our algorithm iteratively applies the private procedures from the previous section. Crucially, the growth assumption allows to reduce the diameter of the domain after each run, hence improving the overall excess loss by carefully choosing the hyper-parameters. We provide full details in Algorithm 2.

Algorithm 2 Epoch-based algorithms for

\kappa

-growth

0: Dataset

\mathcal{S}=(s_{1},\ldots,s_{n})\in\mathbb{S}^{n}

, convex set

\mathcal{X}

, initial point

x_{0}

, number of iterations

T

, privacy parameters

(\varepsilon,\delta)

;

1: Set

n_{0}=n/T

and

D_{0}=\mathsf{diam}_{2}(\mathcal{X})

2: if

\delta=0

then

3: Set

\eta_{0}=\frac{D_{0}}{2L}\min\left(\frac{1}{\sqrt{n_{0}\log(n_{0})\log(1/\beta)}},\frac{\varepsilon}{d\log(1/\beta)}\right)

4: else if

\delta>0

then

\eta_{0}=\frac{D_{0}}{2L}\min\left(\frac{1}{\sqrt{n_{0}\log(n_{0})\log(1/\beta)}},\frac{\varepsilon}{\sqrt{d\log(1/\delta)}\log(1/\beta)}\right)

6: for

i=0

T-1

7: Let

\mathcal{S}_{i}=(s_{1+(i-1)n_{0}},\dots,s_{in_{0}})

8: Set

D_{i}=2^{-i}D_{0}

and

\eta_{i}=2^{-i}\eta_{0}

9: Set

\mathcal{X}_{i}=\{x\in\mathcal{X}:\|x-x_{i}\|_{2}\leq D_{i}\}

10: Run Algorithm 1 on dataset

\mathcal{S}_{i}

with starting point

x_{i}

, privacy parameter

(\varepsilon,\delta)

, domain

\mathcal{X}_{i}

(with diameter

D_{i}

), step size

\eta_{i}

11: Let

x_{i+1}

be the output of the private procedure

12: return

x_{T}

The following theorem summarizes our main upper bound for DP-SCO with growth in the pure privacy model, recovering the rates of the inverse sensitivity mechanism in Section 3. We defer the proof to Section B.3.

Theorem 2.

Let $\beta\leq 1/(n+d)$ , $\mathsf{diam}_{2}(\mathcal{X})\leq D$ and $F(x;s)$ be convex, $L$ -Lipschitz for all $s\in\mathbb{S}$ . Assume that $f$ has $\kappa$ -growth (Assumption 1) with $\kappa\geq\underline{\kappa}>1$ . Setting $T=\left\lceil\frac{2\log n}{\underline{\kappa}-1}\right\rceil$ , Algorithm 2 is $\varepsilon$ -DP and has with probability $1-\beta$

f(x_{T})-\min_{x\in\mathcal{X}}f(x)\leq\frac{1}{\lambda^{\frac{1}{\kappa-1}}}\cdot\widetilde{O}\left(\frac{L\sqrt{\log(1/\beta)}}{\sqrt{n}}+\frac{Ld\log(1/\beta)}{n\varepsilon(\underline{\kappa}-1)}\right)^{\frac{\kappa}{\kappa-1}},

where $\widetilde{O}$ hides logarithmic factors depending on $n$ and $d$ .

Sketch of the proof.

The main challenge of the proof is showing that the iterate achieves good risk without knowledge of $\kappa$ . Let us denote by $D\cdot\rho$ the error guarantee of Proposition 1 (or Proposition 2 for approximate-DP). At each stage $i$ , as long as $x^{\star}=\mathop{\rm argmin}_{x\in\mathcal{X}}f(x)$ belongs to $\mathcal{X}_{i}$ , the excess loss is of order $D_{i}\cdot\rho$ and thus decreases exponentially fast with $i$ . The challenge is that, without knowledge of $\kappa$ , we do not know the index $i_{0}$ (roughly $\tfrac{\log_{2}n}{\kappa-1}$ ) after which $x^{\star}\notin D_{j}$ for $j\geq i_{0}$ and the regret guarantees become meaningless with respect to the original problem. However, in the stages after $i_{0}$ , as the constraint set becomes very small, we upper bound the variations in function values $f(x_{j+1})-f(x_{j})$ and show that the sub-optimality cannot increase (overall) by more than $O(D_{i_{0}}\cdot\rho)$ , thus achieving the optimal rate of stage $i_{0}$ .

∎

Moreover, we can improve the dependence on the dimension for approximate $(\varepsilon,\delta)$ -DP, resulting in the following bounds.

Theorem 3.

Let $\beta\leq 1/(n+d)$ , $\mathsf{diam}_{2}(\mathcal{X})\leq D$ and $F(x;s)$ be convex, $L$ -Lipschitz for all $s\in\mathbb{S}$ . Assume that $f$ has $\kappa$ -growth (Assumption 1) with $\kappa\geq\underline{\kappa}>1$ . Setting $T=\left\lceil\frac{2\log n}{\underline{\kappa}-1}\right\rceil$ and $\delta>0$ , Algorithm 2 is $(\varepsilon,\delta)$ -DP and has with probability $1-\beta$

f(x_{T})-\min_{x\in\mathcal{X}}f(x)\leq\frac{1}{\lambda^{\frac{1}{\kappa-1}}}\cdot\widetilde{O}\left(\frac{L\sqrt{\log(1/\beta)}}{\sqrt{n}}+\frac{L\sqrt{d\log(1/\delta)}\log(1/\beta)}{n\varepsilon(\underline{\kappa}-1)}\right)^{\frac{\kappa}{\kappa-1}},

where $\widetilde{O}$ hides logarithmic factors depending on $n$ and $d$ .

5 Lower bounds

In this section, we develop (minimax) lower bounds for the problem of SCO with $\kappa$ -growth under privacy constraints. Note that taking $\varepsilon\to\infty$ provides lower bound for the unconstrained minimax risk. For a sample space $\mathbb{S}$ and collection of distributions $\mathcal{P}$ over $\mathbb{S}$ , we define the function class $\mathcal{F}^{\kappa}(\mathcal{P})$ as the set of convex functions from $\mathbb{R}^{d}\to\mathbb{R}$ that are $L$ -Lipschitz and has $\kappa$ -growth (Assumption 1). We define the constrained minimax risk [6]

\mathfrak{M}_{n}(\mathcal{X},\mathcal{P},\mathcal{F}^{\kappa},\varepsilon,\delta)\coloneqq\inf_{\widehat{x}_{n}\in\mathcal{A}^{\varepsilon,\delta}}\sup_{(F,P)\in\mathcal{F}^{\kappa}\times\mathcal{P}}\mathbb{E}\left[f(\widehat{x}_{n}(S_{1}^{n}))-\inf_{x^{\prime}\in\mathcal{X}}f(x^{\prime})\right],

(6)

where $\mathcal{A}^{\epsilon,\delta}$ is the collection of $(\varepsilon,\delta)$ -DP mechanisms from $\mathbb{S}^{n}$ to $\mathcal{X}$ . When clear from context, we omit the dependency on $\mathcal{P}$ of the function class and simply write $\mathcal{F}^{\kappa}$ . We also forego the dependence on $\delta$ when referring to pure-DP constraints, i.e. $\mathfrak{M}_{n}(\mathcal{X},\mathcal{P},\mathcal{F}^{\kappa},\varepsilon,\delta=0)\eqqcolon\mathfrak{M}_{n}(\mathcal{X},\mathcal{P},\mathcal{F}^{\kappa},\varepsilon)$ . We now proceed to prove tight lower bounds for $\varepsilon$ -DP in Section 5.1 and $(\varepsilon,\delta)$ -DP in Section 5.2.

5.1 Lower bounds for pure $\varepsilon$ -DP

Although in Section 4 we show that the same algorithm achieves the optimal upper bounds for all values of $\kappa>1$ , the landscape of the problem is more subtle for the lower bounds and we need to delineate two different cases to obtain tight lower bounds. We begin with $\kappa\geq 2$ , which corresponds to uniform convexity and enjoys properties that make the problem easier (e.g., closure under summation or addition of linear terms). The second case, $1<\kappa<2$ , corresponds to sharper growth and requires a different hard instance to satisfy the growth condition.

$\kappa$ -growth with $\kappa\geq 2$ .

We begin by developing lower bounds under pure DP for $\kappa\geq 2$

Theorem 4 (Lower bound for $\varepsilon$ -DP, $\kappa\geq 2$ ).

Let $d\geq 1$ , $\mathcal{X}=\mathbb{B}_{2}^{d}(R)$ , $\mathbb{S}=\{\pm e_{j}\}_{j\leq d}$ , $\kappa\geq 2$ and $n\in\mathbb{N}$ . Let $\mathcal{P}$ be the set of distributions on $\mathbb{S}$ . Assume that

2^{\kappa-1}\leq\frac{L}{\lambda}\frac{1}{R^{\kappa-1}}\leq 2^{\kappa-1}\sqrt{96n}\mbox{~{}~{}and~{}~{}}n\varepsilon\geq\frac{1}{\sqrt{3}}

The following lower bound holds

\mathfrak{M}_{n}(\mathcal{X},\mathcal{P},\mathcal{F}^{\kappa},\epsilon)\geq\frac{1}{\lambda^{\tfrac{1}{\kappa-1}}}\tilde{\Omega}\left(\left(\frac{L}{\sqrt{n}}\right)^{\tfrac{\kappa}{(\kappa-1)}}+\left(\frac{Ld}{n\varepsilon}\right)^{\tfrac{\kappa}{\kappa-1}}\right).

(7)

First of all, note that $L\geq\lambda 2^{\kappa}R^{\kappa-1}$ is not an overly-restrictive assumption. Indeed, for an arbitrary $(\lambda,\kappa)$ -uniformly convex and $L$ -Lipschitz function, it always holds that $L\geq\tfrac{\lambda}{2}R^{\kappa-1}$ . This is thus equivalent to assuming $\kappa=\Theta(1)$ . Note that when $\kappa\gg 1$ , the standard $n^{-1/2}+d/(n\varepsilon)$ lower bound holds. We present the proof in Section C.1.1 and preview the main ideas here.

Sketch of the proof.

Our lower bounds hinges on the collections of functions $F(x;s)\coloneqq a\kappa^{-1}\|x\|_{2}^{\kappa}+b\langle x,s\rangle$ for $a,b\geq 0$ to be chosen later. These functions are [39, Lemma 4] $\kappa$ -uniformly convex for any $s\in\mathbb{S}$ and in turn, so is the population function $f$ . We proceed as follows, we first prove an information-theoretic (non-private) lower bound (Theorem 8 in Appendix C.1.1) which provides the statistical term in (7). With the same family of functions, we exhibit a collection of datasets and prove by contradiction that if an estimator were to optimize below a certain error it would have violated $\varepsilon$ -DP—this yields a lower bound on ERM for our function class (Theorem 9 in Appendix C.1.1). We conclude by proving a reduction from SCO to ERM in Proposition 4. ∎

$\kappa$ -growth with $\kappa\in(1,2]$ .

As the construction of the hard instance is more intricate for $\kappa<2$ , we provide a one-dimensional lower bound and leave the high-dimensional case for future work. In this case we directly obtain the result with a private version of Le Cam’s method [44, 42, 6], however with a different family of functions.

The issue with the construction of the previous section is that the function does not exhibit sharp growth for $\kappa<2$ . Indeed, the added linear function shifts the minimum away from $0$ where the function is differentiable and as a result it locally behaves as a quadratic and only achieves growth $\kappa=2$ . To establish the lower bound, we consider a different sample function $F$ that has growth exactly $1$ on one side and $\kappa$ on the other side. This yields the following

Theorem 5 (Lower bound for $\varepsilon$ -DP, $\kappa\in{(1,2]}$ ).

Let $d=1$ , $\mathbb{S}=\{-1,+1\}$ , $\kappa\in(1,2]$ , $\lambda=1$ , $L=2$ , and $n\in\mathbb{N}$ . There exists a collection of distributions $\mathcal{P}$ such that, whenever $n\varepsilon\geq 1/\sqrt{3}$ , it holds that

\mathfrak{M}_{n}([-1,1],\mathcal{P},\mathcal{F}^{\kappa}_{d=1},\epsilon)=\Omega\left\{\left(\frac{1}{\sqrt{n}}\right)^{\tfrac{\kappa}{\kappa-1}}+\left(\frac{1}{n\varepsilon}\right)^{\tfrac{\kappa}{\kappa-1}}\right\}.

(8)

5.2 Lower bounds under approximate privacy constraints

We conclude our treatment by providing lower bounds but now under approximate privacy constraints, demonstrating the optimality of the risk bound of Theorem 3. We prove the result via a reduction: we show that if one solves ERM with $\kappa$ -growth with error $\Delta$ , this implies that one solves arbitrary convex ERM with error $\phi(\Delta)$ . Given that a lower bound of $\Omega(\sqrt{d}/(n\varepsilon))$ holds for ERM, a lower bound of $\phi^{-1}(\sqrt{d}/(n\varepsilon))$ holds for ERM with $\kappa$ -growth. However, for this reduction to hold, we require that $\kappa\geq 2$ . Furthermore, we consider $\kappa$ to be roughly a constant—in the case that $\kappa$ is too large, standard lower bounds on general convex functions apply.

Theorem 6 (Private lower bound for $(\varepsilon,\delta)$ -DP).

Let $\kappa\geq 2$ such that $\kappa=\Theta(1)$ , $\mathcal{X}=\mathbb{B}_{2}^{d}(D)$ . Let $d\geq 1$ and $\mathbb{S}=\{\pm 1/\sqrt{d}\}^{d}$ . Assume that $n\varepsilon=\Omega(\sqrt{d})$ , then for any $(\varepsilon,\delta)$ mechanism $\mathsf{A}$ , there exists $\lambda>0,F$ and $\mathcal{S}\subset\mathbb{S}$ such that

\mathbb{E}[f_{\mathcal{S}}(\mathsf{A}(\mathcal{S}))]-\inf_{x^{\prime}\in\mathcal{X}}f_{\mathcal{S}}(x^{\prime})\geq\tilde{\Omega}\left[\frac{1}{\lambda^{\tfrac{1}{\kappa-1}}}\left(\frac{L\sqrt{d}}{n\varepsilon}\right)^{\tfrac{\kappa}{\kappa-1}}\right].

Theorem 6 implies that the same lower bound (up to logarithmic factors) applies to SCO via the reduction of [8, Appendix C]. Before proving the theorem, let us state (and prove in Section C.2) the following reduction: if an $(\varepsilon,\delta)$ -DP algorithm achieves excess error (roughly) $\Delta$ on ERM for any function with $\kappa$ -growth, there exists an $(\varepsilon,\delta)$ -DP algorithm that achieves error $\Delta^{(\kappa-1)/\kappa}$ for any convex function. We construct the latter by iteratively solving ERM problems with geometrically increasing $\|\cdot\|_{2}^{\kappa}$ -regularization towards the previous iterate to ensure the objective has $\kappa$ -growth.

Proposition 3 (Solving ERM with $\kappa$ -growth implies solving any convex ERM).

Let $\kappa\geq 2$ . Assume there exists an $(\epsilon,\delta)$ mechanism $\mathsf{A}$ such that for any $L$ -Lipschitz loss $G$ on $\mathcal{Y}$ and dataset $\mathcal{S}$ such that $g_{\mathcal{S}}(x)\coloneqq\frac{1}{n}\sum_{s\in\mathcal{S}}G(x;s)$ exhibits $(\lambda,\kappa)$ -growth, the mechanism achieves excess loss

\mathbb{E}[g_{\mathcal{S}}(\mathsf{A}(\mathcal{S},G,\mathcal{Y}))]-\inf_{y^{\prime}\in\mathcal{Y}}g_{\mathcal{S}}(y^{\prime})\leq\frac{1}{\lambda^{\tfrac{1}{\kappa-1}}}\Delta(n,L,\epsilon,\delta).

Then, we can construct an $(\varepsilon,\delta)$ -DP mechanism $\mathsf{A^{\prime}}$ such that for any $L$ -Lipschitz loss $f$ , the mechanism achieves excess loss

\mathbb{E}[f_{\mathcal{S}}(\mathsf{A}^{\prime}(\mathcal{S}))]-\inf_{x^{\prime}\in\mathcal{X}}f_{\mathcal{S}}(x^{\prime})\leq O\left(D\left[\Delta(n,L,\epsilon/k,\delta/k)\right]^{\tfrac{\kappa-1}{\kappa}}\right),

where $k$ is the smallest integer such that $k\geq\log\left[\frac{\kappa^{\tfrac{1}{\kappa-1}}L^{\tfrac{\kappa}{\kappa-1}}}{2^{2\kappa-3}\Delta(n,L,\varepsilon/k,\delta/k)}\right]$ .

With this proposition, the proof of the theorem directly follows as Bassily et al. [7] prove a lower bound $\Omega(\sqrt{d}/(n\varepsilon))$ for ERM with $(\varepsilon,\delta)$ -DP.

Discussion

In this work, we develop private algorithms that adapt to the growth of the function at hand, achieving the convergence rate corresponding to the “easiest” sub-class the function belongs to. However, the picture is not yet complete. First of, there are still gaps in our theoretical understanding, the most interesting one being $\kappa=1$ . On these functions, appropriate optimization algorithms achieve linear convergence [43] and raise the question, can we achieve exponentially small privacy cost in our setting? Finally, while our optimality guarantees are more fine-grained than the usual minimax results over convex functions, they are still contigent on some predetermined choice of sub-classes. Studying more general notions of adaptivity is an important future direction in private optimization.

Acknowledgments

The authors would like to thank Karan Chadha and Gary Cheng for comments on an early version of the draft.

References

Abadi et al. [2016] M. Abadi, A. Chu, I. Goodfellow, B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. In 23rd ACM Conference on Computer and Communications Security (ACM CCS), pages 308–318, 2016.
Asi and Duchi [2020a] H. Asi and J. Duchi. Near instance-optimality in differential privacy. arXiv:2005.10630 [cs.CR], 2020a.
Asi and Duchi [2020b] H. Asi and J. C. Duchi. Instance-optimality in differential privacy via approximate inverse sensitivity mechanisms. In Advances in Neural Information Processing Systems 33, 2020b.
Asi et al. [2021a] H. Asi, J. Duchi, A. Fallah, O. Javidbakht, and K. Talwar. Private adaptive gradient methods for convex optimization. arXiv:2106.13756 [cs.LG], 2021a.
Asi et al. [2021b] H. Asi, V. Feldman, T. Koren, and K. Talwar. Private stochastic convex optimization: Optimal rates in $\ell_{1}$ geometry. arXiv:2103.01516 [cs.LG], 2021b.
Barber and Duchi [2014] R. F. Barber and J. C. Duchi. Privacy and statistical risk: Formalisms and minimax bounds. arXiv:1412.4451 [math.ST], 2014.
Bassily et al. [2014] R. Bassily, A. Smith, and A. Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 55th Annual Symposium on Foundations of Computer Science, pages 464–473, 2014.
Bassily et al. [2019] R. Bassily, V. Feldman, K. Talwar, and A. Thakurta. Private stochastic convex optimization with optimal rates. In Advances in Neural Information Processing Systems 32, 2019.
Bassily et al. [2020] R. Bassily, V. Feldman, C. Guzmán, and K. Talwar. Stability of stochastic gradient descent on nonsmooth convex losses. In Advances in Neural Information Processing Systems 33, 2020.
Beck and Teboulle [2003] A. Beck and M. Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31:167–175, 2003.
Bolte et al. [2017] J. Bolte, T. P. Nguyen, J. Peypouquet, and B. Suter. From error bounds to the complexity of first-order descent methods for convex functions. Mathematical Programming, 165:471–507, 2017.
Bottou et al. [2018] L. Bottou, F. Curtis, and J. Nocedal. Optimization methods for large-scale learning. SIAM Review, 60(2):223–311, 2018.
Braverman et al. [2016] M. Braverman, A. Garg, T. Ma, H. L. Nguyen, and D. P. Woodruff. Communication lower bounds for statistical estimation problems via a distributed data processing inequality. In Proceedings of the Forty-Eighth Annual ACM Symposium on the Theory of Computing, 2016. URL https://arxiv.org/abs/1506.07216.
Cai and Low [2015] T. Cai and M. Low. A framework for estimating convex functions. Statistica Sinica, 25:423–456, 2015.
Chatterjee et al. [2016] S. Chatterjee, J. Duchi, J. Lafferty, and Y. Zhu. Local minimax complexity of stochastic convex optimization. In Advances in Neural Information Processing Systems 29, 2016.
Chaudhuri et al. [2011] K. Chaudhuri, C. Monteleoni, and A. D. Sarwate. Differentially private empirical risk minimization. Journal of Machine Learning Research, 12:1069–1109, 2011.
Duchi [2019] J. C. Duchi. Information theory and statistics. Lecture Notes for Statistics 311/EE 377, Stanford University, 2019. URL http://web.stanford.edu/class/stats311/lecture-notes.pdf. Accessed May 2019.
Duchi and Ruan [2021] J. C. Duchi and F. Ruan. Asymptotic optimality in stochastic optimization. Annals of Statistics, 49(1):21–48, 2021.
Duchi et al. [2013] J. C. Duchi, M. I. Jordan, and M. J. Wainwright. Local privacy and statistical minimax rates. In 54th Annual Symposium on Foundations of Computer Science, pages 429–438, 2013.
Dwork and Roth [2014] C. Dwork and A. Roth. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3 & 4):211–407, 2014.
Dwork et al. [2006a] C. Dwork, K. Kenthapadi, F. McSherry, I. Mironov, and M. Naor. Our data, ourselves: Privacy via distributed noise generation. In Advances in Cryptology (EUROCRYPT 2006), 2006a.
Dwork et al. [2006b] C. Dwork, F. McSherry, K. Nissim, and A. Smith. Calibrating noise to sensitivity in private data analysis. In Proceedings of the Third Theory of Cryptography Conference, pages 265–284, 2006b.
Dwork et al. [2012] C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. Zemel. Fairness through awareness. In Innovations in Theoretical Computer Science (ITCS), pages 214–226, 2012.
Feldman and Vondrak [2019] V. Feldman and J. Vondrak. High probability generalization bounds for uniformly stable algorithms with nearly optimal rate. In Proceedings of the Thirty Second Annual Conference on Computational Learning Theory, pages 1270–1279, 2019.
Feldman et al. [2020] V. Feldman, T. Koren, and K. Talwar. Private stochastic convex optimization: Optimal rates in linear time. In Proceedings of the Fifty-Second Annual ACM Symposium on the Theory of Computing, 2020.
Garg et al. [2014] A. Garg, T. Ma, and H. L. Nguyen. On communication cost of distributed statistical estimation and dimensionality. In Advances in Neural Information Processing Systems 27, 2014.
Hardt and Talwar [2010] M. Hardt and K. Talwar. On the geometry of differential privacy. In Proceedings of the Forty-Second Annual ACM Symposium on the Theory of Computing, pages 705–714, 2010. URL http://arxiv.org/abs/0907.3754.
Hashimoto et al. [2018] T. Hashimoto, M. Srivastava, H. Namkoong, and P. Liang. Fairness without demographics in repeated loss minimization. In Proceedings of the 35th International Conference on Machine Learning, 2018.
Hazan and Kale [2011] E. Hazan and S. Kale. An optimal algorithm for stochastic strongly convex optimization. In Proceedings of the Twenty Fourth Annual Conference on Computational Learning Theory, 2011. URL http://arxiv.org/abs/1006.2425.
Jin et al. [2019] C. Jin, P. Netrapalli, R. Ge, S. M. Kakade, and M. I. Jordan. A short note on concentration inequalities for random vectors with subgaussian norm. arXiv:1902.03736 [math.PR], 2019.
Juditsky and Nesterov [2010] A. Juditsky and Y. Nesterov. Primal-dual subgradient methods for minimizing uniformly convex functions. URL http://hal.archives-ouvertes.fr/docs/00/50/89/33/PDF/Strong-hal.pdf, 2010.
Juditsky and Nesterov [2014] A. Juditsky and Y. Nesterov. Deterministic and stochastic primal-dual subgradient algorithms for uniformly convex minimization. Stochastic Systems, 4(1):44––80, 2014.
Levy and Duchi [2019] D. Levy and J. C. Duchi. Necessary and sufficient geometries for gradient methods. In Advances in Neural Information Processing Systems 32, 2019.
Levy et al. [2021] D. Levy, Z. Sun, K. Amin, S. Kale, A. Kulesza, M. Mohri, and A. T. Suresh. Learning with user-level privacy. arXiv:2102.11845 [cs.LG], 2021. URL https://arxiv.org/abs/2102.11845.
McMahan et al. [2017] H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, 2017.
Mitzenmacher and Upfal [2005] M. Mitzenmacher and E. Upfal. Probability and computing: Randomized algorithms and probabilistic analysis. Cambridge University Press, 2005.
Nemirovski and Yudin [1983] A. Nemirovski and D. Yudin. Problem Complexity and Method Efficiency in Optimization. Wiley, 1983.
Nemirovski et al. [2009] A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on Optimization, 19(4):1574–1609, 2009.
Nesterov [2008] Y. Nesterov. Accelerating the cubic regularization of newton’s method on convex problems. Mathematical Programming, 112(1):159–181, 2008.
Ramdas and Singh [2013] A. Ramdas and A. Singh. Optimal rates for stochastic convex optimization under tsybakov noise condition. In Proceedings of the 30th International Conference on Machine Learning, pages 365–373, 2013.
Smith and Thakurta [2013] A. Smith and A. Thakurta. Differentially private feature selection via stability arguments, and the robustness of the Lasso. In Proceedings of the Twenty Sixth Annual Conference on Computational Learning Theory, pages 819–850, 2013. URL http://proceedings.mlr.press/v30/Guha13.html.
Wainwright [2019] M. J. Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint. Cambridge University Press, 2019.
Xu et al. [2017] Y. Xu, Q. Lin, and T. Yang. Stochastic convex optimization: Faster local growth implies faster global convergence. In Proceedings of the 34th International Conference on Machine Learning, pages 3821–3830, 2017.
Yu [1997] B. Yu. Assouad, Fano, and Le Cam. In Festschrift for Lucien Le Cam, pages 423–435. Springer-Verlag, 1997.
Zhang et al. [2013] Y. Zhang, J. C. Duchi, M. I. Jordan, and M. J. Wainwright. Information-theoretic lower bounds for distributed estimation with communication constraints. In Advances in Neural Information Processing Systems 26, 2013.
Zhu et al. [2016] Y. Zhu, S. Chatterjee, J. Duchi, and J. Lafferty. Local minimax complexity of stochastic convex optimization. In Advances in Neural Information Processing Systems 29, 2016.

Appendix A Proofs for Section 3

A.1 Proof of Theorem 1

See 1

Let us first prove privacy. The sensitivity of $\|\nabla f_{\mathcal{S}}(x)\|_{2}$ is $2L/n$ as $F$ is $L$ -Lipschitz, therefore following the privacy proof of the smooth inverse sensitivity mechanism [2, Prop. 3.2] we get that $\mathsf{A}_{\mathrm{gr{-}inv}}$ (5) is $\varepsilon$ -DP.

Let us now prove the claim about utility. Denote $\hat{x}=\mathsf{A}_{\mathrm{gr{-}inv}}(\mathcal{S})$ and $E=\frac{2LK}{n\varepsilon}$ with $K$ to be chosen presently. We argue that it is enough to show that $\Pr(G_{\rho}(\hat{x})\geq E)\leq\beta$ . Indeed then with probability at least $1-\beta$ we have $G_{\rho}(\hat{x})\leq E$ , which implies there is $y$ such that $\|\hat{x}-y\|_{2}\leq\rho$ and $\|\nabla f_{\mathcal{S}}(y)\|_{2}\leq E$ , hence using the Kurdyka-Łojasiewicz inequality (2)

	$\displaystyle f_{\mathcal{S}}(\hat{x})-f_{\mathcal{S}}(x^{\star})$	$\displaystyle=f_{\mathcal{S}}(\hat{x})-f_{\mathcal{S}}(y)+f_{\mathcal{S}}(y)-f_{\mathcal{S}}(x^{\star})$
		$\displaystyle\leq L\rho+\frac{e}{\lambda^{\frac{1}{\kappa-1}}}\\|\nabla f_{\mathcal{S}}(y)\\|_{2}^{\frac{\kappa}{\kappa-1}}$
		$\displaystyle\leq L\rho+\frac{e}{\lambda^{\frac{1}{\kappa-1}}}E^{\frac{\kappa}{\kappa-1}}.$

It remains to prove that $\Pr(G_{\rho}(\hat{x})\geq E)\leq\beta$ . Let $S_{0}=\{x\in\mathbb{R}^{d}:\|x-x^{\star}\|_{2}\leq\rho\}$ and $S_{1}=\{x\in\mathbb{R}^{d}:G_{\rho}(x)\geq E\}$ . Note that $G_{\rho}(x)=0$ for any $x\in S_{0}$ as $x^{\star}$ is in the interior of $\mathcal{X}$ which implies $\nabla f_{\mathcal{S}}(x^{\star})=0$ . Hence the definition of the smooth inverse sensitivity mechanism (5) implies

	$\displaystyle\Pr(\mathsf{A}_{\mathrm{gr{-}inv}}(\mathcal{S})\in S_{1})$	$\displaystyle\leq\frac{\mathsf{Vol}(\{x\in\mathbb{R}^{d}:\\|x-x^{\star}\\|_{2}\leq D+\rho\})e^{-\frac{n\varepsilon}{2L}E}}{\mathsf{Vol}(\{x\in\mathbb{R}^{d}:\\|x-x^{\star}\\|_{2}\leq\rho\})}$
		$\displaystyle\leq e^{-K}\left(1+\frac{D}{\rho}\right)^{d}\leq\beta,$

where the last inequality follows by choosing $K=\log(1/\beta)+d\log(1+D/\rho)$ .

Appendix B Proofs for Section 4

We need to the following result on the generalization properties of uniformly stable algorithms [24].

Theorem 7.

[24, Cor. 4.2] Assume $\mathsf{diam}_{2}(\mathcal{X})\leq D$ . Let $\mathcal{S}=(S_{1},\dots,S_{n})$ where $S_{1}^{n}\stackrel{{\scriptstyle\rm iid}}{{\sim}}P$ and $F(x;s)$ is $L$ -Lipschitz and $\lambda$ -strongly convex for all $s\in\mathbb{S}$ . Let $\hat{x}=\mathop{\rm argmin}_{x\in\mathcal{X}}{f_{\mathcal{S}}(x)}$ be the empirical minimizer. For $0<\beta\leq 1/n$ , with probability at least $1-\beta$

f(\hat{x})-f(x^{\star})\leq\frac{cL^{2}\log(n)\log(1/\beta)}{\lambda n}+\frac{cLD\sqrt{\log(1/\beta)}}{\sqrt{n}}.

B.1 Proof of Proposition 1

See 1

We begin by proving the privacy claim. We show that each iterate is $\varepsilon$ -DP and therefore post-processing implies the claim as each sample is used in exactly one iterate. To this end, let $\lambda_{i}=1/\eta_{i}n_{0}$ and note that the minimizer $\hat{x}_{i}$ has $\ell_{2}$ sensitivity $2L/\lambda_{i}n_{0}\leq 4L\eta_{i}$ [25], hence the $\ell_{1}$ -sensitivity is at most $4L\eta_{i}\sqrt{d}$ . Standard properties of the Laplace mechanism [20] now imply that $x_{i}$ is $\varepsilon$ -DP which give the claim about privacy.

Now we proceed to prove utility which follows similar arguments to the localization-based proof in [25]. Letting $\hat{x}_{0}=x^{\star}$ , we have:

\displaystyle f(x_{k})-f(x^{\star})

\displaystyle=\sum_{i=1}^{k}f(\hat{x}_{i})-f(\hat{x}_{i-1})+f(x_{k})-f(\hat{x}_{k}).

First, by using standard properties of Laplace distributions [17], we know that for $\zeta_{i}\sim\mathsf{Lap}(\sigma_{i})$ ,

\Pr(\|\zeta_{i}\|_{2}\geq t)\leq\Pr(\|\zeta_{i}\|_{\infty}\geq t/\sqrt{d})\leq de^{-t/\sqrt{d}\sigma_{i}},

which implies (as $\beta\leq 1/(n+d)$ ) that with probability $1-\beta/2$ we have $\|\zeta_{i}\|_{2}\leq 10\sqrt{d}\sigma_{i}\log(1/\beta)$ for all $1\leq i\leq k$ . Hence

	$\displaystyle f(x_{k})-f(\hat{x}_{k})$	$\displaystyle\leq L\\|x_{k}-\hat{x}_{k}\\|_{2}$
		$\displaystyle\leq L\sigma_{k}\sqrt{d}\log(1/\beta)$
		$\displaystyle\leq 4L^{2}d\frac{\eta_{i}}{\varepsilon}$
		$\displaystyle\leq 4L^{2}d\frac{\eta}{\varepsilon 2^{4i}}\leq\frac{4LD}{n^{2}},$

where the last inequality follows since $\eta=\frac{D\varepsilon}{Ld\log(k/\beta)}$ . Now we use high-probability generalization guarantees of uniformly-stable algorithms. We use Theorem 7 with $F(x;s_{j})+\frac{\|x-x_{i-1}\|_{2}^{2}}{\eta_{i}n_{0}}$ to get that with probability $1-\beta/2$ for each $i$

\displaystyle f(\hat{x}_{i})-f(\hat{x}_{i-1})\leq\frac{\|\hat{x}_{i-1}-x_{i-1}\|_{2}^{2}}{\eta_{i}n_{0}}+{cL^{2}\log(n)\log(1/\beta)\eta_{i}}+\frac{cLD\sqrt{\log(1/\beta)}}{\sqrt{n_{0}}}.

Thus,

	$\displaystyle\sum_{i=1}^{k}f(\hat{x}_{i})-f(\hat{x}_{i-1})$	$\displaystyle\leq\sum_{i=1}^{k}\left\{\frac{\\|\hat{x}_{i-1}-x_{i-1}\\|_{2}^{2}}{\eta_{i}n_{0}}+{cL^{2}\log(n)\log(1/\beta)\eta_{i}}+\frac{cLD\sqrt{\log(1/\beta)}}{\sqrt{n_{0}}}\right\}$
		$\displaystyle\leq\frac{D^{2}}{\eta n_{0}}+\left[\sum_{i=2}^{k}\frac{\sigma_{i-1}^{2}d\log^{2}(1/\beta)}{\eta_{i}n_{0}}\right]+{2cL^{2}\log(n)\log(1/\beta)\eta}+\frac{cLD\sqrt{\log(1/\beta)}k}{\sqrt{n_{0}}}$
		$\displaystyle=\frac{D^{2}}{\eta n_{0}}+\left[\sum_{i=2}^{k}\frac{CL^{2}\eta_{i-1}d^{2}\log^{2}(1/\beta)}{n_{0}\varepsilon^{2}}\right]+{2cL^{2}\log(n)\log(1/\beta)\eta}+\frac{cLD\sqrt{\log(1/\beta)}k}{\sqrt{n_{0}}}$
		$\displaystyle=\frac{D^{2}}{\eta n_{0}}+\frac{CL^{2}\eta d^{2}\log^{2}(1/\beta)}{n_{0}\varepsilon^{2}}\left[\sum_{i=2}^{k}2^{-i}\right]+{2cL^{2}\log(n)\log(1/\beta)\eta}+\frac{cLD\sqrt{\log(1/\beta)}k}{\sqrt{n_{0}}}$
		$\displaystyle\leq LD\cdot O\left(\frac{\sqrt{\log(1/\beta)\log(n)}+\sqrt{\log(1/\beta)}\log^{3/2}(n)}{\sqrt{n}}+\frac{d\log(1/\beta)\log(n)}{n\varepsilon}\right),$

where the last inequality follows by choosing $\eta=\frac{D}{L}\min\left(\frac{1}{\sqrt{n\log(1/\beta)}},\frac{\varepsilon}{d\log(1/\beta)}\right)$

B.2 Proof of Proposition 2

See 2

The proof is similar to the proof of Proposition 1. For privacy, we show in the proof of Proposition 1 that the $\ell_{2}$ -sensitivity of $\hat{x}_{i}$ is upper bounded by $2L/\lambda_{i}n_{0}\leq 4L\eta_{i}$ hence standard properties of the Gaussian mechanism [20] now imply that $x_{i}$ is $(\varepsilon,\delta)$ -DP which implies the final algorithm is $(\varepsilon,\delta)$ -DP using post-processing.

The utility proof follows the same arguments as in the proof of Proposition 1, except that for $\zeta_{i}\sim\mathsf{N}(0,\sigma_{i}^{2})$ we have [30] (since $\zeta_{i}$ is $2\sqrt{2}\sigma_{i}\sqrt{d}$ -norm-sub-Gaussian)

\Pr(\|\zeta_{i}\|_{2}\geq t\sqrt{d})\leq 2e^{-\tfrac{t^{2}}{16\sigma_{i}^{2}}},

implying that $\|\zeta_{i}\|_{2}\leq 4\sqrt{d}\sigma_{i}\log(4/\beta)$ for all $1\leq i\leq k$ with probability $1-\beta/2$ .

B.3 Proofs of Theorems 2 and 3

We first restate Theorems 2 and 3.

See 2

See 3

We start by proving privacy. Since each sample $s_{i}$ is used in exactly one iterate, we only need to show that each iterate is $(\varepsilon,\delta)$ -DP, which will imply the main claim using post-processing. The privacy of each iterate follows directly from the privacy guarantees of Algorithm 1. We proceed to prove utility.

We will prove the utility claim assuming the subroutine used in Algorithm 2 satisfies the following: the output $x_{k+1}$ has error

f(x_{k+1})-\min_{x\in\mathcal{X}}f(x)\leq D_{k}\cdot\rho,

for some $\rho>0$ . Note that in our setting, Proposition 1 implies that $\rho\leq L\cdot O(\frac{\sqrt{\log(1/\beta)}\log n_{0}}{\sqrt{n_{0}}}+\frac{d\log(1/\beta)}{n_{0}\varepsilon})$ for pure-DP and similarly Proposition 2 gives the corresponding $\rho$ for $(\varepsilon,\delta)$ -DP.

The proof has two stages. In the first stage (Lemma B.1), we prove that as long as $i\leq i_{0}$ for some $i_{0}>0$ , then $x^{\star}\in\mathcal{X}_{i}$ and the performance of the algorithm keeps improving. We show that at the end of this stage, the points $x_{i_{0}+1}$ has optimal excess loss. Then, in the second stage (Lemma B.2), we show that the iterates would not move much as the radius $D_{i}$ of the domain is sufficiently small, hence the final accumulated error along these iterations is small.

Let us begin with the first stage. Let $i_{0}$ be the largest $i$ such that $D_{i}\geq(\frac{\kappa 2^{\kappa}\rho}{\lambda})^{\frac{1}{\kappa-1}}$ . We prove that $x^{\star}\in\mathcal{X}_{i}$ for all $0\leq i\leq i_{0}$ where we recall that $\mathcal{X}_{i}=\{x\in\mathcal{X}:\|x-x_{i}\|_{2}\leq D_{i}\}$ and $D_{i}=2^{-i}D_{0}$ .

Lemma B.1.

For all $0\leq i\leq i_{0}$ we have

x^{\star}\in\mathcal{X}_{i}\quad\mbox{~{}~{}and~{}~{}}\quad f(x_{i_{0}+1})-\min_{x\in\mathcal{X}}f(x)\leq 4({2^{\kappa}})^{\frac{1}{\kappa-1}}\frac{1}{\lambda^{\frac{1}{\kappa-1}}}\rho^{\frac{\kappa}{\kappa-1}}.

Proof.

To prove the first part, we need to show that $\|x_{i}-x^{\star}\|_{2}\leq D_{i}$ . Let $\bar{D}_{i}=\|x_{i}-x^{\star}\|_{2}$ . First, note that the claim is true for $i=0$ . Now we assume it is correct for $0\leq i\leq i_{0}-1$ and prove correctness for $i+1$ . Note that the growth condition implies

\bar{D}_{i+1}\leq(\kappa\Delta_{i}/\lambda)^{1/\kappa},

where $\Delta_{i}=f(x_{i+1})-\min_{x\in\mathcal{X}}f(x)\leq D_{i}\cdot\rho$ . Thus we have

\bar{D}_{i+1}\leq(\kappa D_{i}\rho/\lambda)^{1/\kappa}\leq D_{i}/2=D_{i+1},

where the second inequality holds for $i$ that satisfies $D_{i}\geq(\frac{\kappa 2^{\kappa}\rho}{\lambda})^{\frac{1}{\kappa-1}}$ . This proves the first part of the claim. For the second part, note that the definition of $i_{0}$ implies that $D_{i_{0}}\leq 2(\frac{\kappa 2^{\kappa}\rho}{\lambda})^{\frac{1}{\kappa-1}}$ . Therefore, as $x^{\star}\in\mathcal{X}_{i_{0}}$ and the algorithm has error $D_{i}\cdot\rho$ , we have

	$\displaystyle f(x_{i_{0}+1})-\min_{x\in\mathcal{X}}f(x)$	$\displaystyle\leq D_{i_{0}}\cdot\rho$
		$\displaystyle\leq 2({\kappa 2^{\kappa}}/{\lambda})^{\frac{1}{\kappa-1}}\rho^{\frac{\kappa}{\kappa-1}}.$

The claim now follows as $\kappa^{\frac{1}{\kappa-1}}\leq 2$ . ∎

We now proceed to the second stage. The following lemma shows that the accumulated error along the iterates $i>i_{0}$ is small and therefore $x_{T}$ obtains the same error as $x_{i_{0}+1}$ (up to constant factors).

Lemma B.2.

Assume the algorithm has error $D_{i}\cdot\rho$ . Let $i_{0}$ be the largest $i$ such that $D_{i}\geq(\frac{\kappa 2^{\kappa}\rho}{\lambda})^{\frac{1}{\kappa-1}}$ . For all $i\geq i_{0}+1$ we have

f(x_{i+1})-f(x_{i})\leq 2^{-(i-i_{0})}D_{i_{0}}\rho.

In particular, for $T\geq i_{0}+1$ we have

f(x_{T})-\min_{x\in\mathcal{X}}f(x)\leq 8({2^{\kappa}}/{\lambda})^{\frac{1}{\kappa-1}}\rho^{\frac{\kappa}{\kappa-1}}.

Proof.

Note that as $x_{i}\in\mathcal{X}_{i}$ , the guarantees of the algorithm give

f(x_{i+1})-f(x_{i})\leq D_{i}\rho=2^{-(i-i_{0})}D_{i_{0}}\rho.

For the second part of the claim, we have

	$\displaystyle f(x_{T})-\min_{x\in\mathcal{X}}f(x)$	$\displaystyle=f(x_{i_{0}+1})-\min_{x\in\mathcal{X}}f(x)+\sum_{i=i_{0}+1}^{T}f(x_{i+1})-f(x_{i})$
		$\displaystyle\leq D_{i_{0}}\rho+\sum_{i=i_{0}+1}^{T}2^{-(i-i_{0})}D_{i_{0}}\rho\leq 2D_{i_{0}}\rho.$

The claim now follows as $D_{i_{0}}\leq 2(\frac{\kappa 2^{\kappa}\rho}{\lambda})^{\frac{1}{\kappa-1}}$ and $\kappa^{\frac{1}{\kappa-1}}\leq 2$ . ∎

Assuming $T\geq i_{0}+1$ , Theorem 2 and Theorem 3 now follow immediately from Lemma B.2. Indeed, for the case of pure-DP ( $\delta=0$ ), the choice of hyper-parameters in Algorithm 2 and the guarantees of Algorithm 1 (Proposition 1) imply that $\rho\leq L\cdot O(\frac{\sqrt{\log(1/\beta)}\log n_{0}}{\sqrt{n_{0}}}+\frac{Td\log(1/\beta)}{n_{0}\varepsilon})$ , which proves Theorem 2. Similarly, Theorem 3 follows by using the guarantees of of Algorithm 1 for approximate $(\varepsilon,\delta)$ -DP, that ism Proposition 2, which gives $\rho\leq L\cdot O\left(\frac{\sqrt{\log(1/\beta)}\log n_{0}}{\sqrt{n_{0}}}+\frac{T\sqrt{d\log(1/\delta)}\log(1/\beta)}{n_{0}\varepsilon}\right)$ . Note that our choice of stepsize at each iterate implies that Theorem 2 guarantees the desired utility with probability at least $1-\beta^{2}$ , hence the final utility guarantee holds with probability at least $1-T\beta^{2}\geq 1-\beta$ .

It remains to verify $T\geq i_{0}+1$ . Note that by choosing $T\geq\frac{2\log(D_{0}^{\kappa-1}\lambda/\rho)}{\underline{\kappa}-1}$ , we get that $D_{T}\leq(\frac{\kappa 2^{\kappa}\rho}{\lambda})^{\frac{1}{\kappa-1}}$ , hence $T\geq i_{0}+1$ . As we have $\rho\geq L/\sqrt{n_{0}}$ (non-private error) and $D_{0}^{\kappa-1}\leq L/\lambda$ in our setting, we get that choosing $T=\frac{2\log n}{\underline{\kappa}-1}$ gives the claim.

Appendix C Proofs of Section 5

In this section, we provide the proofs for our lower bound under privacy constraints for functions with growth. This section is organized as follows: we prove in Section C.1, the lower bounds under pure-DP and in Section C.2, the lower bounds under approximate-DP. Within Section C.1, we distinguish between $\kappa\geq 2$ (Section C.1.1) and $\kappa\in(1,2)$ (Section C.1.2).

C.1 Proofs of Section 5.1

C.1.1 Proof of Theorem 4

As we preview in the main text, the proof combines the (non-private) information-theoretic lower bounds of Theorem 8 with the (private) lower bound on ERM of Theorem 9. Finally, we show in Proposition 4 that privately solving SCO is harder than privately solving ERM, concluding the proof of the theorem. We restate the theorem and prove these results in sequence.

See 4

Non-private lower bound

We begin the proof of Theorem 4 by proving a (non-private) information-theoretic lower bound for minimizing functions with $\kappa\geq 2$ -growth. We use the standard reduction from estimation to testing [see 33, Appendix A.1] in conjunction with Fano’s method [42, 44].

Theorem 8 (Non-private lower bound).

2^{\kappa-1}\leq\frac{L}{\lambda}\frac{1}{R^{\kappa-1}}\leq 2^{\kappa-1}\sqrt{96n}.

The following lower bound holds

\mathfrak{M}_{n}(\mathcal{X},\mathcal{P},\mathcal{F}^{\kappa})\gtrsim\frac{1}{\lambda^{\tfrac{1}{\kappa-1}}}\left(\frac{L}{\sqrt{n}}\right)^{\tfrac{\kappa}{(\kappa-1)}}.

Proof.

For $\mathcal{V}\subset\{\pm 1\}^{d}$ let us consider the following function and distribution

F(x;s)\coloneqq\frac{\lambda 2^{\kappa-2}}{\kappa}\|x\|_{2}^{\kappa}+\frac{L}{2}\langle x,s\rangle\mbox{~{}~{}and~{}~{}}X\sim P_{v}\mbox{~{}~{}implies~{}~{}}X_{j}=\begin{cases}v_{j}e_{j}\mbox{~{}~{}w.p.~{}~{}}\frac{1+\delta}{2}\\ -v_{j}e_{j}\mbox{~{}~{}w.p.~{}~{}}\frac{1-\delta}{2}.\end{cases}

Since the linear term does not affect uniform convexity, Lemma 4 in [39] guarantees that $f_{v}$ is $(\lambda,\kappa)$ -uniformly convex. Furthermore, for $s\in\mathbb{S}$

\|\nabla F(x;s)\|_{2}\leq\lambda 2^{\kappa-2}R^{\kappa-1}+\frac{L}{2}\leq L,

by assumption, so the functions are $L$ -Lipschitz and satisfy Assumption 1.

Computing the separation. As $\mathbb{E}_{P_{v}}S=\tfrac{\delta}{d}v$ , we have

f_{v}(x)=\frac{\lambda 2^{\kappa-2}}{\kappa}\|x\|_{2}^{\kappa}+\frac{L\delta}{2d}\langle x,v\rangle.

Note that for $u\in\mathbb{R}^{d},\sigma>0$ , it holds that

\inf_{x\in\mathbb{R}^{d}}\sigma\frac{\|x\|_{2}^{\kappa}}{\kappa}+\langle x,u\rangle=-\frac{1}{\kappa^{\star}}\left(\frac{1}{\sigma}\right)^{\tfrac{1}{\kappa-1}}\|u\|^{\tfrac{\kappa}{\kappa-1}}\mbox{~{}~{}at~{}~{}}x_{u}^{\star}=-\left(\frac{1}{\sigma}\right)^{\tfrac{1}{\kappa-1}}\left(\frac{1}{\|u\|_{2}}\right)^{\tfrac{\kappa-2}{\kappa-1}}u.

To make sure that $x_{u}^{\star}\in\mathbb{B}_{2}^{d}(R)$ , we require $\|{u}\|_{2}\leq\sigma R^{\kappa-1}$ . After choosing $\delta$ , we will see that this holds under the assumptions of the theorem. Let us consider the Gilbert-Varshimov packing of the hypercube: there exists $\mathcal{V}\subset\left\{\pm 1\right\}^{d}$ such that $\lvert\mathcal{V}\rvert=\exp(d/8)$ and $d_{\rm Ham}(v,v^{\prime})\geq d/4$ for all $v\neq v^{\prime}\in\mathcal{V}$ . Let us compute the separation

\inf_{x\in\mathbb{B}_{2}^{d}(R)}\frac{f_{v}(x)+f_{v^{\prime}}(x)}{2}=-\frac{1}{4\kappa^{\star}\lambda^{\tfrac{1}{\kappa-1}}}\left(\frac{L\delta}{d}\right)^{\tfrac{\kappa}{\kappa-1}}\left\|\frac{v+v^{\prime}}{2}\right\|_{2}^{\tfrac{\kappa}{\kappa-1}}

Note that $\|(v+v^{\prime})/2\|_{2}=\sqrt{d-d_{\rm Ham}(v,v^{\prime})}\leq\sqrt{3d/4}$ . This yields a separation

d_{\mathsf{opt}}(v,v^{\prime},\mathcal{X})\geq\frac{1-(3/4)^{\kappa/(2\kappa-2)}}{2\kappa^{\star}\lambda^{\tfrac{1}{\kappa-1}}}\left(\frac{L\delta}{\sqrt{d}}\right)^{\tfrac{\kappa}{\kappa-1}}.

Lower bounding the testing error. In the case of a multiple hypothesis test, we use Fano’s method and for $V\sim\mathsf{Uni}\{\mathcal{V}\}$ and $S_{1}^{n}|V=v\stackrel{{\scriptstyle\rm iid}}{{\sim}}P_{v}$ , Fano’s inequality guarantees

\inf_{\psi:\mathbb{S}^{n}\to\mathcal{V}}\Pr(\psi(S_{1}^{n})\neq V)\geq 1-\frac{\mathsf{I}(S_{1}^{n};V)+\log 2}{\log\lvert\mathcal{V}\rvert},

where $\mathsf{I}(X;Y)$ is the Shannon mutual information between $X$ and $Y$ . In our case, we have $\log\lvert\mathcal{V}\rvert\geq d/8$ and $\mathsf{I}(S_{1}^{n};V)\leq n\max_{v\neq v^{\prime}}D_{\rm kl}({P_{v}}|\!|{P_{v^{\prime}}})\leq 3n\delta^{2}$ . In the case $d\geq 48\log 2$ , we choose $\delta=\sqrt{d/(24n)}$ . We handle the one-dimensional case thereafter. For this $\delta$ , we have

\mathfrak{M}_{n}(\mathcal{X},\mathcal{P},\mathcal{F}^{\kappa})\geq\frac{1-\left(\tfrac{3}{4}\right)^{\tfrac{\kappa}{2\kappa-2}}}{4\kappa^{\star}(24)^{\tfrac{\kappa}{2\kappa-2}}}\frac{1}{\lambda^{\tfrac{1}{\kappa-1}}}\left(\frac{L^{2}}{n}\right)^{\tfrac{\kappa}{2\kappa-2}}.

For this choice of $\delta$ , the assumption on $n$ ensures that the minimum remains in $\mathbb{B}_{2}^{d}(R)$ .

One-dimensional lower bound with Le Cam’s method. Since Fano’s method requires $d\geq 48\log 2$ , we finish the proof by providing a lower bound for $d=1$ using Le Cam’s method. We use the same family of functions in one dimension, i.e. $\mathbb{S}=\{\pm 1\}$ , $v\in\{\pm 1\}$ and for $\delta\in[0,1]$ define

F(x;s)=\frac{\lambda 2^{\kappa-2}}{\kappa}\lvert x\rvert^{\kappa}+\frac{L}{2}s\cdot x\mbox{~{}~{}and~{}~{}}X\sim P_{v}\mbox{~{}~{}implies~{}~{}}X=\begin{cases}v&\mbox{~{}~{}w.p.~{}~{}}\frac{1+\delta}{2}\\ -v&\mbox{~{}~{}w.p.~{}~{}}\frac{1-\delta}{2}.\end{cases}

As this is the one-dimensional analog of the previous construction, $F$ remains $L$ -lipschitz and $f$ has $(\lambda,\kappa)$ -growth. A calculation yields that the separation is

d_{\mathsf{opt}}(1,-1,\mathcal{X})\geq\frac{1}{2\lambda^{\tfrac{1}{\kappa-1}}}\left(L\delta\right)^{\tfrac{\kappa}{\kappa-1}},

where we used that $\kappa^{\star}\in[1,2]$ . For $V\sim\mathsf{Uni}\{-1,1\}$ and $S_{1}^{n}|V=v\stackrel{{\scriptstyle\rm iid}}{{\sim}}P_{v}$ . Le Cam’s lemma in conjunction with Pinsker’s inequality yields that

\inf_{\psi:\mathbb{S}^{n}\to\{-1,1\}}\Pr(\psi(S_{1}^{n})\neq V)=\frac{1}{2}(1-\|P_{1}^{n}-P_{-1}^{n}\|_{\rm TV})\geq\frac{1}{2}(1-\sqrt{\tfrac{n}{2}D_{\rm kl}({P_{1}}|\!|{P_{-1}})}).

In our case, we have $D_{\rm kl}({P_{1}}|\!|{P_{-1}})=\delta\log\frac{1+\delta}{1-\delta}\leq 3\delta^{2}$ for $\delta\in[0,1/2]$ . We set $\delta=1/\sqrt{6n}$ , which yields the final result in one dimension

\mathfrak{M}_{n}([-1,1],\mathcal{P},\mathcal{F}^{\kappa}_{d=1})=\Omega\left(\frac{1}{\lambda^{\tfrac{1}{\kappa-1}}}\left(\frac{L}{\sqrt{n}}\right)^{\tfrac{\kappa}{\kappa-1}}.\right)

∎

Privatizing the lower bound via a packing argument

We now show how this construction yields a private lower bound via a packing argument. For $d\geq 1$ , considering the ERM problem, the following private lower bound holds.

Theorem 9 (Private lower bound for ERM).

Let $d\geq 1,\mathcal{X}=\mathbb{B}_{2}^{d}(R)$ , $\mathbb{S}=\{\pm e_{j}\}_{j\leq d}$ , $\kappa\geq 2$ and $n\in\mathbb{N}$ . Let $\mathcal{P}$ be the set of distributions on $\mathbb{S}$ . Assume that

2^{\kappa-1}\leq\frac{L}{\lambda}\frac{1}{R^{\kappa-1}}\leq 2^{\kappa-1}\sqrt{96n}.

Then any $\varepsilon$ -DP algorithm $\mathsf{A}$ has

\sup_{\mathcal{S}\in\mathbb{S}^{n}}\mathbb{E}\left[f_{\mathcal{S}}(\mathsf{A}(\mathcal{S}))-\min_{x\in\mathcal{X}}f_{\mathcal{S}}(x)\right]\gtrsim\frac{1}{\lambda^{\frac{1}{\kappa-1}}}\left(\frac{Ld}{n\varepsilon}\right)^{\frac{\kappa}{\kappa-1}}.

Proof.

First, note that it is enough to prove the following lower bound

\sup_{\mathcal{S}\in\mathbb{S}^{n}}\mathbb{E}\left[\|\mathsf{A}(\mathcal{S})-x^{\star}\|_{2}\right]\gtrsim\frac{1}{\lambda^{\frac{1}{\kappa-1}}}\left(\frac{Ld}{n\varepsilon}\right)^{\frac{1}{\kappa-1}}.

(9)

Indeed, this implies that

	$\displaystyle\sup_{\mathcal{S}\in\mathbb{S}^{n}}\mathbb{E}\left[f(\mathsf{A}(\mathcal{S}))-\min_{x\in\mathcal{X}}f(x)\right]$	$\displaystyle\geq\frac{\lambda}{\kappa}\sup_{\mathcal{S}\in\mathbb{S}^{n}}\mathbb{E}\left[\\|\mathsf{A}(\mathcal{S})-x^{\star}\\|_{2}^{\kappa}\right]$
		$\displaystyle\gtrsim\frac{1}{\kappa\lambda^{\frac{1}{\kappa-1}}}\left(\frac{Ld}{n\varepsilon}\right)^{\frac{\kappa}{\kappa-1}}.$

Let us now prove the lower bound (9). To this end, we consider the function $F(x;s)\coloneqq\frac{\lambda 2^{\kappa-2}}{\kappa}\|x\|_{2}^{\kappa}+\frac{L}{2}\langle x,s\rangle$ where $\|s\|_{2}\leq 1$ . We now construct $M$ datasets $\mathcal{S}_{1},\dots,\mathcal{S}_{M}$ as follows. Let $v_{1},\dots,v_{M}\in\left\{\pm\frac{1}{\sqrt{d}}\right\}^{d}$ be the Gilbert-Varshimov packing of the hypercube: that is, $M\geq\exp(d/8)$ and $d_{\rm Ham}(v_{i},v_{j})\geq d/4$ for all $i\neq j$ . We define $\mathcal{S}_{i}=(\underbrace{v_{i},\dots,v_{i}}_{d/20\varepsilon},0,\dots,0)$ . Note that $d_{\rm Ham}(S_{i},S_{j})\leq d/20\varepsilon$ and that $f(x;\mathcal{S}_{i})=\frac{\lambda 2^{\kappa-2}}{\kappa}\|x\|_{2}^{\kappa}+\frac{L}{2}\frac{d}{20n\varepsilon}\langle x,v_{i}\rangle$ , hence

x_{i}^{\star}=-\left(\frac{1}{\lambda 2^{\kappa-2}}\right)^{\tfrac{1}{\kappa-1}}\left(\frac{40n\varepsilon}{Ld}\right)^{\tfrac{\kappa-2}{\kappa-1}}\frac{Ld}{40n\varepsilon}v_{i}.

Therefore we have

	$\displaystyle\\|x_{i}^{\star}-x_{j}^{\star}\\|_{2}^{2}$	$\displaystyle\geq\left(\frac{1}{\lambda 2^{\kappa-2}}\right)^{\tfrac{2}{\kappa-1}}\left(\frac{40n\varepsilon}{Ld}\right)^{\tfrac{2(\kappa-2)}{\kappa-1}}\frac{L^{2}d^{2}}{1600n^{2}\varepsilon^{2}}$
		$\displaystyle\gtrsim\left(\frac{1}{\lambda 2^{\kappa-2}}\right)^{\tfrac{2}{\kappa-1}}\left(\frac{Ld}{n\varepsilon}\right)^{\tfrac{2}{\kappa-1}}\coloneqq\rho^{2}.$

We are now ready to finish the proof using packing-based arguments [27]. Assume by contradiction there is an $\varepsilon$ -DP algorithm $\mathsf{A}$ such that

\sup_{1\leq i\leq M}\mathbb{E}\left[\|\mathsf{A}(\mathcal{S}_{i})-x_{i}^{\star}\|_{2}\right]\leq\rho/20.

Let $B_{i}=\{x\in\mathcal{X}:\|x-x_{i}^{\star}\|_{2}\leq\rho/2\}$ . Note that the sets $B_{i}$ are disjoint and that Markov inequality implies

\Pr(\mathsf{A}(\mathcal{S}_{i})\in B_{i})=\Pr(\|\mathsf{A}(\mathcal{S}_{i})-x_{i}^{\star}\|_{2}\leq\rho/2)\geq 9/10.

Thus, the privacy constraint now gives

	$\displaystyle 1$	$\displaystyle\geq\sum_{i=1}^{M}\Pr(\mathsf{A}(x_{1})\in B_{i})$
		$\displaystyle\geq\Pr(\mathsf{A}(x_{1})\in B_{1})+e^{-d/20}\sum_{i=2}^{M}\Pr(\mathsf{A}(x_{i})\in B_{i})$
		$\displaystyle\geq\frac{9}{10}(1+e^{-d/20}(M-1)),$

where the second inequality follows since $d_{\rm Ham}(\mathcal{S}_{i},\mathcal{S}_{j})\leq d/20\varepsilon$ . This gives a contradiction for $d\geq 20$ as $M\geq\exp(d/8)$ . For $d=1$ , we can repeat the same arguments with $M=2$ to get the desired lower bound. ∎

Reduction from $\varepsilon$ -DP ERM to $\varepsilon$ -DP SCO

We conclude the proof of the theorem by proving that SCO under privacy constraints is strictly harder than ERM. This is similar to Appendix C in [8] but we require it for pure-DP constraints. We make this formal in here.

We have the following lemma.

Proposition 4.

Let $0<\beta\leq 1/n$ . Assume $\mathsf{A}$ is an $\frac{\varepsilon}{2\log(2/\beta)}$ -DP algorithm that for a sample $\mathcal{S}=(S_{1},\ldots,S_{n})$ with $S_{1}^{n}\stackrel{{\scriptstyle\rm iid}}{{\sim}}P$ achieves with probability $1-\beta/2$ error

f(\mathsf{A}(\mathcal{S}))-\min_{x\in\mathcal{X}}f(x)\leq\gamma.

Then there is an $\varepsilon$ -DP algorithm $\mathsf{A}^{\prime}$ such that for any dataset $\mathcal{S}\in\mathbb{S}^{n}$ has with probability $1-\beta$ ,

f_{\mathcal{S}}(\mathsf{A}^{\prime}(\mathcal{S}))-\min_{x\in\mathcal{X}}f_{\mathcal{S}}(x)\leq\gamma.

Proof.

Given the algorithm $\mathsf{A}$ , we define $\mathsf{A}^{\prime}$ as follows. For an input $\mathcal{S}\in\mathbb{S}^{n}$ , let $P_{\mathcal{S}}$ be the empirical distribution of $\mathcal{S}$ . Then, $\mathsf{A}^{\prime}$ proceeds as follows:

1.

Sample a new dataset $\mathcal{S}_{1}=(S_{1}^{\prime},\dots,S^{\prime}_{n})$ where $S^{\prime}_{i}\sim P_{\mathcal{S}}$
2.

If there is a sample $S_{i}$ that was sampled more than $k=2\log(2/\beta)$ times, return $0$
3.

Else, return $\mathsf{A}(\mathcal{S}_{1})$

We need to prove that $\mathsf{A}^{\prime}$ is $\varepsilon$ -DP and that it has the desired utility. For utility, note that $\mathsf{A}^{\prime}$ returns $0$ at step $2$ with probability at most $\beta/2$ , since we have for every $1\leq i\leq n$

	$\displaystyle\Pr\left(s_{i}\mbox{~{}used more than $k$ times}\right)$	$\displaystyle=\Pr\left(\sum_{j=1}^{n}Z_{i}\geq k\right)$
		$\displaystyle\leq 2^{-k}\leq\beta^{2}/2,$

where $Z_{j}\sim\mathsf{Bernoulli}(p)$ with $p=1/n$ , and the second inequality follows from Chernoff [36, Thm. 4.4] and $\beta\leq 1/10$ . Applying a union bound over all samples, we get that step $2$ returns $0$ with probability at most $\beta/2$ as $\beta\leq 1/n$ . Moreover, Algorithm $\mathsf{A}$ fails with probability at most $\beta/2$ . Therefore, as $f_{\mathcal{S}}(x)=\mathbb{E}_{S\sim P_{\mathcal{S}}}[F(x;S)]$ , we have with probability at least $1-\beta$ ,

f_{\mathcal{S}}(\mathsf{A}^{\prime}(\mathcal{S}))-\min_{x\in\mathcal{X}}f_{\mathcal{S}}(x)\leq\gamma.

Let us now prove privacy. Assume we run algorithm $\mathsf{A}^{\prime}$ on two neighboring datasets $\mathcal{S},\mathcal{S}^{\prime}$ , and let $\mathcal{S}_{1},\mathcal{S}^{\prime}_{1}$ be the datasets produced at step $1$ . Let $B$ denote the event that there was a sample $s_{i}$ that was used more than $k$ times (note that this does not depend on the input). Then for any measurable $\mathcal{O}$ ,

	$\displaystyle\Pr(\mathsf{A}^{\prime}(\mathcal{S})\in\mathcal{O})$	$\displaystyle=\Pr(\mathsf{A}^{\prime}(\mathcal{S})\in\mathcal{O}\mid B)\Pr(B)+\Pr(\mathsf{A}^{\prime}(\mathcal{S})\in\mathcal{O}\mid B^{c})\Pr(B^{c})$
		$\displaystyle\leq e^{\varepsilon}\Pr(\mathsf{A}^{\prime}(\mathcal{S}^{\prime})\in\mathcal{O}\mid B)\Pr(B)+\Pr(\mathsf{A}^{\prime}(\mathcal{S}^{\prime})\in\mathcal{O}\mid B^{c})\Pr(B^{c})$
		$\displaystyle\leq e^{\varepsilon}\Pr(\mathsf{A}^{\prime}(\mathcal{S}^{\prime})\in\mathcal{O}),$

where the first inequality follows from group privacy since $d_{\rm Ham}(\mathcal{S}_{1},\mathcal{S}^{\prime}_{1})\leq k$ and $\mathsf{A}$ is $\varepsilon/k$ -DP. This completes the proof.

∎

C.1.2 Proof of Theorem 5

See 5

Proof.

We follow the same reduction that we used in the proof of Theorem 8. For $\delta\in[0,1/2]$ , we again consider $P_{v}=1$ with probability $\tfrac{1+\delta v}{2}$ and $-1$ otherwise. For $a\in[0,1]$ to be defined later, we construct the following function

F(x;+1)=\begin{cases}\lvert x-a\rvert&\mbox{~{}~{}if~{}~{}}x\leq a\\ \lvert x-a\rvert^{\kappa}&\mbox{~{}~{}if~{}~{}}x\geq a\end{cases}\mbox{~{}~{}and~{}~{}}F(x;-1)=\begin{cases}\lvert x+a\rvert^{\kappa}&\mbox{~{}~{}if~{}~{}}x\leq-a\\ \lvert x+a\rvert&\mbox{~{}~{}if~{}~{}}x\geq-a\end{cases}

Computing the separation. First, let us compute the separation $d_{\mathsf{opt}}(v,v^{\prime},\mathcal{X})$ . We will then choose $a$ to ensure $f_{v}$ has $\kappa$ -growth. By symmetry, assume $v=1$ . $f_{v}$ is increasing on $[a,1]$ and decreasing on $[-1,-a]$ , thus the minimum belongs to $[-a,a]$ and by inspection, is attained at $x=a$ with value $a(1-\delta)$ . Similarly, the minimum of $f_{+1}(x)+f_{-1}(x)$ is attained on $[-a,a]$ with value $2a$ . This yields

d_{\mathsf{opt}}(v,v^{\prime},\mathcal{X})=2a-2a(1-\delta)=2a\delta.

Let us now pick $a$ such that $f_{v}$ has $\kappa$ -growth. Again, by symmetry we only treat the $v=1$ case. We have

\mbox{for~{}}x\geq a,f_{v}(x)-f_{v}^{\star}=\frac{1+\delta}{2}(x-a)^{\kappa}+\frac{1-\delta}{2}(x+a)-a(1-\delta)=\frac{1+\delta}{2}(x-a)^{\kappa}+\frac{1-\delta}{2}(x-a)\geq\lvert x-a\rvert^{\kappa},

where the last inequality is because $(x-a)\leq 1$ and so $(x-a)\geq(x-a)^{\kappa}$ for $\kappa>1$ . In the second case, we have

\mbox{for~{}}x\in[-a,a],f_{v}(x)-f_{v}^{\star}=\delta(a-x).

It holds that $\delta(a-x)\geq(a-x)^{\kappa}$ for all $x\in[-a,a]$ iff $a\leq\tfrac{1}{2}\delta^{\tfrac{1}{\kappa-1}}$ . As a result, we set $a=\tfrac{1}{2}\delta^{\tfrac{1}{\kappa-1}}$ . Finally, for $x\in[-1,-a]$ , we define

h(x)\coloneqq\frac{1+\delta}{2}\lvert x-a\rvert+\frac{1-\delta}{2}\lvert x+a\rvert^{\kappa}-a(1-\delta)-\frac{1}{\kappa}\lvert x-a\rvert^{\kappa}\mbox{~{}~{}for~{}~{}}x\in[-1,-a].

We wish to prove that $h(x)\geq 0$ . First of, note that $h(-a)=\delta^{\tfrac{\kappa}{\kappa-1}}(\tfrac{1}{2}+\tfrac{1}{2}-\tfrac{1}{\kappa})>0$ , whenever $\kappa>1$ . Let us show that $h(x)$ is decreasing on $[-1,-a]$ which suffices to conclude the proof. We have

h^{\prime}(x)=-\frac{1+\delta}{2}-\frac{\kappa(1-\delta)}{2}\lvert x+a\rvert^{\kappa-1}+\lvert x-a\rvert^{\kappa-1}.

First of, note that $h^{\prime}(-a)=-\tfrac{1+\delta}{2}+\delta\leq 0$ and $h^{\prime}(-1)<0$ , thus it suffices to show that if $h^{\prime}$ has an extremum then is it negative. An extremum of this function is a point $x^{\star}$ such that

\lvert a-x^{\star}\rvert=\left(\frac{\kappa(1-\delta)}{2}\right)^{\tfrac{1}{\kappa-2}}\lvert a+x^{\star}\rvert,

which yields that

h^{\prime}(x^{\star})=\lvert a+x^{\star}\rvert^{\kappa-1}\left(\frac{\kappa(1-\delta)}{2}\right)\left[\left(\frac{\kappa(1-\delta)}{2}\right)^{\tfrac{1}{\kappa-2}}-1\right]-\frac{1+\delta}{2}\leq 0,

as $\kappa\leq 2$ . This calculation shows that $f_{v}$ has $(1,\kappa)$ -growth. Finally note that the function is $\kappa\leq 2$ -Lipschitz as desired.

Lower bounding the testing error. It remains to choose the value of $\delta$ . Since we require a lower bound under privacy constraints, in contrast to the one-dimensional section of the proof of Theorem 8, we require the following privatized version of Le Cam’s lemma from [6]

Proposition 5.

[6, Thm. 2] Let $\mathsf{A}\in\mathcal{A}^{\varepsilon}$ be an $\epsilon$ -DP mechanism from $\mathbb{S}^{n}\to\mathcal{X}$ . It holds that

\inf_{\psi:\mathcal{X}\to\{-1,1\}}\inf_{\mathsf{A}\in\mathcal{A}^{\epsilon}}\Pr(\psi(\mathsf{A}(S_{1}^{n}))\neq V)\geq\frac{1}{2}\left(1-\min\left\{2n\varepsilon\|P_{1}-P_{-1}\|_{\rm TV},\|P_{-1}^{n}-P_{1}^{n}\|_{\rm TV}\right\}\right).

With this result, we set $\delta=\max\{1/\sqrt{6n},1/(2\sqrt{3}n\varepsilon)\}$ and lower bound $\max\{a,b\}$ by $a+b$ for readability, which concludes the proof of the theorem. ∎

C.2 Proof for Section 5.2

See 3

Proof of Proposition 3.

Let us first show how to construct the mechanism $\mathsf{A}^{\prime}$ . Let $k\in\mathbb{N}$ be such that $k\geq\log_{2}\left[\frac{\kappa^{\tfrac{1}{\kappa-1}}L^{\tfrac{\kappa}{\kappa-1}}}{2^{2\kappa-3}\Delta(n,L,\varepsilon/k,\delta)}\right]$ and let $\{\lambda_{i}\}_{i\in[k]}$ be a collection of positive scalars. Set $x_{0}\in\mathcal{X}$ , for $i\in\{1,\ldots,k\}$

	$\displaystyle\mbox{define~{}~{}}G_{i}(x;s)=F(x;s)+\frac{\lambda_{i}\cdot 2^{\kappa-2}}{\kappa}\\|x-x_{i-1}\\|_{2}^{\kappa},\mathcal{Y}_{i}\coloneqq\left\{x\in\mathcal{X}:\\|x-x_{i-1}\\|_{2}\leq\left(\frac{L\kappa}{\lambda_{i}2^{\kappa-2}}\right)^{\tfrac{1}{\kappa-1}}\right\}$
	$\displaystyle\mbox{~{}~{}and set~{}~{}}x_{i}=\mathsf{A}(\mathcal{S},G_{i},\mathcal{Y}_{i}),\mbox{~{}with privacy~{}~{}}(\varepsilon/k,\delta/k).$

Finally, define $\mathsf{A}^{\prime}(\mathcal{S})=x_{k}$ . Standard composition theorems [20] guarantee that $\mathsf{A}^{\prime}$ is $(\varepsilon,\delta)$ -DP. Let us analyze its utility; we drop the dependence of $\Delta$ on other variables when clear from context. First of, since $\kappa$ is a constant, note that $G_{i}$ is $c_{0}L$ -Lipschitz with $c_{0}<\infty$ a numerical constant. For simplicity, we define $g_{i}(x)\coloneqq\frac{1}{n}\sum_{s\in\mathcal{S}}G_{i}(x;s)$ and $x_{i}^{\star}=\mathop{\rm argmin}_{x\in\mathcal{Y}_{i}}g_{i}(x)$ . It holds that $g_{i}$ is $(\lambda_{i}2^{\kappa-2},\kappa)$ -uniformly-convex and thus the following growth condition holds

\frac{\lambda_{i}}{\kappa}\mathbb{E}\|x_{i}-x_{i}^{\star}\|_{2}^{\kappa}\leq\mathbb{E}[g_{i}(x_{i})]-g_{i}(x_{i}^{\star})\leq\frac{1}{\lambda_{i}^{\tfrac{1}{\kappa-1}}}\Delta.

Also note that for any point $y\in\mathcal{Y}_{i}$ , it holds that

f_{\mathcal{S}}(x_{i}^{\star})-f(y)\leq\frac{\lambda_{i}2^{\kappa-2}}{\kappa}\|x_{i-1}-y\|_{2}^{\kappa}.

Finally, let us bound the distance to the optimum of $f_{\mathcal{S}}$ at the final iterate. We have

\frac{\lambda_{k}}{\kappa}\|x_{k}-x_{k}^{\star}\|_{2}^{\kappa}\leq g_{k}(x_{k})-g_{k}(x_{k}^{\star})\leq c_{0}L\|x_{k}-x_{k}^{\star}\|_{2}\mbox{~{}~{}which yields~{}~{}}\|x_{k}-x_{k}^{\star}\|_{2}\leq\left(\frac{c_{0}L\kappa}{\lambda_{k}}\right)^{\tfrac{1}{\kappa-1}}.

Let us put the pieces together: for $\lambda>0$ to be determined later and $\nu=\kappa-1$ , set $\lambda_{i}=2^{-\nu i}\lambda$ . After $k$ rounds and denoting $x_{0}^{\star}=\inf_{x\in\mathcal{X}}f_{\mathcal{S}}(x)$ , we have

	$\displaystyle\mathbb{E}[f_{\mathcal{S}}(x_{k})]-f_{\mathcal{S}}(x^{\star})$	$\displaystyle=\sum_{i=1}^{k}\mathbb{E}\left[f_{\mathcal{S}}(x_{i}^{\star})-f_{\mathcal{S}}(x_{i-1}^{\star})\right]+\mathbb{E}\left[f_{\mathcal{S}}(x_{k})-f_{\mathcal{S}}(x_{k}^{\star})\right]$
		$\displaystyle\leq\sum_{i=1}^{k}\frac{\lambda_{i}2^{\kappa-2}}{\kappa}\mathbb{E}\\|x_{i-1}-x_{i-1}^{\star}\\|_{2}^{\kappa}+L\left(\frac{c_{0}L\kappa}{\lambda_{k}}\right)^{\tfrac{1}{\kappa-1}}$
		$\displaystyle\leq\frac{\lambda D^{\kappa}}{\kappa}+\sum_{i=2}^{k}\frac{\lambda_{i}2^{\kappa-2}}{\lambda_{i-1}^{\tfrac{\kappa}{\kappa-1}}}\Delta+L\left(\frac{c_{0}L\kappa}{\lambda_{k}}\right)^{\tfrac{1}{\kappa-1}}$
		$\displaystyle=\frac{\lambda D^{\kappa}}{\kappa}+\frac{\Delta 2^{\kappa-2}}{\lambda^{\tfrac{1}{\kappa-1}}}\sum_{i=2}^{k}2^{-\tfrac{\nu}{\kappa-1}(i-\kappa)}+L\left(\frac{c_{0}L\kappa}{\lambda}\right)^{\tfrac{1}{\kappa-1}}2^{-\tfrac{\nu}{\kappa-1}k}$
		$\displaystyle\leq\frac{\lambda D^{\kappa}}{\kappa}+2^{2\kappa-3}\frac{\Delta}{\lambda^{\tfrac{1}{\kappa-1}}}+\frac{\kappa^{\tfrac{1}{\kappa-1}}(c_{0}L)^{\tfrac{\kappa}{\kappa-1}}2^{-k}}{\lambda^{\tfrac{1}{\kappa-1}}}.$

Finally, note that

k\geq\left\lceil\log_{2}\left[\frac{\kappa^{\tfrac{1}{\kappa-1}}(c_{0}L)^{\tfrac{\kappa}{\kappa-1}}}{2^{2\kappa-3}\Delta}\right]\right\rceil\mbox{~{}~{}so that~{}~{}}\frac{\kappa^{\tfrac{1}{\kappa-1}}(c_{0}L)^{\tfrac{\kappa}{\kappa-1}}2^{-k}}{\lambda^{\tfrac{1}{\kappa-1}}}\leq 2^{2\kappa-3}\frac{\Delta}{\lambda^{\tfrac{1}{\kappa-1}}}.

It then holds that

\mathbb{E}[f_{\mathcal{S}}(x_{k})]-f_{\mathcal{S}}(x^{\star})\leq\lambda\frac{D^{\kappa}}{\kappa}+4^{\kappa-1}\Delta\frac{1}{\lambda^{\tfrac{1}{\kappa-1}}}.

It remains to pick $\lambda$ to minimize the upper bound above. A calculation yields that for $a,b\geq 0$

\inf_{\nu\geq 0}a\nu+\frac{b}{\nu^{\tfrac{1}{\kappa-1}}}=(\kappa-1)^{1/\kappa}a^{1/\kappa}b^{(\kappa-1)/\kappa}\left[\kappa-1+\frac{1}{\kappa-1}\right]\mbox{~{}~{}at~{}~{}}\nu^{\star}=\left(\frac{b}{a(\kappa-1)}\right)^{\tfrac{\kappa-1}{\kappa}}.

Setting $\lambda=4^{\tfrac{(\kappa-1)^{2}}{\kappa}}(\tfrac{\Delta\kappa}{D^{\kappa}(\kappa-1)})^{(\kappa-1)/\kappa}$ yields the regret bound

\mathbb{E}[f_{\mathcal{S}}(x_{k})]-f_{\mathcal{S}}(x^{\star})\leq O(1)D\Delta^{\tfrac{\kappa-1}{\kappa}}.

∎

Proof.

Consider the reduction of Proposition 3. For $c_{1}<\infty$ to be determined later, assume by contradiction that there exists an $(\varepsilon,\delta)$ mechanism such that

\Delta(n,L,\varepsilon,\delta)\leq c_{1}\left(\frac{L\sqrt{d}}{n\varepsilon}\right)^{\tfrac{\kappa}{\kappa-1}}.

Setting $k=\lceil 4\log_{2}(n\varepsilon/\sqrt{d})\log\log_{2}((n\varepsilon/\sqrt{d})^{\kappa/(\kappa-1)})\rceil$ , the condition holds and the result of Proposition 3 guarantees that there exists a numerical constant $c_{2}<\infty$ and a mechanism $\mathsf{A}^{\prime}$ such that

\mathbb{E}[f_{\mathcal{S}}(\mathsf{A}^{\prime}(\mathcal{S}))]-\inf_{x^{\prime}\in\mathcal{X}}f_{\mathcal{S}}(x^{\prime})\leq c_{2}c_{1}^{\tfrac{\kappa-1}{\kappa}}kD\frac{L\sqrt{d}}{n\varepsilon}.

However, Theorem 5.3 in [7] guarantees that there exists $c_{3}>0$ such that for any $(\varepsilon,\delta)$ -DP mechanism $\mathsf{A}^{\prime\prime}$ , it must hold

c_{3}LD\frac{\sqrt{d}}{n\varepsilon}\leq\mathbb{E}[f_{\mathcal{S}}(\mathsf{A}^{\prime\prime}(\mathcal{S}))]-f_{\mathcal{S}}(x^{\star}).

Setting $c_{1}=\tfrac{1}{2}\left(\tfrac{c_{3}}{kc_{2}}\right)^{\tfrac{\kappa}{\kappa-1}}$ yields a contradiction and the desired lower bound by noting that $k$ consists only of log factors. ∎

Adapting to Function Difficulty and Growth Conditions in Private Optimization

Abstract

1 Introduction

1.1 Related work

2 Preliminaries

Notation.

Assumptions.

Assumption 1 (κ\kappa-growth).

Uniform convexity and Kurdyka-Łojasiewicz inequality.

Differential privacy.

Definition 2.1 ([22, 21]).

Lemma 2.1 (Composition [20, Thm. 3.16]).

Lemma 2.2 (Laplace mechanism [20, Thm. 3.6]).

Lemma 2.3 (Gaussian mechanism [20, Thm. A.1]).

Inverse sensitivity mechanism.

3 Adaptive rates through inverse sensitivity for ε\varepsilon-DP

Theorem 1.

4 Efficient algorithms with optimal rates

4.1 High-probability guarantees for convex DP-SCO

Proposition 1.

Proposition 2.

4.2 Algorithms for DP-SCO with growth

Theorem 2.

Sketch of the proof.

Theorem 3.

5 Lower bounds

5.1 Lower bounds for pure ε\varepsilon-DP

κ\kappa-growth with κ≥2\kappa\geq 2.

Theorem 4 (Lower bound for ε\varepsilon-DP, κ≥2\kappa\geq 2).

Sketch of the proof.

κ\kappa-growth with κ∈(1,2]\kappa\in(1,2].

Theorem 5 (Lower bound for ε\varepsilon-DP, κ∈(1,2]\kappa\in{(1,2]} ).

5.2 Lower bounds under approximate privacy constraints

Theorem 6 (Private lower bound for (ε,δ)(\varepsilon,\delta)-DP).

Proposition 3 (Solving ERM with κ\kappa-growth implies solving any convex ERM).

Discussion

Acknowledgments

References

Appendix A Proofs for Section 3

A.1 Proof of Theorem 1

Appendix B Proofs for Section 4

Theorem 7.

B.1 Proof of Proposition 1

B.2 Proof of Proposition 2

B.3 Proofs of Theorems 2 and 3

Lemma B.1.

Proof.

Lemma B.2.

Proof.

Appendix C Proofs of Section 5

C.1 Proofs of Section 5.1

C.1.1 Proof of Theorem 4

Non-private lower bound

Theorem 8 (Non-private lower bound).

Proof.

Privatizing the lower bound via a packing argument

Theorem 9 (Private lower bound for ERM).

Proof.

Reduction from ε\varepsilon-DP ERM to ε\varepsilon-DP SCO

Proposition 4.

Proof.

C.1.2 Proof of Theorem 5

Proof.

Proposition 5.

C.2 Proof for Section 5.2

Proof of Proposition 3.

Proof.

Adapting to Function Difficulty and
Growth Conditions in Private Optimization

Assumption 1 ( $\kappa$ -growth).

3 Adaptive rates through inverse sensitivity for $\varepsilon$ -DP

5.1 Lower bounds for pure $\varepsilon$ -DP

$\kappa$ -growth with $\kappa\geq 2$ .

Theorem 4 (Lower bound for $\varepsilon$ -DP, $\kappa\geq 2$ ).

$\kappa$ -growth with $\kappa\in(1,2]$ .

Theorem 5 (Lower bound for $\varepsilon$ -DP, $\kappa\in{(1,2]}$ ).

Theorem 6 (Private lower bound for $(\varepsilon,\delta)$ -DP).

Proposition 3 (Solving ERM with $\kappa$ -growth implies solving any convex ERM).

Reduction from $\varepsilon$ -DP ERM to $\varepsilon$ -DP SCO