Local Differential Privacy Is Equivalent to Contraction of ${\mathsf{E}}_{\gamma}$-Divergence

A major challenge in modern machine learning applications is balancing statistical efficiency with the privacy of individuals from whom data is obtained. In such applications, privacy is often quantified in terms of Differential Privacy (DP) [1]. DP has several variants, including approximate DP [2], Rényi DP [3], and others [4, 5, 6, 7]. Arguably, the most stringent flavor of DP is local differential privacy (LDP) [8, 9]. Intuitively, a randomized mechanism (or a Markov kernel) is said to be locally differentially private if its output does not vary significantly with arbitrary perturbations of the input.

More precisely, a mechanism is said to be $\varepsilon$-LDP (or pure LDP) if the privacy loss random variable, defined as the log-likelihood ratio of the output for any two different inputs, is smaller than $\varepsilon$ with probability one. One can also consider an approximate variant of this constraint: ${\mathsf{K}}$ is said to be $(\varepsilon,\delta)$-LDP if the privacy loss random variable does not exceed $\varepsilon$ with probability at least $1-\delta$ (see Def. 1 for the formal definition).

The study of statistical efficiency under LDP constraints has gained considerable traction, e.g., [9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 8]. Almost all of these works consider $\varepsilon$-LDP and provide meaningful bounds only for sufficiently small values of $\varepsilon$ (i.e., the high privacy regime). For instance, Duchi et al. [10] studied minimax estimation problems under $\varepsilon$-LDP constraints and showed that for $\varepsilon\leq 1$, the price of privacy is to reduce the effective sample size from $n$ to $\varepsilon^{2}n$. A slightly improved version of this result appeared in [19, 13]. More recently, Duchi and Rogers [20] developed a framework based on the strong data processing inequality (SDPI) [21] and derived lower bounds for minimax estimation risk under $\varepsilon$-LDP that hold for any $\varepsilon\geq 0$.

In this work, we develop an SDPI-based framework for studying hypothesis testing and estimation problems under $(\varepsilon,\delta)$-LDP, extending the results of [20] to approximate LDP. In particular, we derive bounds for both the minimax and Bayesian estimation risks that hold for any $\varepsilon\geq 0$ and $\delta\geq 0$. Interestingly, when setting $\delta=0$, our bounds can be slightly stronger than [10].

Our main mathematical tool is an equivalent expression for DP in terms of ${\mathsf{E}}_{\gamma}$-divergence. Given $\gamma\geq 1$, the ${\mathsf{E}}_{\gamma}$-divergence between two distributions $P$ and $Q$ is defined as

$${\mathsf{E}}_{\gamma}(P\|Q)\coloneqq\frac{1}{2}\int|\text{d}P-\gamma\text{d}Q|-\frac{1}{2}(\gamma-1).$$

(1)

We show that a mechanism ${\mathsf{K}}$ is $(\varepsilon,\delta)$-LDP if and only if

$${\mathsf{E}}_{\gamma}(P{\mathsf{K}}\|Q{\mathsf{K}})\leq\delta{\mathsf{E}}_{\gamma}(P\|Q)$$

for $\gamma=e^{\varepsilon}$ and any pairs of distributions $(P,Q)$ where $P{\mathsf{K}}$ represents the output distribution of ${\mathsf{K}}$ when the input distribution is $P$. Thus, the approximate LDP guarantee of a mechanism can be fully characterized by its contraction under ${\mathsf{E}}_{\gamma}$-divergence. When combined with standard statistical techniques, including Le Cam’s and Fano’s methods [22, 23], ${\mathsf{E}}_{\gamma}$-contraction leads to general lower bounds for the minimax and Bayesian risk under $(\varepsilon,\delta)$-LDP for any $\varepsilon\geq 0$ and $\delta\in[0,1]$. In particular, we show that the price of privacy in this case is to reduce the sample size from $n$ to $n[1-e^{-\varepsilon}(1-\delta)]$.

There exists several results connecting pure LDP to the contraction properties of KL divergence $D_{\mathsf{KL}}$ and total variation distance ${\mathsf{TV}}$. For instance, for any $\varepsilon$-LDP mechanism ${\mathsf{K}}$, it is shown in [10, Theorem 1] that $D_{\mathsf{KL}}(P{\mathsf{K}}\|Q{\mathsf{K}})\leq 2(e^{\varepsilon}-1)^{2}{\mathsf{TV}}^{2}(P,Q)$ and in [13, Theorem 6] that ${\mathsf{TV}}(P{\mathsf{K}}\|Q{\mathsf{K}})\leq\frac{e^{\varepsilon}-1}{e^{\varepsilon}+1}{\mathsf{TV}}(P,Q)$ for any pairs $(P,Q)$. Inspired by these results, we further show that if ${\mathsf{K}}$ is $(\varepsilon,\delta)$-LDP then $D_{f}(P{\mathsf{K}}\|Q{\mathsf{K}})\leq[1-e^{-\varepsilon}(1-\delta)]D_{f}(P\|Q)$ for any arbitrary $f$-divergences $D_{f}$ and any pairs $(P,Q)$.

We show next that the $(\varepsilon,\delta)$-LDP constraint, with $\delta$ not necessarily equal to zero, is equivalent to the contraction of ${\mathsf{E}}_{\gamma}$-divergence.

We note that Duchi et al. [10] showed that if ${\mathsf{K}}$ is $\varepsilon$-LDP then $D_{{\mathsf{KL}}}(P{\mathsf{K}}\|Q{\mathsf{K}})\leq 2(e^{\varepsilon}-1)^{2}{\mathsf{TV}}^{2}(P,Q)$. They then informally concluded from this result that $\varepsilon$-LDP acts as a contraction on the space of probability measures. Theorem 1 makes this observation precise.

According to Theorem 1, a mechanism ${\mathsf{K}}$ is $\varepsilon$-LDP if and only if ${\mathsf{E}}_{e^{\varepsilon}}(P{\mathsf{K}}\|Q{\mathsf{K}})=0$ for any distributions $P$ and $Q$. An example of such Markov kernels is given next.

${\mathsf{E}}_{\gamma}$-divergence underlies all other $f$-divergences, in a sense that any arbitrary $f$-divergence can be represented by ${\mathsf{E}}_{\gamma}$-divergence [42, Corollary 3.7]. Thus, an LDP constraint implies that a Markov kernel contracts for all $f$-divergences, in a similar spirit to ${\mathsf{E}}_{\gamma}$-contraction in Theorem 1.

Notice that this lemma holds for any $f$-divergences and any general family of $(\varepsilon,\delta)$-LDP mechanisms. However, it can be improved if one considers particular mechanisms or a certain $f$-divergence. For instance, it is known that $\eta_{\mathsf{KL}}(\mathsf{BSC}(\omega))=(1-2\omega^{2})$ [21]. Thus, we have $\eta_{\mathsf{KL}}({\mathsf{K}}^{\varepsilon}_{\mathsf{RR}})=(\frac{e^{\varepsilon}-1}{e^{\varepsilon}+1})^{2}$ for the randomized response mechanism ${\mathsf{K}}^{\varepsilon}_{\mathsf{RR}}$ (cf. Example 1), while Lemma 1 implies that $\eta_{\mathsf{KL}}({\mathsf{K}}^{\varepsilon}_{\mathsf{RR}})\leq 1-e^{-\varepsilon}$. Unfortunately, $\eta_{\mathsf{KL}}$ is difficult to compute in closed form for general Markov kernels, in which case Lemma 1 provides a useful alternative.

Next, we extend Lemma 1 for the non-interactive mechanism. Fix an $(\varepsilon,\delta)$-LDP mechanism ${\mathsf{K}}$ and consider the corresponding non-interactive mechanism ${\mathsf{K}}^{\otimes n}$. To obtain upper bounds on $\eta_{f}({\mathsf{K}}^{\otimes n})$ directly through Lemma 1, we would first need to derive privacy parameters of ${\mathsf{K}}^{\otimes n}$ in terms of $\varepsilon$ and $\delta$ (e.g., by applying composition theorems). Instead, we can use the tensorization properties of contraction coefficients (see, e.g., [39, 38]) to relate $\eta_{f}({\mathsf{K}}^{\otimes n})$ to $\eta_{f}({\mathsf{K}})$ and then apply Lemma 1, as described next.

Each of the next three sections provide a different application of the contraction characterization of LDP.

Let $X^{n}=(X_{1},\dots,X_{n})$ be $n$ independent and identically distributed (i.i.d.) samples drawn from a distribution $P$ in a family ${\mathcal{P}}\subseteq{\mathcal{P}}({\mathcal{X}})$. Let also $\theta:{\mathcal{P}}\to{\mathcal{T}}$ be a parameter of a distribution that we wish to estimate. Each user has a sample $X_{i}$ and applies a privacy-preserving mechanism ${\mathsf{K}}_{i}$ to obtain $Z_{i}$. Generally, we can assume that ${\mathsf{K}}_{i}$ are sequentially interactive. Given the sequences $\{Z_{i}\}_{i=1}^{n}$, the goal is to estimate $\theta(P)$ through an estimator $\Psi:{\mathcal{Z}}^{n}\to{\mathcal{T}}$. The quality of such estimator is assessed by a semi-metric $\ell:{\mathcal{T}}\times{\mathcal{T}}\to\mathbb{R}_{+}$ and is used to define the minimax risk as:

$$\mathcal{R}_{n}({\mathcal{P}},\ell,\varepsilon,\delta)\coloneqq\inf_{{\mathsf{K}}^{n}\subset{\mathcal{Q}}_{\varepsilon,\delta}}\inf_{\Psi}\sup_{P\in{\mathcal{P}}}{\mathbb{E}}[\ell(\Psi(Z^{n}),\theta(P))].$$

(5)

The quantity $R_{n}({\mathcal{P}},\ell,\varepsilon,\delta)$ uniformly characterizes the optimal rate of private statistical estimation over the family ${\mathcal{P}}$ using the best possible estimator and privacy-preserving mechanisms in ${\mathcal{Q}}_{\varepsilon,\delta}$. In the absence of privacy constraints (i.e., $Z^{n}=X^{n}$), we denote the minimax risk by $\mathcal{R}_{n}({\mathcal{P}},\ell)$.

The first step in deriving information-theoretic lower bounds for minimax risk is to reduce the above estimation problem to a testing problem [23, 43, 22]. To do so, we need to construct an index set ${\mathcal{V}}$ with $|{\mathcal{V}}|<\infty$ and a family of distributions $\{P_{v},v\in{\mathcal{V}}\}\subseteq{\mathcal{P}}$ such that $\ell(\theta(P_{v}),\theta(P_{v^{\prime}}))\geq 2\tau$ for all $v\neq v^{\prime}$ in ${\mathcal{V}}$ for some $\tau>0$. The canonical testing problem is then defined as follows: Nature chooses a random variable $V$ uniformly at random from ${\mathcal{V}}$, and then conditioned on $V=v$, the samples $X^{n}$ are drawn i.i.d. from $P_{v}$, denoted by $X^{n}\sim P^{\otimes n}_{v}$. Each $X_{i}$ is then fed to a mechanism ${\mathsf{K}}_{i}$ to generate $Z_{i}$. It is well-known [22, 43, 23] that $\mathcal{R}_{n}({\mathcal{P}},\ell)\geq\tau\mathsf{P}_{\mathsf{e}}(V|X^{n})$, where $\mathsf{P}_{\mathsf{e}}(V|X^{n})$ denotes the probability of error in guessing $V$ given $X^{n}$. Replacing $X^{n}$ by its $(\varepsilon,\delta)$-privatized samples $Z^{n}$ in this result, one can obtain a lower bound on $R_{n}({\mathcal{P}},\ell,\varepsilon,\delta)$ in terms of $\mathsf{P}_{\mathsf{e}}(V|Z^{n})$. Hence, the remaining challenge is to lower-bound $\mathsf{P}_{\mathsf{e}}(V|Z^{n})$ over the choice of mechanisms $\{{\mathsf{K}}_{i}\}$. There are numerous techniques for this objective depending on ${\mathcal{V}}$. We focus on two such approaches, namely Le Cam’s and Fano’s method, that bound $\mathsf{P}_{\mathsf{e}}(V|Z^{n})$ in terms of total variation distance and mutual information and hence allow us to invoke Lemmas 1 and 2.

In the minimax setting, the worst-case parameter is considered which usually leads to over-pessimistic bounds. In practice, the parameter that incurs a worst-case risk may appear with very small probability. To capture this prior knowledge, it is reasonable to assume that the true parameter is sampled from an underlying prior distribution. In this case, we are interested in the Bayes risk of the problem.

Let ${\mathcal{P}}=\{P_{X|\Theta}(\cdot|\theta):\theta\in{\mathcal{T}}\}$ be a collection of parametric probability distributions on ${\mathcal{X}}$ and the parameter space ${\mathcal{T}}$ is endowed with a prior $P_{\Theta}$, i.e., $\Theta\sim P_{\Theta}$. Given an i.i.d. sequence $X^{n}$ drawn from $P_{X|\Theta}$, the goal is to estimate $\Theta$ from a privatized sequence $Z^{n}$ via an estimator $\Psi:{\mathcal{Z}}^{n}\to{\mathcal{T}}$. Here, we focus on the non-interactive setting. Define the private Bayes risk as

$$R_{n}^{\mathsf{Bayes}}(P_{\Theta},\ell,\varepsilon,\delta)\coloneqq\inf_{{\mathsf{K}}\in{\mathcal{Q}}_{\varepsilon,\delta}}\inf_{\Psi}{\mathbb{E}}[\ell(\Theta,\Psi(Z^{n}))],$$

(13)

where the expectation is taken with respect to the randomness of both $\Theta$ and $Z^{n}$. It is evident that $R_{n}^{\mathsf{Bayes}}(P_{\Theta},\ell,\varepsilon,\delta)$ must depend on the prior $P_{\Theta}$. This dependence can be quantified by

$${\mathcal{L}}(\zeta)\coloneqq\sup_{t\in{\mathcal{T}}}\Pr(\ell(\Theta,t)\leq\zeta),$$

(14)

for $\zeta<\sup_{\theta,\theta^{\prime}\in{\mathcal{T}}}\ell(\theta,\theta^{\prime})$. Xu and Raginsky [44] showed that the non-private Bayes risk (i.e., $X^{n}=Z^{n}$), denoted by $R_{n}^{\mathsf{Bayes}}(P_{\Theta},\ell)$, is lower bounded as

$$R_{n}^{\mathsf{Bayes}}(P_{\Theta},\ell)\geq\sup_{\zeta>0}\zeta\left[1-\frac{I(\Theta;X^{n})+\log 2}{\log(1/{\mathcal{L}}(\zeta))}\right].$$

(15)

Replacing $I(\Theta;X^{n})$ with $I(\Theta;Z^{n})$ in this result and applying Lemma 2 (similar to Lemma 4), we can directly convert (15) to a lower bound for $R_{n}^{\mathsf{Bayes}}(P_{\Theta},\ell,\varepsilon,\delta)$.

In the following theorem, we provide a lower bound for $R_{n}^{\mathsf{Bayes}}(P_{\Theta},\ell,\varepsilon,\delta)$ that directly involves ${\mathsf{E}}_{\gamma}$-divergence, and thus leads to a tighter bounds than (3). For any pair of random variables $(A,B)\sim P_{AB}$ with marginals $P_{A}$ and $P_{B}$ and a constant $\gamma\geq 0$, we define their $E_{\gamma}$-information as

$$I_{\gamma}(A;B)\coloneqq{\mathsf{E}}_{\gamma}(P_{AB}\|P_{A}P_{B}).$$

We compare Theorem 2 with Corollary 3 in the next example.

We now turn our attention to the well-known problem of binary hypothesis testing under local differential privacy constraint. Suppose $n$ i.i.d. samples $X^{n}$ drawn from a distribution $Q\in{\mathcal{P}}({\mathcal{X}})$ are observed. Let now each $X_{i}$ be mapped to $Z_{i}$ via a mechanism ${\mathsf{K}}_{i}\in{\mathcal{Q}}_{\varepsilon,\delta}$ (i.e., sequential interaction is permitted). The goal is to distinguish between the null hypothesis $H_{0}:Q=P_{0}$ from the alternative $H_{1}:Q=P_{1}$ given $Z^{n}$. Let $T$ be a binary statistic, generated from a randomized decision rule $P_{T|Z^{n}}:{\mathcal{Z}}^{n}\to\mathcal{P}(\{0,1\})$ where $1$ indicates that $H_{0}$ is rejected. Type I and type II error probabilities corresponding to this statistic are given by $\Pr(T=1|H_{0})$ and $\Pr(T=1|H_{1})$, respectively. To capture the optimal trade-off between type I and type II error probabilities, it is customary to define $\beta^{\varepsilon,\delta}_{n}(\alpha)\coloneqq\inf\Pr(T=0|H_{1})$ where the infimum is taken over all kernels $P_{T|Z^{n}}$ such that $\Pr(T=1|H_{0})\leq\alpha$ and non-interactive mechanisms ${\mathsf{K}}^{\otimes n}$ with ${\mathsf{K}}\in{\mathcal{Q}}_{\varepsilon,\delta}$. In the following lemma, we apply Lemma 1 to obtain an asymptotic lower bound for $\beta_{n}^{\varepsilon,\delta}(\alpha)$.

A similar result was proved by Kairouz et al. [13, Sec. 3] that holds only for sufficiently “small” (albeit unspecified) $\varepsilon$ and $\delta=0$. When compared to Chernoff-Stein lemma [45, Theorem 11.8.3], establishing $D_{\mathsf{KL}}(P_{0}\|P_{1})$ as the asymptotic exponential decay rate of $\beta_{n}^{\varepsilon,\delta}(\alpha)$, the above corollary, once again, justifies the reduction of effective sample size from $n$ to $\varphi(\varepsilon,\delta)n$ in the presence of $(\varepsilon,\delta)$-LDP requirement.

Viewing mutual information as a utility measure, we may consider maximizing mutual information under local differential privacy as yet another privacy-utility trade-off. To formalize this, let $X\sim P_{X}$. The goal is to characterize the supremum of $I(X;Z)$ over ${\mathsf{K}}\in{\mathcal{Q}}_{\varepsilon,\delta}$, i.e., the maximum information shared between $X$ and its $(\varepsilon,\delta)$-LDP representation. Such mutual information bounds under local DP have appeared in the literature, e.g., McGregor et al. [46] provided a result that roughly states $I(X;Z)\leq 3\varepsilon$ for ${\mathsf{K}}\in{\mathcal{Q}}_{\varepsilon}$ and Kairouz et al. [13, Corollary 15] showed for sufficiently small $\varepsilon$

$$\sup_{{\mathsf{K}}\in{\mathcal{Q}}_{\varepsilon}}I(X;Z)\leq\frac{1}{2}P_{X}(A)(1-P_{X}(A))\varepsilon^{2},$$

(20)

where $A\subset{\mathcal{X}}$ satisfies $A\in\mathop{\rm arg\,min}_{B\subset{\mathcal{X}}}|P_{X}(B)-\frac{1}{2}|$. Next, we provide an upper bound for the mutual information under LDP that holds for all $\varepsilon\geq 0$ and $\delta\in[0,1]$.