Pure-DP Aggregation in the Shuffle Model:
Error-Optimal and Communication-Efficient

Differential privacy (DP) [DMNS06] is a widely accepted notion used for bounding and quantifying an algorithm’s leakage of personal information. Its most basic form, known as pure-DP, is governed by a single parameter $\epsilon>0$, which bounds the leakage of the algorithm. Specifically, a randomized algorithm $A(\cdot)$ is said to be $\epsilon$-DP if for any subset $S$ of output values, and for any two datasets $D$ and $D^{\prime}$ differing on a single user’s data, it holds that $\Pr[A(D)\in S]\leq e^{\epsilon}\cdot\Pr[A(D^{\prime})\in S]$. In settings where pure-DP is not (known to be) possible, a common relaxation is the so-called approximate $(\epsilon,\delta)$-DP [DKM⁺06], which has an additional parameter $\delta\in[0,1]$. In this case, the condition becomes: $\Pr[A(D)\in S]\leq e^{\epsilon}\cdot\Pr[A(D^{\prime})\in S]+\delta$.

Depending on the trust assumptions, three models of DP are commonly studied. The first is the central model, where a trusted curator is assumed to hold the raw data and required to release a private output. (This goes back to the first work [DMNS06] on DP.) The second is the local model [EGS03, DMNS06, KLN⁺08], where each user’s message is required to be private. The third is the shuffle model [BEM⁺17, CSU⁺19, EFM⁺19], where the users’ messages are routed through a trusted shuffler, which is assumed to be non-colluding with the curator, and which is expected to randomly permute the messages incoming from the different users ($\mathrm{DP}_{\mathrm{shuffle}}$). More formally, a protocol $P=(R,S,A)$ in the shuffle model consists of $3$ procedures: (i) a local randomizer $R(\cdot)$ that takes as input the data of a single user and outputs one or more messages, (ii) a shuffler $S(\cdot)$ that randomly permutes the messages from all the local randomizers, and (iii) an analyst $A(\cdot)$ that consumes the permuted output of the shuffler; the output of the protocol $P$ is the output of the analyst $A(\cdot)$. Privacy in the shuffle model is defined as follows:

For several analytics tasks, low-error algorithms are known in the central model, whereas they are known to be impossible in the local model. For such tasks, low-error algorithms are commonly sought in the shuffle model, since it is more preferable to trust a shuffler than a central curator.

For a discrete distribution $\mathcal{D}$, let $f_{\mathcal{D}}$ denote its probability mass function (PMF). The max-divergence between distributions $\mathcal{D}_{1},\mathcal{D}_{2}$ is defined as $d_{\infty}(\mathcal{D}_{1}\|\mathcal{D}_{2}):=\max_{x\in\operatorname{supp}(\mathcal{D}_{1})}\ln(f_{\mathcal{D}_{1}}(x)/f_{\mathcal{D}_{2}}(x))$.

For two distributions $\mathcal{D}_{1},\mathcal{D}_{2}$ over $\mathbb{Z}^{d}$, we write $\mathcal{D}_{1}*\mathcal{D}_{2}$ to denote its convolution, i.e., the distribution of $z_{1}+z_{2}$ where $z_{1}\sim\mathcal{D}_{1},z_{2}\sim\mathcal{D}_{2}$ are independent. Moreover, let $(\mathcal{D})^{*n}$ denote the $n$-fold convolution of $\mathcal{D}$, i.e., the distribution of $z_{1}+\cdots+z_{n}$ where $z_{1},\dots,z_{n}\sim\mathcal{D}$ are independent. We write $\mathcal{D}\otimes\mathcal{D}^{\prime}$ to denote the product distribution of $\mathcal{D}_{1},\mathcal{D}_{2}$. Furthermore, we may write a value to denote the distribution all of whose probability mass is at that value (e.g., 0 stands for the probability distribution that is always equal to zero).

A distribution $\mathcal{D}$ is infinitely divisible iff, for every positive integer $n$, there exists a distribution $\mathcal{D}_{/n}$ such that $(\mathcal{D}_{/n})^{*n}$. Two distributions we will use here (both supported on $\mathbb{Z}_{\geq 0}$) are:

• •

  Poisson Distribution $\operatorname{Poi}(\lambda)$: This is the distribution whose PMF is $f_{\operatorname{Poi}(\lambda)}(k)=\lambda^{k}e^{-\lambda}/k!$. It satisfies $\operatorname{Poi}(\lambda)_{/n}=\operatorname{Poi}(\lambda/n)$.

• •

  Negative Binomial Distribution $\operatorname{NB}(r,p)$: This is the distribution whose PMF is $f_{\operatorname{NB}(r,p)}(k)=\binom{k+r-1}{k}p^{r}(1-p)^{k}$. It satisfies $\operatorname{NB}(r,p)_{/n}=\operatorname{NB}(r/n,p)$.

  • –

    Geometric Distribution $\operatorname{Geo}(p)$: A special case of the $\operatorname{NB}$ distribution is the geometric distribution $\operatorname{Geo}(p)=\operatorname{NB}(1,p)$, i.e., one with $f_{\operatorname{Geo}(p)}(k)=p(1-p)^{k}$.

Finally, we recall that the discrete Laplace distribution $\operatorname{DLap}(a)$ is a distribution supported on $\mathbb{Z}$ with PMF $f_{\operatorname{DLap}(a)}(x)\propto\exp\left(-a|x|\right)$. It is well known that $\operatorname{DLap}(a)$ is the distribution of $z_{1}-z_{2}$ where $z_{1},z_{2}\sim\operatorname{Geo}(1-\exp(-a))$ are independent. Furthermore, the variance of the discrete Laplace distribution is $\operatorname{Var}(\operatorname{DLap}(a))=\frac{2e^{-a}}{(1-e^{-a})^{2}}$.

We will also use the following well-known lemma¹¹1This can be viewed as a special case of the post-processing property of DP where the post-processing function is adding a random variable drawn from $\mathcal{D}_{3}$. Another way to see that this holds is to simply observe that, for any $y\in\operatorname{supp}(\mathcal{D}_{1}*\mathcal{D}_{2})$, we have $f_{\mathcal{D}_{1}*\mathcal{D}_{3}}(y)=\sum_{z\in\operatorname{supp}(\mathcal{D}_{3})}f_{\mathcal{D}_{3}}(z)\cdot f_{\mathcal{D}_{1}}(y-z)\leq\sum_{z\in\operatorname{supp}(\mathcal{D}_{3})}f_{\mathcal{D}_{3}}(z)\cdot\left(e^{d_{\infty}(\mathcal{D}_{1}\|\mathcal{D}_{2})}\cdot f_{\mathcal{D}_{2}}(y-z)\right)=e^{d_{\infty}(\mathcal{D}_{1}\|\mathcal{D}_{2})}f_{\mathcal{D}_{2}*\mathcal{D}_{3}}(y)$.:

In this section, we will describe a pure-$\mathrm{DP}_{\mathrm{shuffle}}$ algorithm for counting, which is our main result.

By setting $\rho$ arbitrarily close to zero, we can get the mean square error (MSE) to be arbitrarily close to that of the discrete Laplace mechanism, which is known to be (asymptotically) optimal in the central model [GRS09, GV16]. We can get this guarantee for other type of errors, e.g., $\ell_{1}$-error (aka expected absolute error) as well, but for ease of presentation, we only focus on MSE.

Note that Theorem 6 implies Theorem 2 by simply setting $\rho$ to be a positive constant (say, $0.5$).

In this section we analyze the privacy, utility, and communication guarantees of our counting protocol. Throughout the remainder of this section, we assume the distributions and parameters are set as in 7; for brevity, we will not state this assumption in our privacy analysis.

In this work, we have provided pure-$\mathrm{DP}_{\mathrm{shuffle}}$ algorithms that achieve nearly optimal errors for bit summation, real summation, and histogram while significantly improving on the communication complexity compared to the state-of-the-art. Despite this, there are still a number of interesting open questions, some of which we highlight below.

• •

  Protocol with a bounded number of messages. As mentioned briefly in Section 1.1, our protocol can result in an arbitrarily large number of messages per user, although the expected number is quite small. (In fact, the distribution of the number of messages enjoys a strong exponential tail bound.) Is it possible to design a pure-$\mathrm{DP}_{\mathrm{shuffle}}$ protocol where the maximum number of messages is $O\left(\frac{\log n}{\epsilon}\right)$ for binary summation?

  For this question, we note that a rather natural approach is to modify our protocol to make its number of messages bounded. Namely, we replace $\mathcal{D}^{\mathrm{noise}}_{/n}$ and $\mathcal{D}^{\mathrm{flood}}_{/n}$ by a truncated version of their respective distributions. It turns out that the latter is relatively simple (e.g., even replacing it with a Bernoulli distribution also works) because we only require a mild condition in Lemma 10 to hold. On the other hand, for the former, we are using Lemma 9 which only holds for unbounded distributions. We would like to stress that we do not know whether replacing $\mathcal{D}^{\mathrm{noise}}_{/n}$ with a truncated version of the negative binomial distribution with a “symmetrized” the input dependent part²²2This means that w.p. $q$ we output $s$ copies of both $+1$ and $-1$ messages, for both $x_{i}=0$ and $x_{i}=1$ cases. Without this change, the supports of the two cases are not the same and thus it obviously violates pure-DP. violates pure-DP; however, we do not know how to prove that it satisfies pure-DP either, as the probability mass function of their convolutions become somewhat unwieldy.

• •

  Lower bounds on the expected number of messages. Recall that the communication lower bound from [GGK⁺20] only applies to the maximum number of messages sent. Is it possible to prove a communication lower bound on the expected number of messages (even if the maximum number of messages is unbounded)? We note that the techniques from [GGK⁺20] does not apply.

• •

  Histogram protocol for large $B$. Our protocol has communication complexity that grows linearly with $B$, which becomes impractical when $B$ is large. Can we get protocol for histogram whose communication is $O_{\epsilon}\left((\log n)^{O(1)}\right)$ for $B=O(n)$ (while achieving nearly optimal errors)? For approximate-$\mathrm{DP}_{\mathrm{shuffle}}$, a histogram protocol with expected communication of $1+O_{\epsilon}\left(\frac{B(\log(n/\delta)^{O(1)})}{n}\right)$ is known [GKMP20]. It would be interesting to understand if extending such a protocol to the pure-$\mathrm{DP}_{\mathrm{shuffle}}$ setting is possible.

More generally, despite the (by-now) vast literature on the shuffle model, most work have focused attention on approximate-$\mathrm{DP}_{\mathrm{shuffle}}$. It would be interesting to expand the existing study to pure-$\mathrm{DP}_{\mathrm{shuffle}}$ as well.