Optimized Tradeoffs for Private Prediction with Majority Ensembling

We give a (perhaps surprising) affirmative answer to the above question by using our novel data-dependent randomized response framework (DaRRM), which captures all private majority algorithms, we introduce a tractable noise optimization procedure that maximizes the privacy-utility tradeoffs. Furthermore, we can provably achieve a constant factor improvement in utility over simple subsampling by applying data-dependent noise injection when $M_{i}$’s are i.i.d. and $\delta=0$. To our knowledge, this is the first of its work of its kind that gives a tractable utility optimization over the possibly infinite set of privacy constraints.

Data-dependent Randomized Response Majority (DaRRM). We generalize the classical Randomized Response (RR) mechanism and the commonly used subsampling baseline for solving Problem 1.1 and propose a general randomized response framework DaRRM (see Algorithm 1), which comes with a customizable noise function $\gamma$. We show that DaRRM actually captures all algorithms computing the majority whose outputs are at least as good as a random guess (see Lemma 3.3), by choosing different $\gamma$ functions.

Designing $\gamma$ with Provable Privacy Amplification. The choice of the $\gamma$ function in DaRRM allows us to explicitly optimize noise while trading off privacy and utility. Using structural observations, we show privacy amplification by a factor of 2 under mild conditions over applying simple composition in the pure differential privacy setting when the mechanisms $M_{i}$’s are i.i.d. (see Theorem 4.1).

Finding the Best $\gamma$ through Dimension-Reduced Optimization. We further exploit the generality of DaRRM by applying a novel optimization-based approach that applies constrained optimization to find a data-dependent $\gamma$ that maximizes some measure of utility. One challenge is that there are infinitely many privacy constraints, which are necessary for DaRRM with the optimized $\gamma$ to satisfy the given privacy loss. We show that we can reformulate the privacy constraints, which are infinite dimensional, to a finite polynomial-sized constraint set, allowing us to efficiently constrain the optimization problem to find the best $\gamma$, even for approximate differential privacy (see Lemma 5.1). Empirically, we show that with a small $m$ and ${\epsilon}$, the optimized $\gamma$ (see $\gamma_{opt}$ in Figure 2) achieves the best utility among all $\gamma$ functions, even compared to the subsampling and the data-independent baseline. To our knowledge, this is the first utility maximization algorithm that optimizes over all private algorithms by constrained optimization with dimension reduction.

Experiments. In downstream tasks, such as semi-supervised knowledge transfer for private image classification, we compare our DaRRM with an optimized $\gamma$ to compute the private label majority from private teachers against PATE Papernot et al. (2018), which computes the private label majority from non-private teachers. We fix the privacy loss of the output of both algorithms to be the same and find that when the number of teachers $K$ is small, DaRRM indeed has a higher utility than PATE, achieving 10%-15% and 30% higher accuracy on datasets MNIST and Fashion-MNIST, respectively.

Private Composition. Blackbox privacy composition analysis often leads to pessimistic utility guarantees. In the blackbox composition setting, one can do no better than the $O(K\epsilon)$ privacy analysis for pure differential privacy Dwork et al. (2014). For approximate differential privacy, previous work has found optimal constants for advanced composition by reducing to the binary case of hypothesis testing with randomized response; and optimal tradeoffs between $\epsilon,\delta$ for black box composition are given in Kairouz et al. (2015), where there could be a modest improvement $20\%$.

Thus, for specific applications, previous work has turned to white-box composition analysis for improved utility. This includes, for example, moment accountant for private SGD Abadi et al. (2016) and the application of contractive maps in stochastic convex optimization Feldman et al. (2018). For the specific case of model ensembles, Papernot et al. (2018) shows a data-dependent privacy bound that vanishes as the probability of disagreement goes to $0$. Their method provides no utility analysis but they empirically observed less privacy loss when there is greater ensemble agreement.

When $g$ is the maximization function, some previous work shows that an approximately maximum value can be outputted with high probability while incurring $O(\epsilon)$ privacy loss, independently of $K$. Liu & Talwar (2019) proposed a random stopping mechanism for $m=1$ that draws samples uniformly at random from $M_{i}({\mathcal{D}})$ at each iteration. In any given iteration, the sampling halts with probability $\gamma$ and the final output is computed based on the samples collected until that time. This leads to a final privacy cost of only $3{\epsilon}$ for the maximization function $g$, which can be improved to $2{\epsilon}$ (Papernot & Steinke, 2022). In addition to the aforementioned works, composing top-k and exponential mechanisms also enjoy slightly improved composition analysis via a bounded-range analysis Durfee & Rogers (2019); Dong et al. (2020).

Bypassing the Global Sensitivity. To ensure differential privacy, it is usually assumed the query function $g$ has bounded global sensitivity — that is, the output of $g$ does not change much on any adjacent input datasets differing in one entry. The noise added to the output is then proportional to the global sensitivity of $g$. If the sensitivity is large, the output utility will thus be terrible due to a large amount of noises added. However, the worst case global sensitivity can be rare in practice, and this observation has inspired a line of works on designing private algorithms with data-dependent sensitivity bound to reduce the amount of noises added.

Instead of using the maximum global sensitivity of $g$ on any dataset, the classical Propose-Test-Release framework of Dwork Dwork & Lei (2009) uses a local sensitivity value for robust queries that is tested privately and if the sensitivity value is too large, the mechanism is halted before the query release. The halting mechanism incurs some failure probability but deals with the worst-case sensitivity situations, while allowing for lower noise injection in most average-case cases.

One popular way to estimate average-case sensitivity is to use the Subsample-and-Aggregate framework by introducing the notion of perturbation stability, also known as local sensitivity of a function $g$ on a dataset ${\mathcal{D}}$ Thakurta & Smith (2013); Dwork et al. (2014), which represents the minimum number of entries in ${\mathcal{D}}$ needs to be changed to change $g({\mathcal{D}})$. One related concept is smooth sensitivity, a measure of variability of $g$ in the neighborhood of each dataset instance. To apply the framework under smooth sensitivity, one needs to privately estimate a function’s local sensitivity $L_{s}$ and adapt noise injection to be order of $O(\frac{L_{s}}{{\epsilon}})$, where $L_{s}$ can often be as small as $O(e^{-n})$, where $n=|\mathcal{D}|$, the total dataset size Nissim et al. (2007). Generally, the private computation of the smooth sensitivity of a blackbox function is nontrivial but is aided by the Subsample and Aggregate approach for certain functions.

These techniques hinge on the observation that a function with higher stability on ${\mathcal{D}}$ requires less noise to ensure worst case privacy. Such techniques are also applied to answer multiple online functions/queries in model-agnostic learning Bassily et al. (2018). However, we highlight two key differences in our setting with a weaker stability assumption. First, in order to estimate the perturbation stability of $g$ on ${\mathcal{D}}$, one needs to downsample or split ${\mathcal{D}}$ into multiple blocks Thakurta & Smith (2013); Dwork et al. (2014); Bassily et al. (2018), $\hat{{\mathcal{D}}}_{1},\dots,\hat{{\mathcal{D}}}_{B}$ , and estimate the perturbation stability based on the mode of $g(\hat{{\mathcal{D}}}_{1}),\dots,g(\hat{{\mathcal{D}}}_{B})$. This essentially reduces the amount of change in the output of $g$ due to a single entry in ${\mathcal{D}}$, with high probability and replaces the hard-to-estimate perturbation stability of $g$ with an easy-to-compute perturbation stability of the mode. Such a notion of stability has also been successfully applied, along with the sparse vector technique, for model-agnostic private learning to handle exponentially number of queries to a model Bassily et al. (2018). Note that in these cases, since a private stochastic test is applied, one cannot achieve pure differential privacy Dwork et al. (2014). In practice, e.g. federated learning, however, one does not have direct access to ${\mathcal{D}}$, and thus it is impractical to draw samples from or to split ${\mathcal{D}}$. Second, to ensure good utility, one relies on a key assumption, i.e. the subsampling stability of $g$, which requires $g(\hat{{\mathcal{D}}})=g({\mathcal{D}})$ with high probability over the draw of subsamples $\hat{{\mathcal{D}}}$.

Although our intuition in designing DaRRM also relies on the stability of the mode function $g$, previous usage of stability to improve privacy-utility tradeoffs, e.g., propose-test-release Vadhan (2017); Dwork et al. (2014), requires the testing of such stability, based on which one adds a larger (constant) noise $\gamma$. This can still lead to adding redundant noise in our case.

Optimal Randomized Response. Holohan et al. (2017) and Kairouz et al. (2015) show that the classical Randomized Response (RR) mechanism with a constant probability of faithfully revealing the true answer is optimal in certain private estimation problems. Our proposed DaRRM framework and our problem setting is a generalized version of the ones considered in both Holohan et al. (2017) and Kairouz et al. (2015), which not only subsumes RR but also enables a data-dependent probability, or noise addition.

While RR with a constant probability can be shown optimal in problems such as private count queries or private estimation of trait possession in a population, it is not optimal in other problems, such as private majority ensembling, since unlike the former problems, changing one response of the underlying mechanisms does not necessarily change the output of the majority. To explicitly compute the minimum amout of noise required, one needs the output distributions of the underlying mechanisms but this is unknown. To resolve this, our proposed DaRRM framework adds the amount of noise dependent on the set of observed outcomes from the underlying private mechanisms, $\mathcal{S}$, which is a random variable of the dataset and is hence a proxy. This enables DaRRM to calibrate the amount of noise based on whether the majority output is likely to change. The amount of noise is automatically reduced when the majority output is not likely to change.

Second, Holohan et al. (2017) and Kairouz et al. (2015) both consider a special case in our setting where all $K$ private mechanisms are i.i.d., while our approach focuses on the more general setting where each private mechanism can have a different output distribution.

Learning A Good Noise Distribution. There have been limited works that attempt to derive or learn a good noise distribution that improves the utility. For deep neural networks inference, Mireshghallah et al. (2020) attempts to learn the best noise distribution to maximizing utility subject to an entropy Lagrangian, but no formal privacy guarantees were derived. For queries with bounded sensitivity, Geng & Viswanath (2015) demonstrate that the optimal noise distribution is in fact a staircase distribution that approaches the Laplacian distribution as $\epsilon\to 0$.

Private Prediction. Instead of releasing a privately trained model as in private learning, private prediction hides the models and only releases private outputs. Private prediction has been shown as a practical alternative compared to private learning, as performing private prediction is much easier compared to private learning on a wide range of tasks Dwork & Feldman (2018); Naor et al. (2023); van der Maaten & Hannun (2020). Although a privately trained model can make infinitely many predictions at the inference time without incurring additional privacy loss, since differential privacy is closed under post-processing, it has been shown recently that it is indeed possible to make infinitely many private predictions Naor et al. (2023) with a finite privacy loss for specific problems.

We first introduce the definition of differential privacy, simple composition and general composition as follows. The general composition Kairouz et al. (2015) gives a near optimal and closed-form bound on privacy loss under adaptive composition, which improves upon advanced composition Dwork et al. (2014).

We then formalize the error and utility metric in our problem as follows:

Notation. Throughout the paper, we use the same notations defined in Problem 1.1 and Definition 2.4. Furthermore, let ${\mathcal{D}}$ and ${\mathcal{D}}^{\prime}$ to denote a pair of adjacent datasets with one entry being different. Also, let $p_{i}=\Pr[M_{i}({\mathcal{D}})=1]$ and $p_{i}^{\prime}=\Pr[M_{i}({\mathcal{D}}^{\prime})=1]$, $\forall i\in[K]$. We omit the subscript $i$ when all $p_{i}$’s or $p_{i}^{\prime}$’s are equal. ${\mathbb{I}}\{\cdot\}$ denotes the indicator function and $[K]=\{1,2,\dots,K\}$. For the purpose of analysis, let ${\mathcal{L}}({\mathcal{D}})=\sum_{i=1}^{K}M_{i}({\mathcal{D}})\in\{0,1,\dots,K\}$, i.e. the (random) sum of all observed outcomes on dataset ${\mathcal{D}}$. ${\mathcal{D}}$ is omitted when the context is clear. Unless specified, we use the noise function $\gamma:\{0,1,\dots,K\}\to[0,1]$ as input to our algorithms to calibrate the probabilistic noise injection. Unless specified, the privacy allowance $m\in\mathbb{R}$.

We compare the shape and the error ${\mathcal{E}}(\textsf{DaRRM}_{\gamma})$ of different $\gamma$ functions: an optimized $\gamma_{opt}$ and the subsampling $\gamma_{Sub}$ as in Lemma 3.1⁴⁴4Note the subsampling mechanism from Section 4, which enjoys a privacy amplification by a factor of 2, only applies to pure differential privacy settings (i.e., when $\Delta=\delta=0$). However, we focus on the more general approximate differential privacy settings (with $\Delta>0$) in the experiments, and hence, the subsampling baseline we consider throughout this section is the basic version without privacy amplification. To see how the subsampling mechanism from Section 4 with privacy amplification compares against the other algorithms, please refer to Appendix D.1.2.. We also compare against $p_{const}$ in the classical baseline RR (see Section A.1) and ${\mathcal{E}}(\textsf{RR})$. Here, $p_{const}$ can be viewed as a constant noise function $\gamma_{const}(l)=p_{const},\forall l\in\{0,1,\dots,K\}$; and ${\mathcal{E}}(\textsf{RR})$ is the same as ${\mathcal{E}}(\textsf{DaRRM}_{\gamma_{const}})$.

We present the results with $K=11,{\epsilon}=0.1,\Delta=10^{-5}$ and $m\in\{1,3,5,7\}$. We assume there is no prior knowledge about the mechanisms $\{M_{i}\}_{i=1}^{K}$, and set the prior distribution from which $p_{i}$’s are drawn, ${\mathcal{T}}$, to be the uniform distribution, in the optimization objective (Eq. 2) searching for $\gamma_{opt}$. To ensure a fair comparison against the subsampling baseline, we set $\delta$ to be the one by $m$-fold general composition (see Theorem 2.3), which in this case, is $\delta=1-(1-\Delta)^{m}\approx m\Delta$. We plot each $\gamma$ functions over the support $\{0,1,\dots,K\}$ and the corresponding error of each algorithm in Figure 2.

Discussion. In summary, at $m=1$, the optimized noise function $\gamma_{opt}$ overlaps with $\gamma_{sub}$ which corresponds to the subsampling baseline. This agrees with our lower bound on the error in Lemma 3.2, which implies that at $m=1$, subsampling indeed gives the optimal error. When $m>1$, the optimized noise function $\gamma_{opt}$ has the highest probability of outputting the true majority over the support than the $\gamma$ functions corresponding to the baselines. This implies $\textsf{DaRRM}_{\gamma_{opt}}$ has the lowest error (and hence, highest utility), which is verified on the bottom set of plots. More results on comparing the $\textsf{DaRRM}_{\gamma_{opt}}$ optimized under the uniform ${\mathcal{T}}$ against the baselines by general composition (Theorem 2.3) and in pure differential privacy settings (i.e., $\Delta=\delta=0$) for large $K$ and $m$ can be found in Appendix D.1.1 and D.1.2. Furthermore, we include results optimizing $\gamma$ using a non-uniform ${\mathcal{T}}$ prior in Appendix D.1.3.

Semi-supervised Knowledge Transfer. We apply our DaRRM framework in the application of semi-supervised knowledge transfer for private image classification. We follow a similar setup as in PATE Papernot et al. (2017; 2018), where one trains $K$ teachers, each on a subset of a sensitive dataset, and at the inference time, queries the teachers for the majority of their votes, i.e., the predicted labels, of a test sample. Each time the teachers are queried, there is a privacy loss, and we focus on this private prediction subroutine in this section. To limit the total privacy loss over all queries, the student model is also trained on a public dataset without labels. The student model queries the labels of a small portion of the samples in this dataset from the teachers and is then trained using semi-supervised learning algorithms on both the labeled and unlabeled samples from the public dataset.

Baselines. We want the privacy loss per query of a test sample to the teachers to be $({\epsilon}_{query},\delta_{query})$. This can be achieved via two ways: 1) Train $K$ non-private teachers, add Gaussian noise to the number of predicted labels from the teachers in each output class, and output the majority of the noisy votes. This is exactly the GNMax algorithm from PATE Papernot et al. (2018). 2) Train $K$ $({\epsilon},\Delta)$-differentially private teachers and output the majority of the teachers’ votes by adding a smaller amount of noise. This can be computed using DaRRM with an appropriate noise function $\gamma$. We compare the performance of GNMax and DaRRM with two $\gamma$ functions: $\gamma_{opt}$ (i.e., the optimized $\gamma$), and $\gamma_{Sub}$ (i.e., the subsampling baseline). The overall privacy loss over $Q$ queries to the teachers can be computed by general composition (Theorem 2.3).

Experiment Setup. We use samples from two randomly chosen classes — class 5 and 8 — from the MNIST and Fashion-MNIST datasets to form our training and testing datasets. Our MNIST has a total of 11272 training samples and $1866$ testing samples; our Fashion-MNIST has $10000$ training samples and $2000$ testing samples. We train $K=11$ teachers on equally divided subsets of the training datasets. Each teacher is a CNN model. The non-private and private teachers are trained using SGD and DP-SGD Abadi et al. (2016), respectively, for 5 epochs. DaRRM Setup: The Gaussian noise in DP-SGD has zero mean and std. $\sigma_{dpsgd}=12$; the gradient norm clipping threshold is $C=1$. This results in each private teacher, trained on MNIST and Fashion-MNIST, being $({\epsilon},\Delta)=(0.0892,10^{-4})$ and $(0.0852,10^{-4})$-differentially private, respectively, after 5 epochs. We set the privacy allowance $m=3$⁵⁵5 Here, we present results with privacy allowance $m=3$ because we think this is a more interesting case. $m=1$ is less interesting, since one cannot get improvement compared to the subsampling baseline. $m$ close to a $\frac{K}{2}\approx 5$ is also less interesting, as this case seems too easy for our proposed method (the optimized $\gamma$ function is very close to 1, meaning very little noise needs to be added in this case). Hence, we pick $m=3$, which is a case when improvement is possible, and is also potentially challenging for our optimization framework. This is also realistic as most applications would only want to tolerate a constant privacy overhead. See more results with different privacy allowance $m$’s in this setting in Appendix D.2.2. and the privacy loss per query is then computed using general composition under $m$-fold, which give the same privacy loss in the high privacy regime, resulting in $({\epsilon}_{query},\delta_{query})=(0.2676,0.0003)$ on MNIST and $(0.2556,0.0003)$ on Fashion-MNIST. GNMax Setup: We now compute the std. $\sigma$ of the Gaussian noise used by GNMax to achieve a per-query privacy loss of $(m{\epsilon},m\Delta)$, as in the DaRRM setup. We optimize $\sigma$ according to the Renyi differential privacy loss bound of Gaussian noise. Although Papernot et al. (2018) gives a potentially tighter data-dependent privacy loss bound for majority ensembling non-private teachers, we found when $K$ and the number of output classes are small as in our case, even if all teachers agree on a single output class, the condition of the data-dependent bound is not satisfied. Hence, we only use the privacy loss bound of Gaussian noise here to set $\sigma$ in GNMax. See Appendix D.2.1 for more details, including the $\sigma$ values and other parameters. Finally, the per sample privacy loss and the total privacy loss over $Q$ queries, which is computed by advanced composition, are reported in Table 9.

The testing dataset is treated as the public dataset on which one trains a student model. Papernot et al. (2018) empirically shows querying $Q=1\%N$ samples from a public dataset of size $N$ suffices to train a student model with a good performance. Therefore, we pick $Q\in\{20,50,100\}$. We repeat the selection of $Q$ samples 10 times and report the mean test accuracy with one std. in parentheses in Table 1. The $Q$ queries serve as the labeled samples in training the student model. The higher the accuracy of the labels from the queries, the better the final performance of the student model. We skip the actual training of the student model using semi-supervised learning algorithms here.

Discussion. Table 1 shows $\textsf{DaRRM}_{\gamma_{opt}}$ achieves the highest accuracy (i.e., utility) compared to the two baselines on both datasets. First, comparing to $\textsf{DaRRM}_{\gamma_{Sub}}$, we verify that subsampling does not achieve a tight privacy-utility tradeoff, and we can optimize the noise function $\gamma$ in DaRRM to maximize the utility given a target privacy loss. Second, comparing to GNMax, the result shows there are regimes where ensembling private teachers gives a higher utility than directly ensembling non-private teachers, assuming the outputs in both settings have the same privacy loss. Intuitively, this is because ensembling private teachers adds fine-grained noise during both training the teachers and aggregation of teachers’ votes, while ensembling non-private teachers adds a coarser amount of noise only to the teachers’ outputs. This further motivates private prediction from private teachers and the practical usage of DaRRM, in addition to the need of aggregating private teachers in federated learning settings with an honest-but-curious server.