Do not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning

Existing works studying differentially private machine learning in high-dimensional setting can be roughly categorized into two sets. One is treating the optimization of the machine learning objective as a whole mechanism and adding noise into this process. The other one is based on the knowledge transfer of machine learning models, which trains a differentially private publishable student model with private signals from teacher models. We review them one by one.

Differentially private convex optimization in high-dimensional setting has been studied extensively over the years (Kifer et al., 2012; Thakurta & Smith, 2013; Talwar et al., 2015; Wang & Xu, 2019; Wang & Gu, 2019). Although these methods demonstrate good utility on some convex settings, their analyses can not be directly applied to non-convex setting. Right before the submission, we note two independent and concurrent works (Zhou et al., 2020; Kairouz et al., 2020) that also leverage the gradient redundancy to reduce the added noise. Specifically, Kairouz et al. (2020) track historical gradients to do dimension reduction for private AdaGrad. Zhou et al. (2020) requires gradients on some public data and then project the noisy gradients into a public subspace at each update. One core difference between these two works and GEP is that we introduce residual gradient perturbation and GEP produces an unbiased estimator of the private gradients, which is essential for achieving the superior utility. Moreover, we weaken the auxiliary data assumption and introduce several designs that significantly boost the efficiency and applicability of GEP.

One recent progress towards training arbitrary models with differential privacy is Private Aggregation of Teacher Ensembles (PATE) (Papernot et al., 2017; 2018; Jordon et al., 2019). PATE first trains independent teacher models on disjoint shards of private data. Then it trains a student model with privacy guarantee by distilling noisy predictions of teacher models on some public samples. In comparison, GEP only requires some non-sensitive data that have similar natural features as the private data while PATE requires the public data follow exactly the same distribution as the private data and in practice it uses a portion of the test data to serve as public data. Moreover, GEP demonstrates better performance than PATE especially for complex datasets, e.g., CIFAR-10, because GEP can train the model with the whole private data rather than a small shard of data.

The pseudocode of GEP is presented in Algorithm 1. For convenience, we write a set of gradients and a set of basis vectors as matrices with each row being one gradient/basis vector.

The anchor subspace is constructed as follows. We first compute the gradients of the model on an auxiliary dataset ${\mathbb{D}}^{(a)}$ with $m$ samples, which is referred to as the anchor gradients ${\bm{G}}^{(a)}\in{\mathbb{R}}^{m\times p}$. We then use the power method to estimate the principal components of ${\bm{G}}^{(a)}$ to construct a subspace basis ${\bm{B}}\in{\mathbb{R}}^{k\times p}$, which is referred to as the anchor subspace. All these matrices are publishable because ${\mathbb{D}}^{(a)}$ is non-sensitive. We expect that the anchor subspace ${\bm{B}}$ can cover most energy of private gradients when the auxiliary data are not far from private data and $m,k$ are reasonably large.

Suppose that the private gradients are ${\bm{G}}\in{\mathbb{R}}^{n\times p}$. Then, we project the private gradients into the anchor subspace ${\bm{B}}$. The projection produces low-dimensional embeddings ${\bm{W}}={\bm{G}}{\bm{B}}^{T}$ and residual gradients ${\bm{R}}={\bm{G}}-{\bm{G}}{\bm{B}}^{T}{\bm{B}}$. The magnitude of residual gradients is usually much smaller than original gradient even when $k$ is small because of the gradient redundancy.

Then, we aggregate the gradient embeddings and the residual gradients, respectively. We perturb the aggregated embedding and the aggregated residual gradient respectively to guarantee certain differential privacy. Finally, we release the perturbed embedding and the perturbed residual gradient and construct an unbiased estimator of the private gradient: $\tilde{{\bm{v}}}:=(\tilde{\bm{w}}^{T}{\bm{B}}+\tilde{\bm{r}})/n$. This construction process does not resulting in additional privacy loss because of DP’s post-processing property. The privacy analysis of the whole process of GEP is given in Theorem 1.

[] Let $S_{1}$ and $S_{2}$ be the sensitivity of ${\bm{w}}$ and ${\bm{r}}$, respectively, the output of Algorithm 1 satisfies $(\epsilon,\delta)$-DP for any $\delta\in(0,1)$ and $\epsilon\leq 2\log(1/\delta)$ if we choose $\sigma_{1}\geq 2S_{1}\sqrt{2\log(1/\delta)}/\epsilon$ and $\sigma_{2}\geq 2S_{2}\sqrt{2\log(1/\delta)}/\epsilon$.

A common practice to control sensitivity is to clip the output with a pre-defined threshold. In our experiments, we use different thresholds $S_{1}$ and $S_{2}$ to clip the gradient embeddings and residual gradients, respectively. The privacy loss of GEP consists of two parts: the privacy loss incurred by releasing the perturbed embedding and the privacy loss incurred by releasing the perturbed residual gradient. We compose these two parts via the Rényi differential privacy and convert it to $(\epsilon,\delta)$-DP.

We highlight several implementation techniques that make GEP widely applicable and implementable with reasonable computational cost. Firstly, auxiliary non-sensitive data do not have to be the same source as the private data and the auxiliary data can be randomly labeled. This non-sensitive data assumption is very weak and easy to satisfy in practical scenarios. To understand why random label works, a quick example is that for the least squares regression problem the individual gradient is aligned with the feature vector while the label only scales the length but does not change the direction. This auxiliary data assumption avoids conducting principal component analysis (PCA) on private gradients, which requires releasing private high-dimensional basis vectors and hence introduces large privacy loss. Secondly, we use power method (Panju, 2011; Vogels et al., 2019) to approximately estimate the principal components. The new operation we introduce is standard matrix multiplication that enjoys efficient implementation on GPU. The computational complexity of each power iteration is $2mkp$, where $p$ is the number of model parameters, $m$ is the number of anchor gradients and $k$ is the number of subspace basis vectors. Thirdly, we divide the parameters into different groups and compute one orthonormal basis for each group. This further reduces the computational cost. For example, suppose the parameters are divided into two groups with size $p_{1},p_{2}$ and the numbers of basis vectors are $k_{1},k_{2}$, the computational complexity of each power iteration is $2m(k_{1}p_{1}+k_{2}p_{2})$, which is smaller than $2m(k_{1}+k_{2})(p_{1}+p_{2})$. In Appendix B, we analyze the additional computational and memory costs of GEP compared to standard gradient perturbation.

Curious readers may wonder if we can use random projection to reduce the dimensionality as Johnson–Lindenstrauss Lemma (Dasgupta & Gupta, 2003) guarantees that one can preserve the pairwise distance between any two points after projecting into a random subspace of much lower dimension. However, preserving the pairwise distance is not sufficient for high quality gradient reconstruction, which is verified by the empirical observation in Appendix C.

Let ${\bm{g}}:=\frac{1}{n}\sum_{i}{\bm{G}}_{i,:}$ be the target private gradient. For a given anchor subspace ${\bm{B}}$, the residual gradients are defined as ${\bm{R}}:={\bm{G}}-{\bm{G}}{\bm{B}}^{T}{\bm{B}}$. We then analyze how large the residual gradients could be. The following argument holds for all time steps and we ignore the time step index for simplicity.

For the ease of discussion, we introduce ${\bm{\xi}}_{i}:=({\bm{G}}_{i,:})^{T}$ for $i\in[n]$ to denote the the private gradients and the $\hat{{\bm{\xi}}}_{j}:=({\bm{G}}^{(a)}_{j,:})^{T}$ for $j\in[m]$ to denote the anchor gradients. We use $\lambda_{k}(\cdot)$ to denote the $k_{th}$ largest eigenvalue of a given matrix. We assume that the private gradients ${\bm{\xi}}_{1},...,{\bm{\xi}}_{n}$ and the anchor gradients $\hat{{\bm{\xi}}}_{1},...,\hat{{\bm{\xi}}}_{m}$ are sampled independently from a distribution ${\mathcal{P}}$. We assume ${\bm{\Sigma}}:=\mathbb{E}_{{\bm{\xi}}\sim{\mathcal{P}}}{\bm{\xi}}{\bm{\xi}}^{T}\in{\mathbb{R}}^{p\times p}$ to be the population gradient (uncentered) covariance matrix. We also consider the (uncentered) empirical gradient covariance matrix $\hat{{\bm{S}}}:=\frac{1}{m}\sum_{i=1}^{m}\hat{{\bm{\xi}}}_{i}\hat{{\bm{\xi}}}_{i}^{T}$.

One case is that the population gradient covariance matrix ${\bm{\Sigma}}$ is low-rank $k$. In this case we can argue that the residual gradients are $0$ once the number of anchor gradients $m>k$.

We note that $s$-flat is the translate ${\mathbb{F}}_{s}={{\bm{x}}}+{\mathbb{F}}_{s(0)}$ of an $s$-dimensional linear subspace ${\mathbb{F}}_{s(0)}$ in ${\mathbb{R}}^{p}$ and the normal distribution satisfies such condition (Eaton & Perlman, 1973; Muirhead, 2009). Therefore, we have seen that for low-rank case of population covariance matrix, the residual gradients are 0 once $m>k$. In the general case, we measure the expected norm of the residual gradients.

From Lemma 3.2, we can see the larger the number of anchor gradients and the dimension of the anchor subspace $k$, the smaller the residual gradients. We can choose $m,k$ properly such that the upper bound on the expected residual gradient norm is small. This indicates that we may use a smaller clipping threshold and consequently apply smaller noises with achieving the same privacy guarantee.

We next empirically examine the projection error ${\bm{r}}=\sum_{i}{\bm{R}}_{i,:}$ by training a $20$-layer ResNet on CIFAR10 dataset. We try two different types of auxiliary data to compute the anchor gradients: 1) samples from the same source as private data with correct labels, i.e., $2000$ random samples from the test data; 2) samples from different source with random labels, i.e., $2000$ random samples from ImageNet. The relation between the dimension of anchor subspace $k$ and the projection error rate ($\left\lVert\frac{1}{n}{\bm{r}}\right\rVert/\left\lVert{\bm{g}}\right\rVert$) is presented in Figure 5. We can see that the project error is small and decreases with $k$, and the benefit of increasing $k$ diminishes when $k$ is large, which is implied by Lemma 3.2. In practice one can only use small or moderate $k$ because of the memory constraint. GEP needs to store at least $k$ individual gradients and each individual gradient consumes the same amount of memory as the model itself. Moreover, we can see that the projection into anchor subspace of random labeled auxiliary data yields comparable projection error, corroborating our argument that unlabeled auxiliary data are sufficient for finding the anchor subspace.

We also verify that the redundancy of residual gradients is small, by plotting the stable rank of residual gradient matrix in Figure 5. The stable rank of residual gradient matrix is an order of magnitude higher than the stable rank of original gradient matrix. This implies that it could be hard to further approximate ${\bm{R}}$ with low-dimensional embeddings.

We next compare the GEP with a scheme that simply discards the residual gradients and only outputs the perturbed gradient embedding, i.e., the output is $\tilde{{\bm{u}}}:=\tilde{\bm{w}}^{T}{\bm{B}}/n$.

This indicates that ${\tilde{{\bm{u}}}}$ contains a systematic error that makes ${\tilde{{\bm{u}}}}$ always deviate from ${\bm{g}}$ by the residual gradient. This systematic error is the projection error, which is plotted in Figure 5. The systematic error cannot be mitigated by reducing the noise magnitude (e.g., increasing the privacy budget or collecting more private data). We refer to the algorithm releasing ${\tilde{{\bm{u}}}}$ directly as Biased-GEP or B-GEP for short, which can be viewed as an efficient implementation of the algorithm in (Zhou et al., 2020). In our experiments, B-GEP can outperform standard gradient perturbation when $k$ is large but is inferior to GEP. We note that the above remark is made with ignoring the clipping effect (or set a large clipping threshold). In practice, we do apply clipping for the individual gradients at each time step, which makes the expectations in Remark 1 obscure (Chen et al., 2020b). We note that the claim that ${\tilde{{\bm{v}}}}$ is an unbiased estimator of ${\bm{g}}$ is not that precise when applying gradient clipping.

GEP (Algorithm 1) describes how to release one-step gradient with privacy guarantee. In this section, we compose the privacy losses at each step to establish the privacy guarantee for the whole learning process. The differentially private learning process with GEP is given in Algorithm 2 and the privacy analysis is presented in Theorem 2.

[] For any $\epsilon<2\log(1/\delta)$ and $\delta\in(0,1)$, the output of Algorithm 2 satisfies $(\epsilon,\delta)$-DP if we set $\sigma\geq 2\sqrt{2T\log(1/\delta)}/\epsilon$.

If the private gradients are randomly sampled from the full batch gradients, the privacy guarantee can be strengthened via the privacy amplification by subsampling theorem of DP (Balle et al., 2018; Wang et al., 2019; Zhu & Wang, 2019; Mironov et al., 2019). Theorem 2 gives the expected excess error of Algorithm 2. Expected excess error measures the distance between the algorithm’s output and the optimal solution in expectation.

[] Suppose the loss $L(\boldsymbol{\theta})=\frac{1}{n}\sum_{({\bm{x}},y)\in{\mathbb{D}}}\ell(f_{\boldsymbol{\theta}}({\bm{x}}),y)$ is 1-Lipschitz, convex, and $\beta$-smooth. If $\eta=\frac{1}{\beta}$, $T=\frac{n\beta\epsilon}{\sqrt{p}}$, and $\boldsymbol{\bar{\theta}}=\frac{1}{T}\sum_{t=1}^{T}\boldsymbol{\theta}_{t}$, then we have $\mathbb{E}[L(\boldsymbol{\bar{\theta}})]-L(\boldsymbol{\theta_{*}})\leq\mathcal{O}\left(\frac{\sqrt{k\log(1/\delta)}}{n\epsilon}+\frac{\bar{r}\sqrt{p\log(1/\delta)}}{n\epsilon}\right)$, where $\bar{r}=\frac{1}{T}\sum_{t=0}^{T-1}r^{2}_{t}$ and $r_{t}=\max_{i}\left\lVert({\bm{R}}_{t})_{i,:}\right\rVert$ is the sensitivity of residual gradient at step $t$.

The $\bar{r}$ term represents the average projection error over the training process. The previous best expected excess error for gradient perturbation is $\mathcal{O}(\sqrt{p\log(1/\delta)}/(n\epsilon))$ (Wang et al., 2017). As shown in Lemma 3.1, if the gradients locate in a $k$-dimensional subspace over the training process, $\bar{r}=0$ and the excess error is $\mathcal{O}(\sqrt{k\log(1/\delta)}/(n\epsilon))$, independent of the problem ambient dimension $p$. When the gradients are in general position, i.e., gradient matrix is not exact low-rank, Lemma 3.2 and the empirical result give a hint on how small the residual gradients could be. However, it is hard to get a good bound on $\max_{i}\|({\bm{R}}_{t})_{i,:}\|$ and the bound in Theorem 2 does not explicitly improve over previous result. One possible solution is to use a clipping threshold based on the expected residual gradient norm. Then the output gradient becomes biased because of clipping and the utility/privacy guarantees in Theorem 2/2 require new elaborate derivation. We leave this for future work.