Differentially Private ADMM Algorithms for Machine Learning

For solving the generalized ERM problem (1) with a smooth convex function $f(x)\!=\!\frac{1}{n}\sum_{i=1}^{n}\!{f_{i}(x,l_{i})}$ and a complex sparsity inducing regularizer, we introduce an auxiliary variable $y$ and an equality constraint $Ax+By=c$, and can reformulate Problem (1) as follows:

where $A\in\mathbb{R}^{d_{3}\times d_{1}}$, $B\in\mathbb{R}^{d_{3}\times d_{2}}$, and $c\in\mathbb{R}^{d_{3}}$. Therefore, this paper aims to propose efficient differentially private ADMM algorithms for solving the more general equality constrained minimization problem (2).

Let $\rho>0$ be a penalty parameter, and $u$ be a dual variable. Then the augmented Lagrangian function of Problem (2) is:

In an alternating or sequential fasion, at iteration $t$, ADMM performs the following update rules:

This is the classic update form of ADMM [37]. While updating the variable $x$, this update step usually has a high computational complexity, especially when the dataset is very large.

As for differential privacy, there are three main types of perturbation used to solve the empirical risk minimization problems under differential privacy [2, 3]. Output perturbation is to perturb the model parameters. For instance, [38] analyzed that introducing output perturbation can make $k$-anonymous algorithms satisfy differential privacy. [39] analyzed the sensitivity of optimal solution between neighboring databases. Objective perturbation is to perturb the objective function trained by algorithms. [40] proposed an algorithm introduced objective perturbation to solve ERM problems and analyzed its privacy guarantee to satisfy $(\epsilon,\delta)$-DP. Gradient perturbation is to perturb the gradients used for updating parameters by first-order optimization methods. For instance, [19] proposed DP-SVRG by introducing gradient perturbation to the SVRG operator in [29], and used moment accountant to complete both its privacy guarantee and utility guarantee.

In recent years, there are some research works on the ADMM algorithms that satisfies differential privacy. However, most of them are about distributed ADMM algorithms. For instance, [25] analyzed the relationship between privacy guarantee and utility guarantee on their differentially private distributed ADMM algorithms. Of course, there are few research works focused on differentially private centralized ADMM algorithms. For example, [41] proposed two stochastic ADMM algorithms, which satisfy $(\alpha,\beta)$-RDP and provide privacy guarantee of the algorithms. However, there is no research work on differentially private centralized and deterministic ADMM algorithms. Therefore, in this paper, we focus on differentially private centralized deterministic ADMM algorithms, and propose two efficient deterministic DP-ADMM and DP-AccADMM algorithms with privacy guarantees.

[1] introduced the formal notion of differential privacy as follows.

Sensitivity is a key indicator that determines the size of the noise added to the algorithm. That is, to achieve $(\epsilon,\delta)$-DP for a function $q\!:\!\mathbb{D}^{n}\!\to\!\mathbb{R}$, we usually use the following Gaussian mechanism, where the added noise is sampled from Gaussian distribution with variance that is proportional to the $\ell_{2}$-sensitivity of $q(\cdot)$.

Although the definition of $(\epsilon,\delta)$-DP is widely used in the objective and output perturbation methods, the notion of Rényi differential privacy (RDP) [42] is more suitable for gradient perturbation methods including the proposed algorithms.

In [32], Nesterov presented a first-order minimization scheme with a global convergence rate $O(1/T^{2})$ for solving Problem (1). This convergence rate is provably optimal for the class of Lipschitz differentiable functionals. As shown in Algorithm 1, the Nesterov method accelerates the gradient descent by using an overrelaxation step as follows:

where $\theta_{t+1}=(1+\sqrt{4\theta_{t}^{2}+1})/2$ with the initial value $\theta_{1}\!=\!1$.

In 2014, Goldstein et al. [36] proposed an accelerated ADMM algorithm (AccADMM) by introducing Nesterov acceleration. They also proved that their algorithm has a global convergence rate of $O(1/T^{2})$.

For solving the equality constrained minimization problem (2), and the specific algorithmic steps of our deterministic differentially private ADMM (DP-ADMM) algorithm are presented in Algorithm 2.

Analogous to the general ADMM algorithm, our DP-ADMM algorithm updates and iterates the $x,y,u$ variables in an alternating fashion. But when updating $x_{t}$, we use the first-order approximation of $f(x)$ at $x_{t-1}$ with Gaussian noise (i.e., $f(x_{t-1})+\langle\nabla\!f(x_{t-1})+P_{t},\,x\rangle$) to replace $f(x)$, where $P_{t}\!\sim\!N(0,\sigma^{2}I_{d_{1}})$ is the added Gaussian noise, and $\sigma$ is a noisy variance computed by privacy guarantee, which satisfies the Gaussian mechanism. Then we add a squared norm term $\frac{\|x-x_{t-1}\|_{G}^{2}}{2\eta}$ into the following proximal update rule of $x_{t}$,

where $\eta$ is a step-size or learning rate, $G=\gamma I-\eta\rho A^{T}\!A$, $\gamma\geq\gamma_{\min}\equiv\eta\rho\|A^{T}\!A\|_{2}+1$ to ensure that $G\succeq I$, and $\|z\|^{2}_{G}\!=\!z^{T}Gz$ with a given positive semi-definite matrix $G$ as in [23, 43]. Introducing this squared norm term can make the distance between adjacent iterates (i.e., $x_{t-1}$ and $x_{t}$) not be too far, and prevent the noise from affecting the iterates too much.

By using the linearized proximal point method [23], we can obtain the closed-form solution of $x_{t}$ in Eq. (7) as follows:

As shown in Algorithm 2, the update rules of $y_{t}$ in Eq. (3) and $u_{t}$ in Eq. (5) remain unchanged for DP-ADMM. Here, $g(y)$ is the regularization term used in many machine learning problems, e.g., the sparsity inducing term $g(y)\!=\!\lambda\|Ay\|_{1}$. For instance, if it is the $\ell_{1}$-norm regularized term, then the closed-form solution can be easily obtained using the soft-thresholding operator [44], and when it is the $\ell_{2}$-norm term, the closed-form solution can be obtained by derivation.

In this subsection, we theoretically analyze both privacy guarantee and utility guarantee of DP-ADMM. To facilitate our discussion, we first make the following basic assumptions.

Through the Gaussian mechanism of RDP, we can get the relationship between $\alpha$ and $\beta$ of RDP and the variance $\sigma$ of the added noise. Then, we can obtain the relationship between $\epsilon,\delta$ and $\sigma$ by the conversion relationship between RDP and $(\epsilon,\delta)$-DP. Thus, we can get the size of Gaussian noise variance.

The detailed proof of Theorem 1 is given in Appendix A.

Let ($x_{*},y_{*}$) be an optimal solution of Problem (2). Different from distributed differential privacy ADMMs [25, 26], by constructing the following convergence criterion $R(\tilde{x},\tilde{y})$,

where $\tilde{x}\!=\!\frac{1}{T}\sum_{t=1}^{T}\!x_{t}$ and $\tilde{y}\!=\!\frac{1}{T}\sum_{t=1}^{T}\!y_{t}$, we can complete the convergence analysis of DP-ADMM. Then by introducing the noise variance term and finding the number of iterations $T$ required to reach the stopping criterion, we can get the utility bound and gradient complexity of DP-ADMM. The utility bound is an important indicator to measure the performance of differential privacy algorithms including DP-ADMM. It usually reflects the minimum value that the algorithm can converge to. The lower the utility bound is, the smaller the loss can decrease and the better the trained model is.

Gradient complexity reflects the number of gradient calculations required by the model to reach the utility bound. It is usually proportional to the running time of the algorithm and has a certain relationship with the convergence rate of the algorithm. The smaller the gradient complexity is, the fewer the number of gradients need to be calculated.

The detailed proof of Theorem 2 can be found in Appendix B. Theorem 2 shows that the privacy guarantee and utility guarantee of DP-ADMM. That is, the size of the added noise variance required by the algorithm to meet differential privacy requirement is given, and the minimum value of the algorithm’s loss is obtained. Compared with the theoretical analysis of traditional ADMMs, we find that the convergence rate of DP-ADMM does not change. This means that differential privacy protection does not affect the convergence rate of the optimization algorithm or the speed at which the optimization algorithm converges to the optimal value, but the optimal value that the algorithm converges to.

Non-accelerated ADMMs including DP-ADMM can be used to solve many machine learning problems with equality constraints, but it has a flaw that the convergence speed is not fast enough as in the non-private case. Especially when approaching to the exact solution, the convergence speed is very slow. Therefore, we propose an accelerated version of our DP-ADMM algorithm, called DP-AccADMM.

Inspired by the accelerated ADMM algorithm [36] using the Nesterov method, we introduce the Nesterov accelerated technique into our DP-ADMM algorithm. The detailed steps of our accelerated differentially private ADMM (DP-AccADMM) algorithm are presented in Algorithm 3. In particular, we add a weight coefficient during the iteration process to linearly combine the two adjacent iterates, so that the current iterate can be updated by using the information of the previous iterate. That is, the current descent direction is the combination of the current optimal direction and the previous descent direction, which will make the current iterate to move forward a distance on the previous descent direction. Note that the weight coefficient for momentum acceleration changes with the increase of iterations. In our DP-AccADMM algorithm, we set the weight coefficient $\theta_{t}$ at the $t$-th iteration as follows:

where $\theta_{1}\!=\!1$. And the momentum accelerated update rule of $\hat{x}_{t}$ is defined as:

Such an accelerated scheme is consistent with the acceleration method proposed by Nesterov [34] and the accelerated ADMM algorithm proposed by Goldstein et al. [36]. In addition, the dual variable $\hat{u_{t}}$ also has the same accelerated scheme as $\hat{x}_{t}$, as shown in Step 7 of Algorithm 3.

Although we do not give the theoretical analysis of DP-AccADMM, we conduct many experiments to compare the performance of the two algorithms, DP-AccADMM and DP-ADMM, whose noise level is the same, as shown in Theorem 1. We will report many experimental results and give some discussions in the next section.

To evaluate performance of the proposed algorithms, we consider the following $\ell_{1}$-norm regularized graph-guided fused Lasso problem,

where $f_{i}$ is the logistic loss function on the feature-label pair $(l_{i},m_{i})$, i.e., $\log(1+\exp(-m_{i}l_{i}^{T}x))$, and $\lambda$ is the regularization parameter. We set $A=[W;I]$ as in [43, 45, 46], where $W$ is the sparsity pattern of the graph obtained by sparse inverse covariance selection [47]. We used four publicly available datasets in our experiments, as listed in Table I.

We fix $\delta\!=\!10^{-3}$ for all the experiments, and we choose $\mu=0.5$ and $\rho=1$, where $\mu$ is an intermediate constant in the solution process and $\rho$ is the penalty parameter of the ADMM algorithms. But in our theoretical analysis, $\rho$ will not influence the utility bound of our differentially private algorithms. Thus, we fix $\rho$ as a constant. Moreover, we set $\gamma=1$ and set $\eta$ as a constant on each dataset but differs among these datasets.

Fig. 1 plots the objective gap (i.e., the objective value minus the minimum value) of the differential privacy ADMM algorithms as the number of iterations increases. Actually, the accelerated ADMM algorithm can achieve the convergence rate of $O(1/T^{2})$, and it is much faster than the traditional ADMM algorithm. In our experiments, it can be seen that the convergence speed of DP-AccADMM is almost the same as that of its non-differential privacy counterpart, AccADMM. In addition, we can see that there are two gaps between two pairs of algorithms. These gaps are the embodiment of the utility boundary in the utility analysis in the experiments. And the gap between DP-AccADMM and AccADMM is larger than that between DP-ADMM and ADMM. It means that the Nesterov acceleration may increase the utility bound of the algorithm, which will reduce the utility of the algorithm. Therefore, we do another experiment to show the performance of test accuracy of all these algorithms.

Fig. 2 plots the objective value and test accuracy of the differential privacy ADMM algorithms with different privacy budgets $\epsilon$ for solving $\ell_{1}$-norm regularized graph-guided fused Lasso. We can see that AccADMM converges much faster than ADMM in terms of objective value. DP-AccADMM performs worse than DP-ADMM, when the private budget $\epsilon$ is small (e.g., $\epsilon\!=\!0.01$). But with the increase of the budget $\epsilon$ (e.g., $\epsilon\!=\!0.07$ on bio_train), DP-AccADMM outperforms DP-ADMM in terms of both objective value and test accuracy. As the private budget increases, the objective value of DP-AccADMM gradually approaches to that of AccADMM, and the objective value of DP-ADMM approaches to that of ADMM. Thus, the Nesterov acceleration usually leads to a worse utility, when $\epsilon$ is small. However, when $\epsilon$ becomes larger, DP-AccADMM gets better and outperforms DP-ADMM.

From test accuracy results, we can see that DP-AccADMM also performs worse than DP-ADMM, when the private budget $\epsilon$ is small. With the increase of $\epsilon$, the test accuracy of DP-AccADMM gradually improves, approaching or even exceeding the test accuracy of DP-ADMM. Moreover, the test accuracy of DP-AccADMM gradually approaches to that of AccADMM and the test accuracy of DP-ADMM gradually approaches to that of ADMM. Thus, for test accuracy, we can see that the Nesterov acceleration technique usually leads to a worse test accuracy, when the private budget $\epsilon$ is small (e.g., $\epsilon\!=\!0.02$). However, with the increase of $\epsilon$ (e.g., $\epsilon\!=\!0.08$ on a8a and ijcnn1), the test accuracy of DP-AccADMM gradually increases and outperforms DP-ADMM until it gradually approaches to the test accuracy of AccADMM.

Before giving the proof of Theorem 1, we present the following lemmas. We first give the following relationship between RDP and $(\epsilon,\delta)$-DP [42]:

This lemma shows the conversion relationship between $(\alpha,\beta)$-RDP and $(\epsilon,\delta)$-DP. And RDP has the following composition properties [42]:

This lemma shows the changes of the coefficients $\alpha,\beta$ that satisfy the RDP during the compounding process of a series of random mechanisms of the algorithm iteration.

This lemma shows the relationship between the satisfied RDP coefficients $\alpha,\beta$ and the size of added noise variance $\sigma$.

Proof of Theorem 1:

Before giving the proof of Theorem 2, we first present the following lemmas.

This lemma presents the relationship between the optimal solution of the dual problem and the optimal solution of the original problem. Let us simply express $x_{t}$ with $x_{t-1}$ and the gradient term, which is convenient for our subsequent proof.

Setting the derivative with respect to $x$ at $x_{t}$ to zero, we have

Let

Then $g_{t}+\frac{1}{\eta}G(x-x_{t-1})=0$, and $x_{t}=x_{t-1}-\eta G^{-1}g_{t}$. Thus, we get a simple representation of $x_{t}$.

Proof of Theorem 2: