The Role of Adaptive Optimizers
for Honest Private Hyperparameter Selection^††thanks: Authors SM and SS have equal contribution and are listed in alphabetical order. Authors XH, GK, OT have equal contribution and are listed in alphabetical order.

[LT19] propose a random stopping algorithm (LT) to output a ‘good’ hyperparameter candidate from a pool of $K$ candidates, {$x_{1},\ldots,x_{K}$}. They assume sampling access to a randomized mechanism $Q(D)$ which samples $i\sim[K]$, and returns the $i$-th candidate $x_{i}$, and a score $q_{i}$ for this candidate. It is a random stopping algorithm, in which at every iteration, a candidate is picked from $Q$ i.i.d. with replacement and a $\gamma$-biased coin is flipped to randomly stop the algorithm. When the algorithm stops, the candidate with the maximum score seen so far is outputted. In the approximate DP version of this algorithm, an extra parameter $\Upsilon$ is set to limit the total of number of iterations. The pseudocode of this algorithm is deferred to Appendix A.

Theorem 4.1 expresses the privacy cost of the algorithm in terms of the privacy cost of individual learners, and parameters of the algorithm itself. The $\delta_{2}$ parameter does not significantly affect the final epsilon $\varepsilon_{f}$ of the algorithm and in practice, one can set it to a very small value ($10^{-20}$). Though a small value of $\delta_{2}$ has little effect on $\delta_{f}$, it increases the hard stopping time of the algorithm, $\Upsilon$.

To understand the LT algorithm, we will compare the privacy costs of training a single hyperparameter candidate with a final $\varepsilon_{f},\delta_{f}$ budget via LT and compare it with the privacy cost $\varepsilon_{1},\delta_{1}$ of the underlying individual learner. This setting might seem unnatural for LT as it was designed to select from a pool of candidates but we choose this setting to show the minimum privacy cost overhead associated with LT and later show how the privacy cost changes for multiple candidates (varying $\gamma$). To use LT, one needs to figure out the $\varepsilon_{1},\delta_{1}$ via Theorem 4.1 using the final $\varepsilon_{f},\delta_{f}$ values and in this case, $\gamma=1$ (as we have just one candidate). The individual learner is then trained using $\varepsilon_{1},\delta_{1}$ budget.

Due to the delicate balance of $\delta_{f}$ in Theorem 4.1, one can see the $\delta_{1}$ comes out to be much smaller than $\delta_{f}$. This change in $\delta_{1}$ results in a blowup of $\varepsilon_{1}$ and hence, the final privacy cost of the LT algorithm ($3\varepsilon_{1}+3\sqrt{2\delta_{1}}$), is much larger than what it would have been for learning one candidate without LT. We call this increase the blowup of privacy. We measure this blowup in Figure 1(left), for the setting of $\sigma=4$, $L=250$, $T=10,000$ with varying dataset sizes ($n$). It can be seen that for $n=5,000$, the blowup is 4.8x whereas for for $n=950,000$, the blowup is almost 7.3x (note the log scale on y-axis). Qualitatively similar trends persist for other choices of noise multiplier, lot size and iterations. We add more experiments to compare LT vs MA with varying candidate sizes in Appendix B.

Furthermore, we show that although LT entails a privacy blowup, decreasing $\gamma$ (corresponding to training more individual learners with $\varepsilon_{1},\delta_{1}$) doesn’t result in a significant difference in the final epsilon guaranteed by LT. In Figure 1(middle), we show the final epsilon cost for different dataset size and varying values of $\gamma\in[0.001,0.01,0.1,1]$. It is interesting to note here that with smaller $\gamma$ values, one can train many candidates (in expectation, $\frac{1}{\gamma}$) for negligible additional privacy cost. The blowup to train $1$ candidate ($\gamma=1$) versus $1,000$ candidates ($\gamma=0.001$) increases from $33$ to $39$ for $n=5,000$ and increases from $0.49$ to $0.69$, for $n=950,000$. This increase is minimal in comparison to advanced composition, which grows proportional to $\mathcal{O}(\sqrt{k})$. However, another resource at play is the total training time, proportional to $1/\gamma$ (i.e., the total number of candidates). In summary, the LT algorithm is effective if an analyst has the privacy budget to afford the initial blowup, as the privacy cost of testing additional hyperparameters is insignificant.

We learnt from the previous section that, LT permits selection from a large pool of hyperparameters (depending on the $\gamma$ value) but incurs a constant privacy blowup. We compare LT with tuning using Moments Accountant (MA); for tuning via MA each hyperparameter candidate is trained by adding necessary Gaussian noise at each iteration, and the best hyperparameter candidate is selected at the end, MA is used as the composition mechanism for arriving at the final privacy cost of this process. We notice that with the same initial privacy blowup of the LT algorithm, MA is able to compose a considerable number of hyperparameter candidates. In Figure 1(right), we show the number of candidates that can be composed using MA for the minimum privacy cost for running the LT algorithm ($\gamma=1$), for the setting of $\sigma=4$, $L=250$, $T$ = $10,000$ and varying dataset size ($n$) on the x-axis. As the $T$ and $L$ is set constant, bigger $n$ values in this graph correspond to fewer epochs of training and hence, worse utility. Depending on dataset size, MA can compose $14$ candidates for $n=5000$ and up to $175$ candidates when $n=100000$. It is perhaps surprising how well a standard composition technique performs versus LT. This information can be highly valuable to a practitioner who has limited privacy budget. Qualitatively similar trends persist for other choices of batch size, noise multiplier, and iterations.

From our experiments for both these tuning procedures, we conclude that while tuning with LT entails an initial privacy blowup, the additional privacy cost for trying more candidates (smaller $\gamma$) is minimal. Even though this has an additional computation cost, it can be appealing when an analyst wants to try numerous hyperparameters. On the other hand, for the same overall privacy cost, MA can be used to compose a significant number of hyperparameter candidates. Additionally, MA allows access to all intermediate learners, whereas LT allows access to only the final output parameters. In the sequel, this conclusion will be useful in making the naive MA approach a more appealing tool for some settings (e.g., tighter privacy budgets).

While many hyperparameters are restricted due to computational and privacy/utility targets, the learning rate $\alpha$ and the clipping threshold $C$ have no a priori bounds. In what follows, we show an interplay between these parameters by first theoretically analyzing the convergence of DPSGD. We then explore an illustrative experiment which demonstrates their entanglement. In the following theorem, we derive a bound on the expected excess risk of DPSGD and while doing so, show that the optimal learning rate, $\alpha_{opt}$, is proportional to the inverse of $C$. The proof appears in Appendix F.

Though Theorem 5.1 gives a closed-form expression for the optimum learning rate for the convergence bound to hold, it is a function of the parameter $R$, which is unknown a priori to the analyst. Given constant $T$ and $\sigma$, the optimal learning rate $\alpha_{opt}$ is inversely proportional to the clipping norm $C$. This is crucial information in practice because these parameters vary among datasets and are unbounded. This unboundedness property thus requires us to search over very large ranges of $C$ and $\alpha$ when we have no prior knowledge of the dataset. It is natural to ask whether one can fix the clipping norm $C$ and search only over a wide range for the learning rate $\alpha$ (or vice versa). We explore this relationship experimentally via simulations on a synthetic dataset as well as on the ENRON dataset, showing that fixing one of these two hyperparameters may often but not always result in an optimal model.

We train a linear regression model on a $10$-dimensional synthetic dataset of input-label pairs $(x,y)$ sampled from a distribution $\mathcal{D}$ as follows: $x_{1},\dots,x_{d}\sim\mathcal{U}(0,1),y=x\cdot w^{*},w^{*}=10\cdot\bm{1}^{d}$. We use the initialization $w_{0}=\bm{0}^{d}$ and train for 5000 iterations. We set the initial weights of the model $w_{0}=\bm{0}^{d}$, $d=10$, $k=10$ and train this linear regression model for $100$ iterations. In the non-private setting, this model converges quickly with any reasonable learning rate, but in the private setting, we notice that the training loss depends heavily on the choice of $\alpha$ and $C$. Figure 3 shows a heat map for the log training loss when trained on ($\alpha$,$C$) pairs taken from a large grid consisting of $[1,2,4,5,8]$ at scales of $[10^{-4},10^{-3},10^{-2},10^{-1},10^{0},10^{1}]$. The best candidates (lowest 1 percentile of loss values) we demarcate with white pixels.

We observe two fundamental phenomena from this figure. First, to achieve the best accuracy, $\alpha$ and $C$ need to be tuned on a large grid spanning several orders of magnitude for each of these parameters. Second, multiple ($\alpha$,$C$) pairs achieve the best accuracy and all lie on the same diagonal, validating our theory for an inverse relation between learning rate and clipping norm. As mentioned earlier, one might hypothesize that by setting the clipping norm $C$ constant and tuning $\alpha$ (corresponding to a vertical line in Figure 5) or vice versa, one could eliminate tuning a hyperparameter. However, note that not all $C$ and $\alpha$ values correspond to the lowest loss. This phenomenon is evident by noticing that not all vertical or horizontal lines on this figure have white pixels. This happens, for example, at the extremes (e.g., at the top-right corner), but also for several intermediate and standard choices (e.g., $C=0.1$ or $0.2$). Again, the analyst has no way of knowing this a priori.

In Figure 3, we detail the results for the same simulation experiment with DPAdam (with default Adam hyperparameters) as the underlying optimizer. Here we notice that the inverse relation between $\alpha$ and $C$ no longer hold as we intuited earlier. In order to be able to compare these two figures, we mark all candidates with lower losses than that of the lowest 1 percentile of Figure 3 with white in Figure 3. Figure 3 suggests that there is a small range of ideal choices for the initial learning rate $\alpha$, and within these good choices of the $\alpha$, DPAdam is significantly robust to the choice of clip value, as we see white pixels for a large range of clip values that encompass clip values used in practice. We repeat the same experiment over the ENRON dataset instead of our synthetic dataset, and observe similar trends as seen in Figure 5 and Figure 5. We conclude that to privately tune non-adaptive optimizers, we require a larger grid of hyperparameter options than their adaptive counterparts.

Adaptive optimizers automatically adapt over the learning rate $\alpha$, requiring us to tune only over the clipping norm $C$. But recall our key question: can we train models that perform competitively with the fine-tuned counterparts from DPSGD?

Adam [KB14], the canonical adaptive optimizer introduces two new hyperparameters, which are the first and second moment exponential decay parameters ($\beta_{1}$ and $\beta_{2}$). In the non-private setting, these parameters are relatively insensitive, and default values of $\alpha=0.001,\beta_{1}=0.9$, and $\beta_{2}=0.999$ are recommended based on empirical findings, requiring no additional tuning for this hyperparameter triple. Hence before we compare DPAdam and DPSGD, we first find and establish such recommended values for this hyperparameter triple in the DP setting next, and then show that DPAdam with a small hyperparameter space performs competitively with DPSGD in Section 6.

To establish default choices of $\alpha$, $\beta_{1}$, and $\beta_{2}$ for DPAdam, we evaluate this private optimizer over four diverse datasets Table 1, and two learning models including logistic regression and a neural network with one 100 neurons hidden layer (TLNN). These selected datasets include both low-dimensional data (where the number of samples greatly outnumbers the dimensionality) and high-dimensional data (where the number of samples and dimensionality are at same scale). Since we still have a large hyperparameter space to tune over, for the rest of this work, we fix a constant lot size ($L=250$), and consider tuning over three different noise levels, $\sigma\in[2,4,8]$, so that we can study the effects of tuning the other hyperparameters more thoroughly. All experiments are repeated three times and averaged before reporting. Additionally, in this particular experiment since we focus on $\alpha,\beta_{1},$ and $\beta_{2}$, we also fix the clipping threshold $C=0.5$, and $T=2500$ iterations of training. For each dataset and model, we run DPAdam three times with hyperparameters $(\alpha,\beta_{1},\beta_{2})$ from the grids, $\alpha\in[0.001,0.05,0.01,0.2,0.5]$, $\beta_{1},\beta_{2}\in[0.8,0.85,0.9,0.95,0.99.0.999]$.

We show that the default hyperparameter choice ($\alpha,\beta_{1},\beta_{2}$) of Adam in the non-private setting also works well for DPAdam. Figure 6 shows the boxplots of testing accuracies of DPAdam over the different hyperparameter choices. When $\alpha$ is $0.001$ (same as in the non-private setting), all the datasets and models have final testing accuracies (marked in black) close to the best possible (and in most cases it is in fact the best) accuracy. Furthermore, we also highlight the accuracy of the suggested default choice ($\alpha=0.001,\beta_{1}=0.9,\beta_{2}=0.999$) using gold dots. Hence, for the ease of using DPAdam, we suggest the non-private default values for these parameters in the private setting as well and hence in all our subsequent experiments.

For brevity, we show experiments for $\sigma=4$ in Figure 7, results for other values of $\sigma$ are displayed in Appendix G. For each dataset and model, we train three times for each hyperparameter candidate and report the max every 100 iterations, corresponding to the dark lines for each optimizer. We note that their maxima are extremely similar. However, Table 3 shows the final privacy costs incurred by each of these max accuracy lines, and reflects our claims from Section 4 that using fewer hyperparameter candidates and composing privacy via MA gives a much tighter privacy guarantee.

Additionally in Figure 7, DPSGD and DPMomentum have pastel dotted lines corresponding to their mean accuracy attained using the MA composition that provides the tightest privacy guarantees for DPAdam. These pastel lines are the mean accuracies (with 95% CI) from 100 repetitions of this experiment. Since DPAdam has only 4 hyperparameter candidates, for this experiment, we sample 4 of the candidates at random for DPSGD and DPMomentum so that they all incur the same privacy cost. Since the candidate pool is significantly larger for DPSGD and DPMomentum, we additionally scrutinize the parameter grid for them and prune the learning rates that perform poorly. Our pruning process (detailed in the supplement) is quite generous, and favours minimizing the hyperparameter space of DPSGD and DPMomentum as much possible.¹¹1Note, pruning itself is of course unfair; the intent was to design a DP optimizer that can be used on any data distributions that we have no prior knowledge of. To do so with DPSGD one would have to consider a significantly wide range of $(\alpha,C)$ pairs to cover ‘good’ candidates as we illustrated in Section 5 Despite the pruning advantage we see that these optimizers perform subpar than DPAdam when constrained with privacy.

DPAdam results have less variance than DPSGD due to its adaptive learning rate. To understand this phenomenon better, we inspect DPAdam’s update step. DPAdam being an adaptive optimizer picks per-parameter ESS as $\frac{\alpha}{\sqrt{\hat{v}_{t}}+\xi}$, which is the base learning rate $\alpha$ scaled by the second moment of the individual parameter gradients. We notice that when $g\rightarrow 0$, the ESS for DPAdam converges for the first moment gradient, which innately accounts for the clip bound one is training with. This may happen at later iterations, when the model is close to its minima and the gradients get close to zero.

Theorem 7.1 gives a closed form expression that ESS converges to. We can use this value in place of $\frac{\alpha}{\sqrt{\hat{v}_{t}}+\xi}$ in the update step from the inception of the learning process. Since the second-moment updates (e.g., $\hat{v}_{t}$) are not used anymore, removing them results in our new optimizer $\mathsf{DPAdamWOSM}$. We provide a pseudo-code for $\mathsf{DPAdamWOSM}$ in the appendix.

We evaluate $\mathsf{DPAdamWOSM}$ by running it alongside DPAdam and ADADP with the same hyperparameter grid in the appendix. For brevity, we show experiments on $\sigma=4$ and others appear in the supplement. In Figure 8, we show the maximum and median accuracy curves for all the optimizers. We display the median accuracy curves (shown in dotted), as an indicator of the quality of the entire pool of hyperparameter candidates for a given optimizer; which in this case is strictly over the choices of clip. The max lines for ADADP lies beneath DPAdam and $\mathsf{DPAdamWOSM}$ for all dataset except Adult. Also, the max accuracy line for $\mathsf{DPAdamWOSM}$ runs alongside DPAdam which means that it can perform as good as DPAdam throughout training. The median line for $\mathsf{DPAdamWOSM}$ also performs alongside DPAdam and in some cases is able to beat it (e.g, the median for $\mathsf{DPAdamWOSM}$ for MNIST-LR and MNIST-TLNN lies above the median line of DPAdam). This occurrence is seen because $\mathsf{DPAdamWOSM}$ uses the converged ESS from the first iteration of training.