Differentially Private Distributed Estimation and Learning

Given the adjacency structure $A$, at every round $t\in\{1,2,3,\ldots\}$, each agent $i$ receives a private signal $\bm{s}_{i,t}$ that is sampled from an exponential family distribution with the natural sufficient statistic $\xi(\cdot)\in\mathbb{R}$ and a common, unknown parameter $\theta$ belonging to a measurable set $\Theta$. The goal of the agents is to collectively estimate the common value of $\mathbb{E}_{\theta}\left[\xi(\bm{s})\right]$ by combining their samples. This is achieved through a consensus algorithm by forming an estimate $\bm{\nu}_{i,t}$ and exchanging it with their network neighbors in a distributed manner respecting the adjacency structure of $A$. The agents also want to control the information that is leaked about their signals and the estimates of their neighbors $\{\bm{\nu}_{j,t-1}\}_{j\in\mathcal{N}_{i}}$. They add noise $\bm{d}_{i,t}$ to their updates. The noise level should be high enough not to violate an ${\varepsilon}$ differential privacy budget. Briefly, we say that a mechanism $\mathcal{M}$ is ${\varepsilon}$-DP when for any pair of “adjacent” inputs (i.e., private signals or private signals and neighboring estimates), the logarithm of the probability ratio of any output in the range space of $\mathcal{M}$ being resulted from either of the adjacent inputs is bounded by ${\varepsilon}$: $|\log({\mathbb{P}[\mathcal{M}(\bm{s})\in R]}/{\mathbb{P}[\mathcal{M}(\bm{s}^{\prime})\in R]})|\leq{\varepsilon}$, for all adjacent pairs ($\bm{s}$ and $\bm{s}^{\prime}$) in the input domain and any subset $R$ of the output range space. Our specific notion of adjacency between the input pairs will be determined by the nature of information leaks, i.e., private signals or private signals and network neighborhoods, against which the exchanged estimates ($\bm{\nu}_{i,t}$) are being protected (Figure 1).

Our goal is to find differentially private aggregation mechanisms $\mathcal{M}$ with fast convergence guarantees that minimize CoP parametrized by noise distributions $\left\{\mathcal{D}_{i,t}\right\}_{i\in[n],t\geq 1}$ of the agents and subject to their ${\varepsilon}$ differential privacy budget constraints, i.e., “s.t. ${\varepsilon}$-DP”:

From now on, we will refer to this optimal value as the cost of privacy. In our analysis, the CoP and CoD are proportional to the signal and noise variance. The convergence rate also depends on the number of nodes $n$ and the spectral gap $\beta^{\star}$ of the doubly-stochastic adjacency matrix $A$, which dictates the convergence rate of $A^{t}$ to its limiting matrix $(\mathds{1}\mathds{1}^{T})/n$, as $t\to\infty$.

Subsequently, the noise distribution can be optimized by minimizing the weighted variances of the noise terms in the upper bounds subject to DP constraints. Our main results are summarized in Table 1. These consist of minimum variance unbiased estimation (Section 3) and online learning (Section 4) of expected values under (i) protection of the private signals (Signal DP), and (ii) protection of the private signals and the local neighborhoods (Network DP). The following Informal Theorem summarizes our theoretical contributions along with Table 1:

Informal Theorem. If $\Delta=\max_{\bm{s}\in\mathcal{S}}\left|\frac{d\xi(\bm{s}_{j})}{d\bm{s}_{j}}\right|$ is the global sensitivity of $\xi$ at time $t$, and $[a_{ij}]_{i,j=1}^{n}$ are the weights of the adjacency matrix, then the upper bound in CoP is minimized by the Laplace noise with parameters $\frac{\Delta}{{\varepsilon}}$ for the case of Signal DP, and parameters $\frac{\max\{\max_{j\in\mathcal{N}_{i}}a_{ij},\Delta\}}{{\varepsilon}}$ for the case of network DP.

Moreover, whenever the global sensitivity $\Delta$ is unbounded, something that happens to a variety of sufficient statistics $\xi$, we rely on the smooth sensitivity introduced by Nissim et al., (2007) and derive the same algorithms (but with parameters depending on the smooth sensitivity instead of $\Delta$), which achieve $({\varepsilon},\delta)$-DP, namely using the smooth sensitivity introduces a compromise in the ${\varepsilon}$-DP guarantee by a small information leakage probability $\delta$, relaxing the DP constraint for a mechanism $\mathcal{M}$ as follows: $\mathbb{P}[\mathcal{M}(\bm{s})\in R]\leq e^{{\varepsilon}}{\mathbb{P}[\mathcal{M}(\bm{s}^{\prime})\in R]}+\delta$, for all adjacent input pairs ($\bm{s}$ and $\bm{s}^{\prime}$) and all subsets $R$ of the output range space.

Our results relate to different bodies of literature across the engineering, statistics, and economics disciplines, and in what follows, we shall expand upon these relations.

The work most importantly connected to our work is the work of Rizk et al., (2023), which introduces a first-order DP method for distributed optimization over a network of agents. While their work presents a general first-order method for DP distributed optimization, which is suitable for a larger family of optimization problems that include MVUE, adapting their method to our task comes with important trade-offs in the quality of the estimation, as running their method results in significantly higher error (up to $1000\times$ more; see Figure 7) for the MVUE task due to the repeated inclusion of the signal in the belief updates. Moreover, contrary to ours, their method does not support the online learning regime. Finally, the concept of “graph-homomorphic noise” proposed in their paper is equivalent to the Network DP regime of our paper.

Statistical disclosure control of donated data, e.g., submitting recommendations to a public policy agency or submitting online product reviews to an e-commerce platform, requires safeguarding donors’ privacy against adversarial attacks where standard anonymization techniques are shown to be vulnerable to various kinds of identification (Barbaro et al., , 2006), linkage and cross-referencing (Sweeney, , 2015; Narayanan and Shmatikov, , 2008; Sweeney, , 1997), and statistical difference and re-identification attacks (Kumar et al., , 2007). Here, we propose to analyze the efficiency of distributed estimation where agents learn from each other’s actions protected by the gold standard of differential piracy (Dwork, , 2011) to optimize statistical precision against the privacy risks to data donors who engage in social learning.

DP can be implemented using central or local schemes. In centralized DP implementations, individual data are collected, and privacy noise is added to data aggregates, used, e.g., in the U.S. Census Bureau’s Disclosure Avoidance system (US Census, , 2020). In local implementations, DP noising is done as data is being collected. Local methods forego the need for any trusted parties and typically provide more fundamental protection that can withstand a broader range future infiltration, e.g., even a government subpoena for data cannot violate the privacy protection when collected data is itself subject to privacy noising — e.g., Google, LinkedIn, and Apple’s DP noising of their user data (Apple Differential Privacy Team, , 2017; Erlingsson, , 2014; Cardoso and Rogers, , 2022; Rogers et al., , 2020), cf. (Wilson et al., , 2020; Guevara, , 2019).

Regarding privacy and networks, Koufogiannis and Pappas, (2017) present a privacy-preserving mechanism to enable private data to diffuse over social networks, where a user wants to access another user’s data and provide privacy guarantees on the privacy leak, which depends on the shortest path between two users in the network. Alexandru and Pappas, (2021) study the problem of private weighted sum aggregation with secret weights, where a central authority wants to compute the weighted sum of the local data of some agents under multiple privacy regimes. Rahimian et al., 2023a ; Rahimian et al., 2023b study influence maximization using samples of network nodes that are collected in a DP manner.

Our work contributes to the above line of work by introducing a novel DP mechanism for distributed estimation and learning of exponential family distributions. Particularly, to the best of our knowledge, in the online learning regime, our algorithm introduces a novel weighting scheme that can protect both the individual signals and the neighboring beliefs, which can efficiently learn the expected value of the sufficient statistic. Moreover, we also provably derive the optimal distributions that minimize the convergence time of the algorithm and show that they are the Laplace distributions with appropriately chosen parameters.

Contrary to most of these methods, which are first-order optimization methods that are suitable to a large variety of losses compared to our method, our zero-order belief updates for the MVUE are simple, more efficient, and have significantly lower error than first-order methods (see Figure 7 for comparison with Rizk et al., (2023)). Moreover, our method can learn from data that arrive in an online way, whereas methods such as Rizk et al., (2023); Zhang et al., (2022) are offline. Finally, most of these approaches rely on SGD. In contrast, our method focuses more on the decision-theoretic and statistical problem of estimating the expected value of the sufficient statistics of signals generated by an exponential family distribution.

Let $\Theta$ be any measurable set, and in particular, not necessarily finite. Consider a network of $n$ agents and suppose that each agent $i\in[n]$ observes an i.i.d. samples $\bm{s}_{i}$ from a common distribution $\ell(\mathord{\cdot}|\theta)$ over a measurable sample space $\mathcal{S}$ (For simplicity in our proofs, we consider the simple case of 1D signals, i.e., $\mathcal{S}\subseteq\mathbb{R}$. Extending to multi-dimensional signals (i.e., $\mathcal{S}\subseteq\mathbb{R}^{s}$) is straightforward and considers the $\ell_{\infty}$ norm of the partial derivatives.). We assume that $\ell(\mathord{\cdot}|\theta)$ belongs to a one-parameter exponential family so that it admits a probability density or mass function that can be expressed as

where $\xi(\bm{s})\in\mathbb{R}$ is a measurable function acting as a complete sufficient statistic for the i.i.d. random samples $\bm{s}_{i}$, and $\alpha:\Theta\to\mathbb{R}$ is a mapping from the parameter space $\Theta$ to the real line $\mathbb{R}$, $\tau(\bm{s})>0$ is a positive weighting function, and $\varkappa(\alpha):=\ln\int\limits_{s\in\mathcal{S}}\tau(s)e^{\alpha\xi(s)}ds$ is a normalization factor known as the $\log$-partition function. In (2), $\xi(\mathord{\cdot})$ is a complete sufficient statistic for $\theta$. It is further true that $\sum_{i=1}^{n}\xi(\bm{s}_{i})$ is a complete sufficient statistic given the $n$ i.i.d. signals that the agents have received (Bickel and Doksum, , 2015, Section 1.6.1). In particular, any inferences that involve the unknown parameter $\theta$ based on the observed signals $\overline{\bm{s}}=(\bm{s}_{i})_{i\in[n]}$ can be equivalently performed given $\sum_{i=1}^{n}\xi(\bm{s}_{i})$. The agents aim to estimate the expected value of $\xi(\mathord{\cdot})$: $\bm{m}_{\theta}=\mathbb{E}\left[\xi(\bm{s}_{i})\right]$, with as little variance as possible. The Lehmann-Scheffé theory — cf. (Casella and Berger, , 2002, Theorem 7.5.1) — implies that any function of the complete sufficient statistic that is unbiased for $\bm{m}_{\theta}$ is the almost surely unique minimum variance unbiased estimator of $\bm{m}_{\theta}$. In particular, the minimum variance unbiased estimator of $\bm{m}_{\theta}$ given the initial data sets of all nodes in the network is given by: $\widehat{\bm{m}_{\theta}}=({1}/{n})\sum_{i=1}^{n}\xi(\bm{s}_{i})$.

For concreteness, we can consider a group of $n$ suppliers whose private signals consist of their contracts, sales orders, and fulfillment data. These suppliers would benefit from aggregating their private information to better estimate market conditions captured by the unknown parameter $\theta$, e.g., to predict future demand. However, sharing their private signals would violate the privacy of their customers and clients. In Section 2.2, we explain how the agents can compute the best (minimum variance) unbiased estimator of $\bm{m}_{\theta}$ using average consensus algorithms Olshevsky, (2014) that guarantee convergence to the average of the initial values without direct access to each other’s private signals.

We consider an undirected network graph and let the undirected network $\mathcal{G}(\mathcal{V}=[n],\mathcal{E})$ which corresponds to a Markov chain with a doubly-stochastic symmetric adjacency/transition matrix $A=[a_{ij}]_{i,j=1}^{n}$ with the uniform stationary distribution. For instance, such an adjacency matrix $A=[a_{ij}]_{i,j=1}^{n}$ can defined according to the Metropolis-Hastings weights (Boyd et al., , 2004): $a_{ij}=1/\max\{\deg(i),\deg(j)\}$ if $(j,i)\in\mathcal{E}$, and $[A]_{ij}=0$ otherwise for $i\neq j$; furthermore, $a_{ii}=1-\sum_{j\neq i}a_{ij}$. This choice of weights leads to a Markov chain where the stationary distribution is the uniform distribution (Boyd et al., , 2004), and the agents can set these weights locally based on their own and neighboring degrees without the global knowledge of the network structure. For choices of $A$ that yield the fastest mixing Markov chain (but may not be locally adjustable), see Boyd et al., (2004).

The mechanisms for convergence, in this case, rely on the product of stochastic matrices, similar to the mixing of Markov chains (cf. Levin et al., (2009); Shahrampour et al., (2015)); hence, many available results on mixing rates of Markov chains can be employed to provide finite time grantees after $T$ iteration of the average consensus algorithm for fixed $T$. Such results often rely on the eigenstructure (eigenvalues/eigenvectors) of the communication matrix $A$, and the facts that it is a primitive matrix and its ordered eigenvalues satisfy $-1<\lambda_{n}(A)\leq\lambda_{n-1}(A)\leq\ldots\leq\lambda_{1}(A)=1$, as a consequence of the Perron-Frobenius theory (Seneta, , 2006, Theorems 1.5 and 1.7).

Moreover, another mechanism considers learning the expected values in an online way where agents receive signals at every round and then update their estimate by averaging the estimates of their neighbors, their own estimate, and the new signals (see Algorithm 2).

The $1/t$ discounting provided in the above algorithm enables learning the expected values $\bm{m}_{\theta}=\mathbb{E}_{\theta}\left[\xi(\bm{s})\right]$ asymptotically almost surely with a variance that scales as $O(1/t)$; i.e., linearly in time. As shown in Rahimian and Jadbabaie, 2016b , the variance upper bound comprises two terms. The former term considers the rate at which the Markov chain with transition matrix $A$ is mixing and is governed by the spectral gap, i.e., the second largest magnitude of the eigenvalues of $A$. The latter term captures the diminishing variance of the estimates with the increasing number of samples gathered by all the agents in the network.

Now that we have the necessary background in distributed estimation, we present the two DP protection mechanisms that our paper considers: the Signal DP and the Network DP.

We present our first algorithm, which considers Minimum Variance Unbiased Estimation (MVUE). In this task, we aim to learn the MVUE of $\bm{m}_{\theta}$, i.e., to construct the estimate $\widehat{\bm{m}_{\theta}}=(1/n)\sum_{i=1}^{n}\xi(\bm{s}_{i})$ through local information exchange. The non-private version of this algorithm is presented in Algorithm 1 according to which, the agents start with some private signals $\{\bm{s}_{i}\}_{i\in[n]}$, calculate the sufficient statistics $\{\xi(\bm{s}_{i})\}_{i\in[n]}$ and then exchange these initial estimates with their local neighbors. Algorithm 1 converges to $\widehat{\bm{m}_{\theta}}$ in $t=O\left(\frac{\log(nM_{n}/\rho)}{\log(1/\beta^{\star})}\right)$ steps to $\rho$-accuracy, which depends on the number of nodes $n$, the maximum absolute value $M_{n}$ of the sufficient statistics, and the spectral gap $\beta^{\star}$ of the transition matrix $A$.

In its DP version, the algorithm proceeds similarly to the non-DP case, except each agent $i$ adds noise $\bm{d}_{i}$ to their sufficient statistic $\xi(\bm{s}_{i})$. As we show later, to respect ${\varepsilon}$-DP, the noise $\bm{d}_{i}$ depends on the agent’s realized signal $\bm{s}_{i}$, the sufficient statistics $\xi(\cdot)$, and the privacy budget ${\varepsilon}$. We provide the algorithm for the differentially private in Algorithm 3:

Regarding convergence, in Theorem 3.1, we prove that the convergence error is incurred due to two sources. The first source of error is the error due to the omniscient observer, which is the same as in the non-DP case, and the second source of error is incurred due to the privacy noise. Briefly, the latter term can be roughly decomposed to correspond to two terms: the former term is due to estimating the minimum variance unbiased estimator due to the noise, i.e. $(1/n)\sum_{i=1}^{n}\bm{d}_{i}$ and is vanishing with a rate proportional to $\log\left(\sum_{i=1}^{n}\mathbb{V}\left[\bm{d}_{i}\right]\right)/\log(1/\beta^{\star})$, and an additional non-vanishing term which is due to the mean squared error of $(1/n)\sum_{i=1}^{n}\bm{d}_{i}$ which corresponds to the sum of the variances of the signals.

To minimize the convergence error, it suffices to minimize each variance $\mathbb{V}\left[\bm{d}_{i}\right]$ subject to ${\varepsilon}$-DP constraints. By following recent results on the DP literature (cf. Koufogiannis et al., (2015)) we deduce that the variance minimizing distribution under ${\varepsilon}$-DP constraints are the Laplace distributions with parameters $\Delta/{\varepsilon}$, where $\Delta$ corresponds to the signal sensitivity (see below) and ${\varepsilon}$ is the privacy budget. We present our Theorem (proved in Section A.1):

Finally, we note that similarly, in the multi-dimensional case, the sensitivity would be $\Delta=\max_{\bm{s}\in\mathcal{S}}\left\|\nabla_{\bm{s}}\xi(\bm{s})\right\|_{\infty}$.

For example, we will consider the case of mean estimation in a log-normal distribution with known variance, a common task in many sensor networks, as we argue in Section 5. The sufficient statistic in this case is $\xi(\bm{s})=\log\bm{s}$, and the local sensitivity at distance $k$ can be computed to be $k/\bm{s}$. Therefore, the smooth sensitivity is (note the global sensitivity in this case is unbounded):

If agents have access to several signals, and the exact formula of $\xi$ is not known, the sensitivity can still be approximated via samples as shown in Wood and Zhang, (1996).

We consider the online learning framework where the agents aim to learn the common expected value $\bm{m}_{\theta}=\mathbb{E}_{\theta}\left[\xi(\bm{s})\right]$ of the sufficient statistics of their signal distributions. In this regime, the agents observe signals $\bm{s}_{i,t}$ at every time $t\geq 1$ and update their estimates $\bm{\nu}_{i,t}$ by weighing the information content of their most recent private signals, $\xi(\bm{s}_{i,t})$, their previous estimates, $\bm{\nu}_{i,t-1}$, and the estimates of their neighbors, $\{\bm{\nu}_{j,t-1}\}_{j\in\mathcal{N}_{i}}$. The mechanisms that we analyze in this section will accommodate two types of privacy needs with convergence and ${\varepsilon}$-DP guarantees for the agents:

1. $\mathcal{M}^{S}$:

   Signal DP (Algorithm 4). Here the agent adds noise to privatize their belief $\bm{\nu}_{i,t}$ with respect to their signal. To assert the consistency of the estimator, the agent averages her previous estimate and the estimates of her neighbors with weight $(t-1)/t$ and her signal with weight $1/t$, similarly to Algorithm 2. Therefore, the local sensitivity of this mechanism is the sensitivity of the sufficient statistic weighted by $1/t$ and equals $\Delta\nu_{i,t}^{\mathcal{M}^{S}}=\frac{\Delta}{t}$.

2. $\mathcal{M}^{N}$:

   Network DP (Algorithm 5). Here we consider the protection of the agent’s signals $\bm{s}_{i,t}$ together with their local neighborhood, namely the neighboring beliefs $\left\{\bm{\nu}_{j,t-1}\right\}_{j\in\mathcal{N}_{i}}$. We note that when deciding the noise level at time $t$, we do not need to include the agent’s own belief from time $t-1$ in the DP protection; $\nu_{i,t-1}$ including all the private signals up to and including at time $t-1$ remain protected by the property of the post-processing immunity. If we were to use Algorithm 4 for Network DP the sensitivity of the mechanism would be $\max\left\{\frac{\Delta}{t},\frac{t-1}{t}\max_{j\in\mathcal{N}_{i}}a_{ij}\right\}$ which approaches 1 as $t\to\infty$ and violates the consistency of the estimator $\bm{\nu}_{i,t}$. For this reason, we need to adapt the weighting scheme of Algorithm 4 to be consistent and respect Network DP. We give more details of the altered algorithm (Algorithm 5) in Section 4.3.

Here we will analyze the performance of Algorithm 4 where agents only protect their signals and present the corresponding error analysis. The error of the estimates $\bm{\nu}_{i,t}$ compared to the expected value $\bm{m}_{\theta}=\mathbb{E}_{\theta}\left[\xi(\bm{s})\right]$, again consists of two terms; one due to decentralization and the statistics of the signals themselves (CoD), and another due to the variances of the added DP noise variables (CoP). Each of these two terms decays at a rate of $1/\sqrt{t}$. They, in turn, can be bounded by the sum of two terms: a constant that is due to the principal eigenvalue of $A$ and represents the convergence of the sample average to $\bm{m}_{\theta}$, and $n-1$ terms due to $|\lambda_{i}(A)|$ for $2\leq i\leq n$ which dictate the convergence of the estimates $\bm{\nu}_{i,t}$ to their sample average. The latter depends on the number of nodes $n$ and the spectral gap $\beta^{\star}$ of matrix $A$. We formalize our results as follows (proved in Section A.3):

Above, we briefly discussed why a learning algorithm that puts weights $\frac{t-1}{t}$ on the network predictions and $\frac{1}{t}$ on the private signal would not work (and in fact, the dynamics become divergent in that case). In Algorithm 5, we present a different learning scheme that uses weights $\frac{1}{t}$ for the private signals and neighboring observations and $1-\frac{1}{t}(2-a_{ii})$ for the previous beliefs of the agent. The motivation behind this learning scheme is that the sensitivity is now going to be $\Delta\bm{\nu}_{i,t}^{\mathcal{M}^{N}}=\frac{1}{t}\max\left\{\max_{j\in\mathcal{N}_{i}}a_{ij},\Delta\right\}$ which goes to zero, instead of $\max\left\{\frac{t-1}{t}\max_{j\in\mathcal{N}_{i}}a_{ij},\frac{1}{t}\Delta\right\}$ which approaches 1. The drawback is that the added self-weight on one’s own beliefs at every time step will slow down the mixing time and convergence. In the sequel, we present Algorithm 5 and analyze its performance.

We can write the above system in matrix notation as $\overline{\bm{\nu}}_{t}=\left(B(t)-\frac{1}{t}I\right)\overline{\bm{\nu}}_{t-1}+\frac{1}{t}\overline{\bm{\xi}}_{t}+\frac{1}{t}\overline{\bm{d}}_{t}$ where $b_{ii}(t)=1-\frac{1}{t}(1-a_{ii})$ and $b_{ij}(t)=\frac{1}{t}a_{ij}$ for all $j\neq i$. First, we study the convergence of Algorithm 5 when no noise is added, i.e., $\overline{\bm{d}}_{t}=0$. Note that similarly to Algorithm 4, the error term is comprised of two terms, one owing to the principal eigenvalues of $C(t)=B(t)-\frac{1}{t}I$, i.e., $\lambda_{1}(C(t))=1-1/t$ which controls the convergence of the sample average of the estimates to $\bm{m}_{\theta}$, and $n-1$ terms due to $|\lambda_{i}(C(t))|$ which control the convergence of the estimates $\bm{\nu}_{i,t}$ to their sample average.