Differentially Private Federated Learning:
Servers Trustworthiness, Estimation, and Statistical Inference

Federated learning is an efficient approach for training machine learning models on distributed networks, such as smartphones and wearable devices, without moving data to a central server (konevcny2016federated, ; kairouz2021advances, ; li2020federated, ). Since its proposal in (mcmahan2017communication, ), federated learning has gained significant attention in both practical and theoretical machine learning communities. One of the key attractions of federated learning is its ability to provide a certain level of data privacy by keeping raw data on local machines. However, without specific design choices, there are no formal privacy guarantees. To fully exploit the benefits of federated learning, researchers have introduced the concept of differential privacy (abadi2016deep, ; dwork2018privacy, ; dwork2006calibrating, ; dwork2014algorithmic, ; dwork2017exposed, ) to quantify the exact privacy level in federated learning. A series of research papers have focused on federated learning with differential privacy, applying various algorithms and methods (hu2020personalized, ; truex2020ldp, ; wei2020federated, ). Despite these efforts, there remains a significant gap between practical usage and statistical guarantees, particularly in the high-dimensional setting with sparsity assumptions, where theoretical results for the optimal rate of convergence and statistical inference results are largely missing.

In this paper, we focus on studying the estimation and inference problems in the federated learning setting under differential privacy, particularly in the high-dimensional regime. In federated learning, there are several local machines containing data sets from different sources, and a central server to coordinate all local machines to train learning models collaboratively. We present our key results in two major settings for privacy and federated learning. In the first setting, we consider an untrusted central server (lowy2021private, ; wei2020federated, ; hu2020personalized, ) where each machine sends only privatized information to the central server. For example, when using smartphones, where users may not fully trust the server and do not want their personal information to be directly updated on the remote central server. In the second setting, we consider a trusted central server where each machine sends raw information without making it private. (mcmahan2022federated, ; geyer2017differentially, ; mcmahan2017learning, ) For example, in different hospitals, patient data may not be shared among hospitals to protect patient privacy, but they can all report their data to a central server, such as a non-profit organization or an institute, to gain more information and publish statistics on certain diseases.

In the first part of our paper, we demonstrate that under the assumption that the central server is untrusted, the optimal rate of convergence for mean estimation is O($sd/(mn\epsilon^{2})$), where $m$ is the number of local machines and each containing $n$ data points, $d$ is the parameter of interest, $s$ is the sparsity level, and $\epsilon$ is the privacy parameter. As commonly assumed in high-dimensional settings where the dimension is comparable or even larger than the number of data, such an optimality result shows the incompatibility of untrusted central server setting and high-dimensional statistics. As a result, we can only hope to get a good estimation under the trusted central server setting in the high-dimensional regime.

In the second part of the paper, we consider the case of a trusted central server and design algorithms that allow for accurate estimations and obtain a near-optimal rate of convergence up to logarithm factors. We also present statistical inference results, including the construction of coordinate-wise confidence intervals with privacy guarantees, and the solution to conduct simultaneous inference privately. This will assist in hypothesis testing problems and construction of confidence intervals for a given subset of indices of a vector simultaneously in high-dimensional settings. We emphasize that our algorithms for estimation and inference are suited for practical purposes, considering its capacity to (1) leverage data from multiple devices to improve machine learning models and (2) draw accurate conclusions about a population from a sample while preserving individual privacy. For instance, in healthcare, we could combine patient data from multiple hospitals to develop more accurate models for disease diagnosis and treatment, while ensuring that patient privacy is protected. We summarize our major contributions as follows:

• •

  For the untrusted central server setting, we provably show that federated learning is not suited for high-dimensional mean estimation problems by providing the optimal rate of convergence under the untrusted central server constraints. This suggests us to consider a trusted central server setting to utilize federated learning for such problems.

• •

  For the trusted central server setting, we design novel algorithms to achieve private estimation with federated learning. We first consider the estimation in homogeneous federated learning setting and then we extend it to a more complicated heterogeneous federated learning setting. We also provide a sharp rate of convergence for our algorithm in both settings.

• •

  In addition, we consider statistical inference problems in both homogeneous and heterogeneous federated learning settings. We provide algorithms for coordinate-wise and simultaneous confidence intervals, which are two common inference problems in high-dimensional statistics. It is worth mentioning that our proposed methods for high-dimensional differentially private inference problems are novel and unique, which has not been developed even for the single-source and non-federated learning setting. Theoretical results show that our proposed confidence intervals are asymptotically valid, supported by simulations.

In the literature, several works focused on designing private algorithms in federated learning/distributed learning based on variants of stochastic gradient decent algorithms. (agarwal2018cpsgd, ) proposed a communication efficient algorithm, CP-SGD algorithm for learning models with local differential privacy (LDP). (erlingsson2020encode, ) proposed a distributed LDP gradient descent algorithm by applying LDP on gradients with ESA framework (bittau2017prochlo, ). (girgis2021shuffled, ) extended works on LDP approach for federated learning and proposed a distributed communication-efficient LDP stochastic gradient descent algorithm through shuffled model and analyzed the upper bound of the convergence rate. However, the trade-off between statistical accuracy and the privacy cost has not been considered in these works.

In the distributed settings, the trade-off between statistical accuracy and information constraints has been discussed in various papers. Two common types of information constraints are communication constraints and privacy constraints. We refer to (zhang2013information, ; braverman2016communication, ; han2018geometric, ; barnes2020lower, ; garg2014communication, ) for more discussions on communication constraints, considering the situation where the bits of the information during communication have constraints.

A series of work discusses the trade-off between accuracy and privacy in high-dimensional and non-federated learning problems, including top-$k$ selection (steinke2017tight, ), sparse mean estimation (cai2019cost, ), linear regression (cai2019cost, ; talwar2015nearly, ), generalized linear models (cai2020cost, ), latent variable models (zhang2021high, ). However, the discussion on privacy constraints in the distributed settings are still largely lacking. Among the existing works, most of them focus on the local differential privacy (LDP) constraint. In (barnes2020fisher, ), the mean estimation under $\ell_{2}$ loss for Gaussian and sparse Bernoulli distributions are discussed. (duchi2019lower, ) discussed the lower bounds under LDP constraints in the blackboard communication model for mean estimation of product Bernoulli distributions and sparse Gaussian distributions. acharya2020general proposed a more general approach to combine both communication constraints and privacy constraints. Compared with previous works, we focus on the problem where there are $n$ data points on each machine. Our interest lies in the $(\epsilon,\delta)$-DP instead of LDP, which is a weaker constraint containing broader settings. We further note that, compared with the blackboard communication model (braverman2016communication, ; garg2014communication, ), in the federated learning setting, we assume that the existence of a central server and that each server is only allowed to communicate with the central server. This setting enables us to enhance more privacy.

When we are finalizing this paper⁴⁴4An initial draft of this paper was published as a Ph.D. dissertation in 2023 zhethesis ., we realized an independent and concurrent work li2024federated . li2024federated also considers differentially private federated transfer learning under high-dimensional sparse linear regression model. Namely, they proposed a notion of federated differential privacy that allows multiple rounds of $(\epsilon,\delta)$-differentially private transmissions between local machines and the central server, and provides algorithms to filter out irrelevant sources, and exploit information from relevant sources to improve the performance of estimation of target parameters. We differntiate our research with their paper as follows: (1) While they consider differentially private federated learning under untrusted server setting, we deal with both trusted and untrusted server settings. We also highlight a fundamental difficulty of pure $\epsilon$-differentially private estimation under untrusted central server settings in federated learning by establishing a tight minimax lower bound, and resort to trusted server settings for estimation and inference problems. (2) While their investigation centers on differentially private estimation within a federated transfer learning framework—specifically focusing on parameter estimation for a target distribution using similar source data—our work focuses on private estimation and inference for parameters that are either common across all participating machines, or vary across different machines.

We also cite papers that provided us inspirations for the design of our proposed algorithms and methods. (javanmard2014confidence, ) introduces a de-biasing produce for the statistical inference problems. (li2020transfer, ) considers the transfer learning problem in high-dimensional settings, which enables us to combine information from other sources to benefit the estimation problems. Such idea could be adopted in the hetergeneous federated learning problems. For the simultaneous inference problems, we refer to (zhang2017simultaneous, ; yu2022distributed, ), which discussed how to conduct simultaneous inference for high-dimensional problems.

Notation. We introduce several notations used throughout the paper. Let $\bm{v}=(v_{1},v_{2},\dots,v_{d})^{\top}\in\mathbb{R}^{d}$ represent a vector. Given a set of indices $\mathcal{S}$, $\bm{v}_{\mathcal{S}}$ refers to the components of $\bm{v}$ corresponding to the indices in $\mathcal{S}$. The $\ell_{q}$ norm of $\bm{v}$, for $1\leq q\leq\infty$, is given by $\|\bm{v}\|_{q}$, whereas $\|\bm{v}\|_{0}$ represents the number of non-zero elements in $\bm{v}$, also called as its sparsity level.

We use $m$ to indicate the number of machines, $n$ for the number of samples per machine, $d$ for the dimensionality of vectors, and $s$ for their sparsity level. The total number of samples across all machines is denoted by $n_{0}=m\cdot n$. Additionally, we define the truncation function $\Pi_{T}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}$, which projects a vector onto the $\ell_{\infty}$ ball of radius $T$ centered at the origin.

For a matrix $\Sigma$, $\max_{\|\bm{v}\|_{2}=1,\|\bm{v}\|_{0}\leq s}\bm{v}^{\top}\Sigma\bm{v}$ and $\min_{\|\bm{v}\|_{2}=1,\|\bm{v}\|_{0}\leq s}\bm{v}^{\top}\Sigma\bm{v}$ denote the largest and smallest $s$-restricted eigenvalues of $\Sigma$, denoted as $\mu_{s}(\Sigma)$ and $\nu_{s}(\Sigma)$, respectively.

For sequences $a_{n}$ and $b_{n}$, $a_{n}=o(b_{n})$ implies $a_{n}/b_{n}\rightarrow 0$ as $n$ grows, $a_{n}=O(b_{n})$ signifies that $a_{n}$ is upper bounded by a constant multiple of $b_{n}$, and $a_{n}=\Omega(b_{n})$ indicates that $a_{n}$ is lower bounded by a constant multiple of $b_{n}$, where constants are independent of $n$. The notation $a_{n}\asymp b_{n}$ denotes that $a_{n}$ is both upper and lower bounded by constant multiples of $b_{n}$.

In this work, we often use symbols $c_{0},c_{1},m_{0},m_{1},C,C^{\prime},K,K^{\prime}$ to represent universal constants. Their specific values may vary depending on the context, but they are independent from other tunable parameters.

We start form the basic concepts and properties of differential privacy (dwork2006calibrating, ). The intuition behind differential privacy is that a randomized algorithm produces similar outputs even when an individual’s information in the dataset is changed or removed, thereby preserving the privacy of individual data. The formal definition of differential privacy is given below.

In the above definition, the two parameters $\epsilon,\delta$ control the privacy level. From the definition, with smaller $\epsilon$ and $\delta$, the outcomes given adjacent $\bm{X}$ and $\bm{X}^{\prime}$ become closer, making it harder for an adversary to distinguish if the original dataset is $\bm{X}$ or $\bm{X}^{\prime}$, indicating the privacy constraint becomes more stringent. Furthermore, when $\delta=0$, we could use $\epsilon$-differentially private as the abbreviation of $(\epsilon,0)$-differentially private.

In the rest of this section, we introduce several useful properties of differential privacy and how to create a differential private algorithm from non-private counterparts. One common strategy is through noise injection. The scale of noise is characterized by the sensitivity of the algorithm:

We then introduce two mechanisms. For algorithms with finite $\ell_{1}$-sensitivity, we add Laplace noises to achieve differential privacy, while for $\ell_{2}$-sensitivity, we inject Gaussian noises.

The post-processing and composition properties are two key properties in differential privacy, which enable us to design complicated differentially private algorithms by combining simpler ones. Such properties are pivotal in the design of algorithms in later chapters.

We also mention NoisyHT algorithm (Algorithm 1) introduced by dwork2018differentially , which stands for the noisy hard-thresholding algorithm. The algorithm aims to pursue both sparsity of the output and privacy at the same time.

In the last step, $P_{S}(\bm{u})$ denotes the operator that makes $\bm{u}_{S^{c}}=0$ while preserving $\bm{u}_{S}$. This algorithm could be seen as a private top-k selection algorithm, which helps build our proposed algorithm in later section.

Specifically, when the sparsity $s$ is chosen to be $1$, the algorithm outputs the maximum element chosen after a single iteration in the private manner. We refer this special case as the Private Max algorithm, which is implemented in Algorithm 7 used for simultaneous inference.

Federated learning introduced in (mcmahan2017communication, ) is a technique designed to train a machine learning algorithm across multiple devices, without exchanging data samples. A central server coordinates the process, with each local machine sending model updates to be aggregated centrally. Figure 1 illustrates the basic concept of federated learning

One characteristic of federated learning is that the training of machine learning models occurs locally, and only parameters and updates are transferred to the central server and shared by each node. Specifically, communication between local machines and the server is bidirectional: machines send updates to the central server, and in return, they receive aggregated information after processing. Communication among local machines is prohibited to prevent privacy leakage. Intuitively, federated learning inherently provides a certain level of privacy.

Although without rigorous definitions, there are two main branches of central server settings in federated learning: the untrusted central server setting and the trusted central server setting (lowy2021private, ; wei2020federated, ; mcmahan2022federated, ; geyer2017differentially, ). In the first setting, where the central server is untrusted, each piece of information sent from the machine to the central server should be differentially private. In the second setting, we assume a trusted central server exists. In this scenario, it is safe to send raw information from the machine to the central server without additional privacy measures. However, to prevent information leakage among local machines, the information sent back from the server should also be differentially private.

Another key aspect of federated learning is that the datasets on each local machine are commonly not independent and identically distributed (i.i.d.). This allows federated learning to train on heterogeneous datasets, aligning with practical scenarios where the datasets on different machines are typically diverse and their sizes may also vary. We will demonstrate that federated learning can efficiently estimate the local model when models on different local machines differ but share some similarities, a concept we refer to as heterogeneous federated learning in Section 5.

In this paper, we assume that there exists a central server and $m$ local machines. We denote the data on these machines by $\bm{X}_{1},\bm{X}_{2},\bm{X}_{3},\dots,\bm{X}_{m}$, respectively, with $\bm{X}_{i}\in\mathbb{R}^{n_{i}\times d}$. On any machine $i=1,2,\dots,m$, there are $n_{i}$ data points $\bm{X}_{i}=[\bm{X}_{i1},\bm{X}_{i2},\dots,\bm{X}_{in_{i}}]$. For simplicity, we assume that there are equal data points $n=n_{1}=n_{2}=\dots=n_{i}$ for each machine. We note that the result could be easily generalized to cases where the sample sizes on each machine differ.

We consider both untrusted and trusted central server settings. For the untrusted setting, we require that the information sent from local machines to the server is private. In this scenario, we show that in the high-dimensional setting, even with sparsity assumptions, it is impossible to achieve small estimation error when the central server is untrusted. In the trusted setting, we consider the high-dimensional linear regression problem $\bm{Y}=\bm{X}{\bm{\beta}}+\bm{W}$ with $s$-sparse ${\bm{\beta}}$. We will first study the case where all machines share the same ${\bm{\beta}}$, (referred to homogeneous federation learning,) and then study a more general case where models on different machines are not equal, but share certain similarities (referred to heterogeneous federation learning.) We show that our algorithm can adapt to such similarity—with larger similarity, the algorithm achieves a faster rate of convergence.

In this section, we consider the setting of a trusted central server, where local machines fully trust the central server and send unprivatized information to it without implementing privacy measures. However, when the central server sends information back to the local machines, it must ensure that this information is privatized to avoid any privacy leakage across local machines.

In this subsection, we first focus on the statistical estimation problems in this setting and then develop inference results in the next subsection. More specifically, our primary focus is on the linear regression problem in a high-dimensional setting, where the ground truth, denoted as ${\bm{\beta}}$, is a sparse $d$-dimensional vector. We initially study the simpler case in this section, where the underlying generative models for each local machine are identical, which we refer to as the homogeneous federated learning setting. A more complicated heterogeneous setting will be discussed in the following section. Specifically, we consider the following high-dimensional linear regression model:

where we assume $\bm{W}$ is the error term whose coordinates are independent and following sub-Gaussian distribution with variance proxy $\sigma^{2}$, denoted by $W_{i}\sim\text{subG}(\sigma^{2})$. $\bm{X}$ is a random matrix whose rows are following sub-Gaussian distribution with a covariance matrix $\bm{\Sigma}$.

We first introduce the parameter estimation algorithm under differentially private federated settings with a trusted central server.

In Step 1 of Algorithm 2, the information computed on each local machine is transmitted to the central server. The second step involves calculations performed at the central server. Prior to sending the information back to the local machines, it undergoes privacy preservation through the application of the NoisyHT algorithm, as introduced in Algorithm 1. Subsequently, the local machine updates its estimation based on the information received from the central server.

We compare Algorithm 2 with Algorithm 4.2 in (cai2019cost, ), which addresses the private estimation in linear regression under non-federated learning settings. Unlike the latter, our algorithm does not transmit all data points to the central server. Instead, we calculate the gradient updates locally on each machine and send only these local gradients to the server. This design enhances privacy protection, as the original data remains visible only on the local machine and is not exposed externally. Furthermore, this approach of gradient updates also reduces communication costs by transmitting only a $d$-dimensional vector from each local machine for the gradient update. Previous research has also considered non-private distributed methods for linear regression problems, such as (lee2017communication, ; zhang2012communication, ). Our algorithm, however, ensures differential privacy. In practice, the sparsity level $s$ can be determined using a private version of cross-validation, while other parameters may be pre-chosen based on our theoretical analysis.

In this subsection, we focus on statistical inference problems in the homogeneous federated learning setting, such as constructing coordinate-wise confidence intervals for parameters and performing simultaneous inference. To begin, we develop a method for constructing coordinate-wise confidence intervals, for example, for the $k$-th index of ${\bm{\beta}}$, $\beta_{k}$. However, it is important to note that the output of Algorithm 2 is biased due to hard thresholding. To overcome this bias, we employ a de-biasing procedure, a common technique in high-dimensional statistics, as demonstrated in previous studies (javanmard2014confidence, ). This procedure involves approximating the $k$-th column of the precision matrix $\bm{\Theta}=\bm{\Sigma}^{-1}$ to construct confidence intervals for each $\beta_{k}$. Subsequently, we focus on obtaining an estimate of the precision matrix in a private manner.

The structure of Algorithm 3 is similar to Algorithm 2, as both adopt an iterative communication between the central server and the local machines; the information is initially transmitted from the local machines to the server, then, the central server performs calculations and use the NoisyHT algorithm (Algorithm 1) to ensure the privacy of the information. Subsequently, each local machine updates the gradient and progresses to the next iteration. The primary distinction between two algorithms lies in the computation of the gradient on each machine.

Denote the output of Algorithm 2 as $\hat{\bm{\beta}}$ and the output of Algorithm 3 as $\hat{\bm{\Theta}}_{k}$. Then the de-biased differentially private estimator of $\beta_{k}$ is given by

where $\bm{g}_{i}=(1/n)\sum_{j=1}^{n}(\bm{X}_{ij}^{\top}\hat{\bm{\beta}}-\Pi_{R}(y_{ij}))\bm{X}_{ij}$, and $E_{k}$ is the injected random noise to ensure privacy, following a Gaussian distribution $N(0,8\Delta_{1}^{2}\log(1.25/\delta)/(n^{2}m^{2}\epsilon^{2}))$, where $\Delta_{1}=\sqrt{s}c_{1}c_{x}R+sc_{0}c_{1}c_{x}^{2}$ with some constants $c_{0},c_{1},c_{x}$ defined later.

The debiased estimator in (4.1) enables us to construct a differentially private confidence intervals. Although the variance $\sigma$ of the error term $\bm{W}$ in the linear regression model is usually unknown, we can estimate $\sigma$ from the data in a private manner. The estimation is based on the residual term between the response $\bm{Y}$ and the fitted value $\bm{X}\hat{{\bm{\beta}}}$. We summarize the method to estimate $\sigma$ in the private federated learning setting in Algorithm 4.

When examining the convergence rates of $\hat{{\bm{\beta}}}$ and $\hat{\bm{\Theta}}_{k}$ in Theorem 3, we observe the crucial roles of the largest and smallest restricted eigenvalues of $\bm{\Sigma}$. Since these eigenvalues directly influence the construction of confidence intervals and cannot be directly obtained from the data, their private estimation becomes essential. Below, we outline an algorithm to estimate the largest restricted eigenvalue, $\mu_{s}(\Sigma)$. To estimate the smallest restricted eigenvalue, $\nu_{s}(\Sigma)$, the same algorithm can be used by modifying Step 4 from “argmax” to “argmin”.

Based on Algorithms 4 and 5, we provide a constuction for coordinate-wise confidence intervals in Algorithm 6.

So far we focused on constructing confidence intervals for individual coordinates of the parameter vector $\bm{\beta}$. However, in high-dimensional settings, we are often interested in group inference problem, where we test hypotheses involving multiple coordinates simultaneously. Specifically, we consider the problem of testing the null hypothesis given by

against the alternative hypothesis,

for at least one $k\in G$, where $G$ is a subset of all coordinates $\{1,2,\dots,d\}$ and we allow $|G|$ to be the same order as $d$. Additionally, we also construct simultaneous confidence intervals for all coordinates in $G$. Note that the problem discussed above are common in high-dimensional data analysis, with applications such as multi-factor analysis of variance (hothorn2008simultaneous, ), additive modeling (wiesenfarth2012direct, ). Previous research works have discussed similar problems in the non-private setting, including (chernozhukov2013gaussian, ; zhang2017simultaneous, ; yu2022distributed, ).

To address the problem, simultaneous inference can be conducted using a test statistic

Major challenges of simultaneous inference in a private federated learning setting include: (1) minimizing the communication cost from local machines to the server while retaining all data on the local machines, and (2) ensuring the privacy of the procedure, which necessitates a tailored privacy-preserving mechanism at each step of the algorithm.

In our framework, we propose an algorithm based on the bootstrap method. As previously mentioned, to build confidence intervals, our interest lies in the statistic computed by the maximum coordinate of $\hat{\beta}_{k}^{\text{u}}-\beta_{k}$ over $G$. By decomposing this statistic, we obtain a term $\frac{\sigma}{\sqrt{mn}}\sum_{i=1}^{m}\sum_{j=1}^{n}\hat{\bm{\Theta}}\bm{X}_{ij}({\mathbf{y}}_{ij}-\bm{X}_{ij}^{T}{\bm{\beta}})$. To determine the distribution of this term, we bootstrap the residuals ${\mathbf{y}}_{ij}-\bm{X}_{ij}^{T}{\bm{\beta}}$.

We outline the algorithm as follows: we first estimate $\hat{{\bm{\beta}}}$ and $\hat{\bm{\Theta}}_{k}$ using Algorithm 2 and 3, respectively. Accordingly, by stacking $\hat{\bm{\Theta}}_{k}$ for all $k$, we get an estimator of the precision matrix $\bm{\hat{}}{\bm{\Theta}}$. The details are provided in Algorithm 7.

On line 4 of Algorithm 7, we employ the Private Max algorithm, which we mentioned earlier as a variation of NoisyHT algorithm (Algorithm 1) by directly picking $s=1$, to obtain the maximum element in a vector in a private manner. It is also important to note that the Private Max algorithm is applied to a subset of $G$. After presenting the algorithm, we denote $M$ as:

$M$ is used as the statistic for inference problems later.

As previously mentioned, we can easily construct a simultaneous confidence interval for each $k\in G$ by:

where $C_{U}(\alpha)$ is obtained from our algorithm with prespecified $\alpha$. We can similarly perform hypothesis testing; first calculate the test statistic and obtain $C_{U}(\alpha)$ from our algorithm with prespecified $\alpha$, then reject if the statistic lies in the rejection region.

In this subsection, we provide theoretical guarantee for the algorithms and methods discussed in the previous subsections. Before proceeding, we outline key assumptions concerning the design matrix $\bm{X}$, precision matrix $\bm{\Theta}$, and the true parameter $\bm{\beta}$ of the linear regression model, which are essential for our subsequent analyses.

• (P1)

  Parameter Sparsity: The true parameter vector ${\bm{\beta}}$ satisfies $\|{\bm{\beta}}\|_{2}<c_{0}$ for some constant $0<c_{0}<\infty$ and $\|{\bm{\beta}}\|_{0}\leq s_{0}^{*}=o(n)$.

• (P2)

  Precision matrix sparsity: For each column of the precision matrix $\bm{\Theta}_{k}$, $k=1,2,\dots,d$, it satisfies that $\|\bm{\Theta}_{k}\|_{2}<c_{1}$ for some constant $0<c_{1}<\infty$ and $\|\bm{\Theta}_{k}\|_{0}\leq s_{1}^{*}=o(n)$.

• (D1)

  Design Matrix: for each row of the design matrix $\bm{X}$, denote by $\bm{x}$, $\bm{x}\bm{\Sigma}^{-1/2}$ is sub-Gaussian with sub-Gaussian norm $\kappa:=\|\bm{\Sigma}^{-1/2}\bm{x}\|_{\psi_{2}}$.

• (D2)

  Bounded Eigenvalues of the covariance matrix: For the covariance matrix $\bm{\Sigma}=\mathbb{E}\bm{x}\bm{x}^{\top}$, there exists a constant $0<L<\infty$ such that $0<1/L<\lambda_{\min}(\bm{\Sigma})\leq\lambda_{\max}(\bm{\Sigma})<L$.

The above assumptions (P1) and (P2) bounds the $\ell_{2}$ norm and $\ell_{0}$ norm of the parameters ${\bm{\beta}}$ and $\bm{\Theta}_{k}$, and assumption (D1) guarantees that each row of $X$ follows a sub-Gaussian distribution, and assumption (D2) requires the covariance matrix has bounded eigenvalues. These assumptions are commonly used for theoretical analysis of differentially private algorithms and debiased estimators (cai2020cost, ; cai2019cost, ; javanmard2014confidence, ).

With assumptions (P1)-(D2), we analyze the algorithms we presented. We begin with the estimation problem and provide a rate of convergence of $\hat{\bm{\beta}}$ and $\hat{\bm{\Theta}}_{k}$.

The upper bound of Algorithm 3 in (4.3) can be interpreted as follows. The first term represents the statistical error, while the second term accounts for the privacy cost. Furthermore, the result is comparable to that of Theorem 4.4 in (cai2019cost, ), which addresses private linear regression in a non-federated setting. This comparison suggests that the federated learning approach does not affect the convergence rate adversely; instead, it allows us to leverage the benefits of federated learning. We also note that the advantages of federated learning will be further explored in the heterogeneous federated learning setting, which will be discussed in the next chapter.

The remainder of this subsection presents the theoretical results for the inference problem. We begin with the construction of coordinate-wise confidence intervals. As mentioned before, $\sigma$ is usually unknown and we estimate $\sigma$ in a private manner, presented in Algorithm 4. Lemma 4.1 states the statistical guarantee of our algorithm.

Next, we consider a simplified version of the confidence interval, where the privacy cost is dominated by the statistical error. In this scenario, we assume that the privacy level is relatively low and the privacy constraints are loose, meaning that the privacy parameters $\epsilon$ and $\delta$ are relatively large, allowing for nearly cost-free estimation. We present our result in the following theorem.

Theorem 4 assumes that the privacy cost is dominated by the statistical error. However, when the privacy constraint is more stringent with small privacy parameters $\epsilon$ and $\delta$, the privacy cost may be larger than the statistical error. In this scenario, we generalize Theorem 4 to analyze Algorithm 6. We note that the largest and smallest restricted eigenvalues of $\hat{\bm{\Sigma}}$ also need to be estimated by Algorithm 5. Lemma 4.2 quantifies the estimation error of the largest restricted eigenvalue of $\hat{\bm{\Sigma}}$.

We then present a theoretical result for the confidence interval in a more general case in Theorem 5.

Compared to the non-private counterpart in (javanmard2014confidence, ), we claim that our confidence interval has a similar form but with additional noise injected to ensure privacy. When the noise level is low, the confidence interval closely approximates the non-private counterpart, allowing us to nearly achieve privacy without incurring additional costs. Furthermore, when the privacy level is high, the confidence interval has a larger length to attain the same confidence level.

Finally, for the simultaneous inference problems, we demonstrate that $\alpha$-quantile of statistic $M$ in (4.2) is close to the $\alpha$-quantile of $U$ calculated in Algorithm 7 for each $\alpha\in(0,1)$ using the bootstrap method. The next theorem states the statistical properties of Algorithm 7.