Differentially private sliced inverse regression in the federated paradigm

We revisit the concept of differential privacy before moving on. Intuitively, the privacy level set $(\delta,\epsilon)$ in definition (1) measures how much information $\mathcal{M}(\cdot)$ reveals about any individual record in the dataset $\mathcal{D}$, where the smaller value of $\epsilon$ or $\delta$ imposes the more stringent privacy constraint. This definition leads to some nice properties of differential privacy. The two most useful ones are listed as follows, which provide a convenient way to construct complex differentially private algorithms.

A standard approach for developing differentially private algorithms involves introducing random noises to the output of their non-private counterparts, see (Dwork et al., 2014). One such technique, known as the Gaussian mechanism, employs independently and identically distributed (i.i.d.) Gaussian noise. The noise’s scale is governed by the $l_{2}$ sensitivity of the algorithm, which quantifies the maximum change in the algorithm’s output resulting from substituting a single record in its input data set. The formal definition is presented below.

Next we briefly review sliced inverse regression. Let $Y\in\mathbb{R}$ be a response variable and $X=(X_{1},\dots,X_{p})^{\mathrm{\scriptscriptstyle T}}\in\mathbb{R}^{p}$ be the associated covariates, with covariance matrix $\Sigma={\mathrm{cov}}(X)$. Assume the SDR subspace $\mathcal{S}_{Y|X}=\operatorname{span}(\beta)$, where $\beta\in\mathbb{R}^{p\times d}$ is the basis matrix and $d\ (d<p)$ is the structure dimension. SIR seeks for an unbiased estimate of $\mathcal{S}_{Y|X}$. Without loss of generality, assume $Y$ is categorical with $H$ fixed classes, denoted as $Y\in[H]$; when $Y$ is numerical, we simply follow Li (1991) to construct a discrete version $\widetilde{Y}$ of $Y$ by partitioning its range into $H$ slices without overlapping. This discretization operation will not lose information for estimating SDR subspace, i.e., $\mathcal{S}_{\widetilde{Y}|X}=\mathcal{S}_{Y|X}$, given $H$ sufficiently large (Bura and Cook, 2001), and we refer to a value $h\in[H]$ as a slice or a class. In preparation, let $p_{h}\triangleq\mathrm{pr}(Y=h)$ and $m^{0}_{h}\triangleq E\{X-E(X)|Y=h\}$, the kernel matrix is then given by

Li (1991) has shown that, if $X$ satisfies the so-called linear conditional mean (LCM) condition, i.e., $E(X\mid\beta^{\mathrm{\scriptscriptstyle T}}X)$ is a linear function of $\beta^{\mathrm{\scriptscriptstyle T}}X$, then $\Sigma^{-1}m^{0}_{h}\in\mathcal{S}_{Y\mid X}$ for $h\in[H]$. This result implies that $\Sigma^{-1}\operatorname{col}(\Lambda^{0})\subseteq\mathcal{S}_{Y|X}$. In practice, we further assume the coverage condition, that is, $\Sigma^{-1}\operatorname{col}(\Lambda^{0})=\mathcal{S}_{Y|X}$, so the first $d$ eigenvectors of $\Sigma^{-1}\operatorname{col}(\Lambda^{0})$ spans $\mathcal{S}_{Y|X}$ and the remaining $p-d$ eigenvalues equal to zero.

At the sample level, we can compute the plug-in estimator $\widehat{\Sigma}$ and $\operatorname{col}(\widehat{V}_{H}^{0})$ to estimate $\mathcal{S}_{Y|X}$, where $\widehat{V}_{H}^{0}$ is the matrix formed by the top $d$ principal eigenvectors of $\widehat{\Lambda}^{0}$. Lin et al. (2018) has shown that the space spanned by $\widehat{\Sigma}^{-1}\widehat{V}_{H}^{0}$ yields a consistent estimate of $\mathcal{S}_{Y|X}$ if and only if $\rho_{n}:=\frac{p}{n}\rightarrow 0$ as $n\rightarrow\infty$, when the slice number $H$ is a fixed integer independent of $n$ and $p$, and the structure dimension $d$ is bounded.

Consider a dataset $\mathcal{D}\triangleq\{\mathcal{D}^{(1)},\cdots,\mathcal{D}^{(K)}\}$ distributed across $K$ clients, where $\mathcal{D}^{(k)}=\{(x_{i},y_{i}):x_{i}\in\mathbb{R}^{p},y_{i}\in[H]\}_{i=1}^{n_{k}}$ is the subset stored on the $k$th client with $n_{k}$ samples, $k\in[K]$. Let $N=\sum_{k=1}^{K}n_{k}$ be the total size of $\mathcal{D}$. Generally, we do not require $n_{k}>p$ for any single client $k\in[K]$, thus on some or even all clients, data may follow a high-dimensional setting ($p>n_{k}$, or $p\gg n_{k}$). Meanwhile, given $K$ sufficiently large, it is also reasonable to assume a low-dimensional setting for the whole dataset $\mathcal{D}$, that is, $p<N$.

To handle this decentralized dataset, we need modify the classical SIR method slightly. Denote $m_{h}\triangleq E[\{X-E(X)\}{\mathrm{1}}(Y=h)]$, we immediately have $\Sigma^{-1}m_{h}\in\mathcal{S}_{Y\mid X}$ for $h\in[H]$, since $m_{h}=p_{h}m^{0}_{h}$. For convenience of expression, define the slice mean matrix $M\triangleq(m_{1},\dots,m_{H})\in\mathbb{R}^{p\times H}$ and the kernel matrix $\Lambda\triangleq MM^{{\mathrm{\scriptscriptstyle T}}}$, then

Using $\Lambda$ instead of $\Lambda^{0}$ is beneficial for our federated computation. The first advantage arises in scenarios with imbalanced classes, where a client only has a small portion of observations for certain classes. Estimating $\Lambda^{0}$ in such cases can be highly unstable, resulting in a biased global estimator. This situation is frequently encountered in streaming data, as discussed by Cai et al. (2020) in developing their online SIR estimator. In our problem, utilizing $\Lambda$ further allows us to eliminate privacy leakage in uploading $\widehat{p}_{h}$’s, thus avoiding unnecessary complexities in both algorithm design and theoretical analysis. Additionally, as detailed in Section 3, $\Lambda$ has a smaller $l_{2}$-sensitivity, reducing the noise scale required for perturbation strategies.

Algorithm 1 presents a federated framework for computing SIR on the decentralized dataset $\mathcal{D}$, catching a glimpse of our FSIR method. In this framework, each client $k\in[K]$ operates independently to estimate the slice mean matrix as $\widehat{M}^{(k)}$ and the covariance matrix as $\widehat{\Sigma}^{(k)}$ based on their local dataset $\mathcal{D}^{(k)}$, without needing to access external data. Subsequently, these two matrices are uploaded to the central server after a perturbation operation to ensure differential privacy (see details later). A global estimate of the SDR subspace is obtained on the server in Step 4, achieved through the merging of $\widetilde{M}^{(k)}$ and $\widetilde{\Sigma}^{(k)}$ for all $k\in[K]$, where $\widetilde{M}^{(k)}$ and $\widetilde{\Sigma}^{(k)}$ denoting the perturbed matrices.

At first glance, the pooling and averaging operations at Step $2$ and $3$ on the server might suggest that FSIR is merely another application of the FedAvg algorithm (McMahan et al., 2017). However, FSIR distinguishes itself by emphasizing on one-shot communication and, more importantly, privacy protection. Throughout the procedure, each client only computes and uploads local model estimates to the server, ensuring that sensitive records would remain localized with their owner. In this way, FSIR restricts the central server or any other client $j$ can only probe the dataset of client $k$ through its uploaded estimates. Meanwhile, admitting the existence of potential malicious participants who may conduct tracing attacks on the shared estimates, FSIR necessitates the differential privacy of both $\widetilde{M}^{(k)}$ and $\widetilde{\Sigma}^{(k)}$. This privacy requirement ensures that any operation performed on these estimates does not cause additional privacy breaches, thanks to the post-processing property of differential privacy.

An essential step in Algorithm 1 involves the construction of differentially private $\widetilde{\Sigma}^{(k)}$ and $\widetilde{M}^{(k)}$ for each client $k$. Privacy-preserving estimation of covariance matrices has been extensively studied, as evidenced by a large body of literature (Chaudhuri et al., 2012; Grammenos et al., 2020; Biswas et al., 2020; Wang and Xu, 2021). While it is not the primary focus of this paper, we simply adapt the approach suggested by Chaudhuri et al. (2012) within our framework, corresponding to Step $3$ on the client. The privacy of corresponding operation is guaranteed by Lemma 1. In the next section, our attention shifts towards the development of a differentially private slice mean matrix, whose singular subspace plays a crucial role in estimating $\mathcal{S}_{Y|X}$.

Notice the condition $\|X_{i}\|\leq 1$ is not commonly encountered in the realm of sufficient dimension reduction. We address this by devising a strategy to perturb $\widehat{\Sigma}/c_{R}^{2}=\frac{1}{n}(X/c_{R})(X/c_{R})^{\top}$ in accordance with the magnitudes suggested in Lemma 1. Here, $c_{R}$ denotes the maximum norm ${\mathrm{max}}_{i}\|X_{i}\|$, which is controlled by the truncation level $R$ (see Section 3.1). Notably, this scaling transformation has no impact on the estimation of $\mathcal{S}_{Y|X}$.

On the client-side, a natural approach for constructing a differential private slice mean matrix is to apply the i.i.d. Gaussian mechanism to the mean vector of each slice; see Proposition 2, which can be easily derived from Theorem A.1 in Dwork et al. (2014).

Clearly, the noise scale is determined by both of the privacy level set $(\epsilon,\delta)$ and the $l_{2}$-sensitivity $\Delta_{2}$. Recall that sliced inverse regression usually assumes that $X$ is integrable and has an elliptical distribution to satisfy the LCM condition, which does not guarantee $\Delta_{2}<\infty$ in general. Therefore, we truncate $X$ to restrict it on a finite support. Given the truncation level $R$, let

Denote $\overline{m}_{h}=(\overline{m}_{h,1},\dots,\overline{m}_{h,p})^{{\mathrm{\scriptscriptstyle T}}}$ and $\overline{M}\triangleq(\overline{m}_{1},\dots,\overline{m}_{H})\in\mathbb{R}^{p\times H}.$ We then utilize $\overline{M}$ rather than $\widehat{M}$ to meet the assumption $\Delta_{2}<\infty$. Since a sample can be only located in one slice, we easily obtain $\Delta_{2}=2R\sqrt{p}/n$ over a pair of data sets which only differ by one single entry, where $n$ is the client sample size.

The differential privacy is always guaranteed at the expense of estimation accuracy (Wasserman and Zhou, 2010; Bassily et al., 2014). In real applications, a client could have extremely limited observations, causing the scale of added noises significantly larger than of the signal. For this, we suggest to set an upper bound $\sigma_{0}^{2}$ as a tolerated scale of noise and calculate the minimal sample size

required by each client. Thus for client $k\in[K]$, only if $n_{k}\geq n_{\epsilon,\delta,p,R}$, we compute and upload $\widetilde{M}_{k}$ following Proposition 2.

The use of vanilla Gaussian mechanism in the previous section is intuitive and simple, yet has a significant limitation: it does not account for the impact of perturbation operation on the left singular space of $\overline{M}$, which is crucial for estimating $\mathcal{S}_{Y|X}$, as revealed by (3). To mitigate this, we propose a novel perturbation strategy, termed vectorized Gaussian (VGM) mechanism, to provide $(\epsilon,\delta)$-differential privacy for $\overline{M}$ without sacrificing too much information we needed. We start by giving its definition.

Obviously, the noise vector $\xi$ is characterized by the covariance matrix $\Sigma_{\xi}$. In particular, if $\Sigma_{\xi}=\{2\Delta_{2}^{2}\log(1.25/\delta)/\epsilon^{2}\}I_{p}$, the VGM mechanism coincides with the i.i.d. Gaussian mechanism. Theorem 3 provides a sufficient condition for holding the $(\epsilon,\delta)$-differential privacy of VGM mechanism.

By Taylor expansion and some algebraic computations, we can further obtain

For expression convenience, we assume the client sample size $n\geq n_{\epsilon,\delta,p,R}$ and use

as an approximate upper bound of $\|\Sigma_{\xi}^{-1}\|_{2}$. We further denote the diagonal elements of $V_{\xi}$ by $v_{1},\dots,v_{p}$, which are arranged in decreasing order.

Theorem 3 presents an inspiring result that allows for flexible design of the eigenvectors $U_{\xi}$ while achieving differential privacy by bounding $s_{p}$. Building upon this finding, we proceed by performing singular value decomposition on $\overline{M}$, yielding $\overline{M}=W_{1}SW_{2}^{{\mathrm{\scriptscriptstyle T}}}$. We take $U_{\xi}=W_{1}$ and sort the diagonal elements of $V_{\xi}$ in the same order as the diagonal elements of $S$, while enforcing the constraint $v_{p}\geq\sigma^{2}_{vgm}$. The key idea here is to construct a $p$-dimensional noise vector $\xi$ with a specific covariance structure, such that its eigenspace aligns with the left singular subspace of the slice mean matrix $\overline{M}$. Thus the perturbed matrix $\widetilde{M}=\overline{M}+E_{0}$ can preserve the relevant information about $\mathcal{S}_{Y|X}$, where $E_{0}$ is a noise matrix whose columns are generated from $\xi$.

Algorithm 2 outlines more details about our strategy. Steps $1$ to $3$ split the left singular space of $\overline{M}$ into two parts: a $d$-dimensional subspace spanned by the leading singular vectors and its orthogonal complement. Since the first part will be utilized to construct $\mathcal{S}_{Y|X}$, we preserve these eigenvectors and maintain their ordering based on corresponding eigenvalues. Meanwhile, we propose that the eigenvectors in the second part have no significant difference thus share the same eigenvalue. Following this, Steps $4$ construct a matrix $V$ by appropriately arranging diagonal elements, which leads to $\Sigma_{\xi}=W_{1}VW_{1}^{{\mathrm{\scriptscriptstyle T}}}$ and $\|\Sigma_{\xi}^{-1}\|_{2}=\sigma^{2}_{vgm}$. Finally, Steps $5$-$6$ generate noise vectors and upload perturbed slice mean matrix.

In many scenarios, client data can be high-dimensional ($p>n$) or even ultra-high-dimensional ($p=o(\exp(n^{\alpha}))$). Since SIR relies on $p/n\rightarrow 0$ as $n\rightarrow\infty$ to guarantee the consistency of $\widehat{V}_{H}$ (Lin et al., 2018), we introduce a feature screening procedure as an initial step to handle these data. Recall that SIR is developed upon the observation that $\Sigma^{-1}\{E(X|Y)-E(X)\}\in\mathcal{S}_{Y\mid X}$, implying that $\mathcal{S}_{Y|X}$ degenerates if $E(X|Y)-E(X)=0$ almost surely. This motivates us to define the active set as

and utilize the criterion

to identify $\mathcal{T}$. To take advantage of the decentralized data, we compute $\omega_{j,h}^{(k)}$ on each client $k\in[K]$ and vote for screening on the server. The entire procedure is designed to operate in a federated paradigm and we call it the Collaborative Conditional Mean Difference (CCMD) filter. We clarify its implementation details in Algorithm 3. With the inclusion of the CCMD filter, our FSIR framework now successfully handles high-dimensional data and is fully operational.