Differentially Private Sliced Wasserstein Distance

We start by stating the main problem we are interested in: to show the privacy properties of the random mechanism

where $\mathbf{X}\in\mathbb{R}^{n\times d}$ is a matrix (a dataset), $\mathbf{U}\in\mathbb{R}^{d\times k}$ a random matrix made of $k$ uniformly distributed unit-norm vectors of $\mathbb{R}^{d}$ and $\mathbf{V}\in\mathbb{R}^{n\times k}$ a matrix of $k$ zero-mean Gaussian vectors (also called the Gaussian Mechanism).

We show that ${\cal M}$ is differentially private and that it is the core component of the Sliced Wassertein Distance (SWD) computed thanks to random projection directions (the unit-norm matrix $\mathbf{U}$) and, in turn, SWD inherits¹¹1This is a slight abuse of vocabulary as the Sliced Wasserstein Distance takes two inputs and not only one. the differential private property of ${\cal M}$. In the way, we show that the population version of the resulting differentially private SWD is a distance, that we dub the Gaussian Smoothed SWD.

DP is a theoretical framework to analyze the privacy guarantees of algorithms. It rests on the following definitions.

The following notion of privacy, proposed by Mironov [22], which is based on Rényi $\alpha$-divergences and its connections to $(\varepsilon,\delta)$-differential privacy will ease the exposition of our results (see also [2, 3, 32]):

As we shall see, the role of $f$ will be played by the Random Direction Projections operation or the Sliced Wasserstein Distance (SWD), a randomized algorithm itself, and the mechanism to be studied is the composition of two random algorithms, SWD and the Gaussian mechanism. Proving the (R)DP nature of this mechanism will rely on a high probability bound on the sensitivy of the Random Direction Projections/SWD combined with the result of Remark 2.

Let $\Omega\in\mathbb{R}^{d}$ be a probability space and $\mathcal{P}(\Omega)$ the set of all probability measures over $\Omega$. The Wasserstein distance between two measures $\mu$, $\nu\in\mathcal{P}(\Omega)$ is based on the so-called Kantorovitch relaxation of the optimal transport problem, which consists in finding a joint probability distribution $\gamma^{\star}\in\mathcal{P}(\Omega\times\Omega)$ such that

where $c(\cdot,\cdot)$ is a metric on $\Omega$, known as the ground cost (which in our case will be the Euclidean distance), $\Pi(\mu,\nu)\doteq\{\gamma\in\mathcal{P}(\Omega\times\Omega)|\pi_{1\#}\gamma=\mu,\pi_{2\#}\gamma=\nu\}$ and $\pi_{1},\pi_{2}$ are the marginal projectors of $\gamma$ on each of its coordinates. The minimizer of this problem is the optimal transport plan and for $q\geq 1$, the $q$-Wasserstein distance is

A case of prominent interest for our work is that of one-dimensional measures, for which it was shown by Rabin et al. [26], Bonneel et al. [5] that the Wasserstein distance admits a closed-form solution which is

where $F^{-1}_{\cdot}$ is the inverse cumulative distribution function of the related distribution. This combines well with the idea of projecting high-dimensional probability distributions onto random 1-dimensional spaces and then computing the Wasserstein distance, an operation which can be theoretically formalized through the use of the Radon transform [5], leading to the so-called Sliced Wasserstein Distance

where $\mathcal{R}_{\mathbf{u}}$ is the Radon transform of a probability distribution so that

with $\mathbf{u}\in\mathbb{S}^{d-1}\doteq\{\mathbf{u}\in\mathbb{R}^{d}:\|\mathbf{u}\|_{2}=1\}$ be the $d$-dimensional hypersphere and $u_{d}$ the uniform distribution on $\mathbb{S}^{d-1}$.

In practice, we only have access to $\mu$ and $\nu$ through samples, and the proxy distributions of $\mu$ and $\nu$ to handle are $\hat{\mu}\doteq\frac{1}{n}\sum_{i=1}^{n}\delta_{\mathbf{x}_{i}}$ and $\hat{\nu}\doteq\frac{1}{m}\sum_{i=1}^{m}\delta_{\mathbf{x}^{\prime}_{i}}.$ By plugging those distributions into Equation 3, it is easy to show that the Radon transform depends only the projection of $\mathbf{x}$ on $\mathbf{u}$. Hence, computing the sliced Wasserstein distance amounts to computing the average of 1D Wasserstein distances over a set of random directions $\{\mathbf{u}_{j}\}_{j=1}^{k}$, with each 1D probability distribution obtained by projecting a sample (of $\hat{\mu}$ or $\hat{\nu}$) on $\mathbf{u}_{i}$ by $\mathbf{x}^{\top}\mathbf{u}_{i}$. This gives the following empirical approximation of SWD

given $\mathbf{U}$ a matrix of $\mathbb{R}^{d\times k}$ of unit-norm column $\mathbf{u}_{j}$.

In order to uncover its $(\epsilon,\delta)$-DP , we analyze the sensitivity of the random direction projection in SWD. Let us consider the matrix $\mathbf{X}\in\mathbb{R}^{n\times d}$ representing a dataset composed of $n$ examples in dimension $d$ organized in row (each sample being randomly drawn from the distribution $\mu$). One mechanism of interest is

where $\mathbf{v}$ is a vector whose entries are drawn from a zero-mean Gaussian distribution. Let $\mathbf{X}$ and $\mathbf{X}^{\prime}$ be two matrices in $\mathbb{R}^{n\times d}$ that differ only on one row, say $i$ and such that $\|\mathbf{X}_{i,:}-\mathbf{X}_{i,:}^{\prime}\|_{2}\leq 1$, where $\mathbf{X}_{i,:}\in\mathbb{R}^{d}$ and $\mathbf{X}_{i,:}^{\prime}\in\mathbb{R}^{d}$ are the $i$-th row of $\mathbf{X}$ and $\mathbf{X}^{\prime}$, respectively. For ease of notation, we will from now on use

Instead of considering a randomized mechanism that projects only according to a single random direction, we are interested in the whole set of projected (private) data according to the random directions sampled through the Monte-Carlo approximation of the Sliced Wasserstein distance computation (4). Our key interest is therefore in the mechanism

and in the sensitivity of $\mathbf{X}\mathbf{U}$. Because of its randomness, we are interested in a probabilistic tail-bound of $\|\mathbf{X}\mathbf{U}-\mathbf{X}^{\prime}\mathbf{U}\|_{F}$, where the matrix $\mathbf{U}$ has columns independently drawn from $\mathbb{S}^{d-1}$.

The above bound on the squared sensitivity has been obtained by first showing that the random variable $\left\|\mathbf{X}\mathbf{U}-\mathbf{X}^{\prime}\mathbf{U}\right\|_{F}^{2}$ is the sum of $k$ iid Beta-distributed random variables and then by using a Bernstein inequality. This bound, referred to as the Bernstein bound, is very conservative as soon as $\delta$ is very small. By calling the Central Limit Theorem (CLT), assuming that $k$ is large enough ($k>30$), we get under the same hypotheses (proof is in the appendix) that

where $z_{1-\delta}=\Phi^{-1}(1-\delta)$ and $\Phi$ is the cumulative distribution function of a zero-mean unit variance Gaussian distribution. This bound is far tighter but is not rigourous due to the CLT approximation. Figure 1 presents an example of the probability distribution histogram of $\|(\mathbf{X}-\mathbf{X}^{\prime})^{\top}\mathbf{U}\|_{F}^{2}=\|(\mathbf{X}_{i,:}-\mathbf{X}_{i,:}^{\prime})^{\top}\mathbf{U}\|_{2}^{2}$ for two fixed arbitrary $\mathbf{X}_{i,:}$, $\mathbf{X}_{i,:}^{\prime}$ and for $10000$ random draws of $\mathbf{U}$. It shows that the CLT bound is numerically far smaller than the Bernstein bound of Lemma 2. Then, using the $\mathbf{w}(k,\delta)$-based bounds jointly with the Gaussian mechanism property gives us the following proposition.

The above DP guarantees apply to the full dataset. Hence, when learning through mini-batches, we benefit from the so-called privacy amplification by the “subsampling” principle, which ensures that a differentially private mechanism run on a random subsample of a population leads to higher privacy guarantees than when run on the full population [4]. On the contrary, gradient clipping/sanitization acts individually one each gradient and thus do not fully benefit from the subsampling amplification, as its DP property may still depend on the batch size [9].

This Gaussian mechanism on the random direction projections $\mathcal{M}(\mathbf{X})$ can be related to the definition of the empirical SWD as each $\mathbf{x}_{i}^{\top}\mathbf{u}_{j}$ corresponds to one entry of $\mathbf{X}\mathbf{U}$. Hence, by adding a Gaussian noise to each projection, we naturally derive our empirical DP Sliced Wasserstein distance, which inherits the differential property of $\mathcal{M}(\mathbf{X})$, owing to the post-processing proposition [12].

We have analyzed the sensitivity of the random direction projection central to SWD and we have proposed a Gaussian mechanism to obtain a differentially private SWD (DP-SWD) which steps are depicted in Algorithm 1. In our use-cases, DP-SWD is used in a context of learning to match two distributions (one of them requiring to be privately protected). Hence, the utility guarantees of our DP-SWD is more related to the ability of the mechanism to distinguish two different distributions rather than on the equivalence between SWD and DP-SWD. Our goal in this section is to investigate the impact of adding Gaussian noise to the source $\mu$ and target $\nu$ distributions in terms of distance property in the population case.

Since $\mathcal{R}_{\mathbf{u}}$, as defined in Equation (3), is a push-forward operator of probability distributions, the Gaussian mechanism process implies that the Wasserstein distance involved in SWD compares two 1D probability distributions which are respectively the convolution of a Gaussian distribution and $\mathcal{R}_{\mathbf{u}}\mu$ and $\mathcal{R}_{\mathbf{u}}\nu$. Hence, we can consider DP-SWD uses as a building block the 1D smoothed Wasserstein distance between $\mathcal{R}_{\mathbf{u}}\mu$ and $\mathcal{R}_{\mathbf{u}}\nu$ with the smoothing being ensured by $\mathcal{N}_{\sigma}$ and its formal definition being, for $q\geq 1$,

While some works have analyzed the theoretical properties of the Smoothed Wasserstein distance [15, 14], as far as we know, no theoretical result is available for the smoothed Sliced Wasserstein distance, and we provide in the sequel some insights that help its understanding. The following property shows that DP-SWD preserves the identity of indiscernibles.

These properties are strongly relevant in the context of our machine learning applications. Indeed, while they do not tell us how the value of DP-SWD compares with SWD, at fixed $\sigma>0$ or when $\sigma\rightarrow 0$, they show that they can properly act as (for any $\sigma>0$) loss functions to minimize if we aim to match distributions (at least in the population case). Naturally, there are still several theoretical properties of $\text{DP}_{\sigma}\text{SWD}_{q}^{q}$ that are worth investigating but that are beyond the scope of this work.

Most recent approaches [13] that proposed DP generative models considered it from a GAN perspective and applied DP-SGD [1] for training the model. The main idea for introducing privacy is to appropriately clip the gradient and to add calibrated noise into the model’s parameter gradient during training [29, 9, 34]. This added noise make those models even harder to train. Furthermore, since the DP mechanism applies to each single gradient, those approaches do not fully benefit from the amplification induced by subsampling (mini-batching) mechanism [4]. The work of Chen et al. [9] uses gradient sanitization and achieves privacy amplification by training multiple discriminators, as in [18], and sampling on them for adversarial training. While their approach is competitive in term of quality of generated data, it is hardly tractable for large scale dataset, due to the multiple (up to 1000 in their experiments) discriminator trainings.

Instead of considering adversarial training, some DP generative model works have investigated the use of distance on distributions. Harder et al. [17] proposed random feature based maximum-mean embedding distance for computing distance between empirical distributions. Cao et al. [7] considered the Sinkhorn divergence for computing distance between true and generated data and used gradient clipping and noise addition for privacy preservation. Their approach is then very similar to DP-SGD in the privacy mechanism. Instead, we perturb the Sliced Wasserstein distance by smoothing the distributions to compare. This yields a privacy mechanism that benefits subsampling amplification, as its sensitivity does not depend on the number of samples, and that preserves its utility as the smoothed Sliced Wasserstein distance is still a distance.

Sliced Wasserstein Distance leverages on Radon transform for mapping high-dimensional distributions into 1D distributions. This is related to projection on random directions and the sensitivity analysis of those projections on unit-norm random vector is key. The first use of random projection for differential privacy has been introduced by Kenthapadi et al. [19]. Their approach was linked to the distance preserving property of random projections induced by the Johnson-Lindenstrauss Lemma. As a natural extension, LeTien et al. [21] and Gondara & Wang [16] have applied this idea in the context of optimal transport and classification. The fact that we project on unit-norm random vector, instead of any random vector as in Kenthapadi et al. [19], requires a novel sensitivity analysis and we show that this sensitivity scales gracefully with ratio of the number of projections and dimension of the distributions.

The goal of this experiment is to illustrate the behaviour of the DP-SWD compared with the SWD in controlled situations. We consider the source and target distributions as isotropic Normal distributions of unit variance with added privacy-inducing Gaussian noise of different variances. Both distributions are Gaussian of dimension $5$ and the means of the source and target are respectively $\mathbf{m}_{\mu}=0$ and $\mathbf{m}_{\nu}=c1$ with $c\in[0,1]$. Figure 2 presents the evolution of the distances averaged over $5$ random draws of the Gaussian and noise. When source and target distributions are different, this experiment shows that DP-SWD follows the same increasing trend as SWD. This suggests that the order relation between distributions as evaluated using SWD is preserved by DP-SWD, and that the distance DP-SWD is minimized when $\mu=\nu$, which are important features when using DP-SWD as a loss.

We conduct experiments for evaluating our DP-SWD distance in the context of classical unsupervised domain adaptation (UDA) problems such as handwritten digit recognitions (MNIST/USPS), synthetic to real object data (VisDA 2017) and Office 31 datasets. Our goal is to analyze how DP-SWD performs compared with its public counterpart SWD [20], with one DP deep domain adaptation algorithm DP-DANN that is based on gradient clipping [31] and with the classical non-private DANN algorithm. Note that we need not compare with [21] as their algorithm does not learn representation and does not handle large-scale problems, as the OT transport matrix coupling need be computed on the full dataset. For all methods and for each dataset, we used the same neural network architecture for representation mapping and for classification. Approaches differ only on how distance between distributions have been computed. Details of problem configurations as well as model architecture and training procedure can be found in the appendix. Sensitivity has been computed using the Bernstein bound of Lemma 2.

Table 1 presents the accuracy on the target domain for all methods averaged over $10$ iterations. We remark that our private model outperforms the DP-DANN approach on all problems except on two difficult ones. Interestingly, our method does not incur a loss of performance despite the private mechanism. This finding is confirmed in Figure 3 where we plot the performance of the model with respect to the noise level $\sigma$ (and thus the privacy parameter $\varepsilon$). Our model is able to keep accuracy almost constant for $\varepsilon\in[3,10]$.

In the context of generative models, our first task is to generate synthetic samples for MNIST and Fashion MNIST dataset that will be afterwards used for learning a classifier. We compare with different gradient-sanitization strategies like DP-CGAN [29], and GS-WGAN [9] and a model MERF [17] that uses MMD as distribution distance. We report results for our DP-SWD using two ways for computing the sensitivity, by using the CLT bound and the Bernstein bound, respectively noted as DP-SWD-c and DP-SWD-b. All models are compared with the same fixed budget of privacy $(\varepsilon,\delta)=(10,10^{-5})$. Our implementation is based on the one of MERF [17], in which we just plugged our DP-SWD loss in place of the MMD loss. The architecture of ours and MERF’s generative model is composed of few layers of convolutional neural networks and upsampling layers with approximately 180K parameters while the one of GS-WGAN is based on a ResNet with about 4M parameters. MERF’s and our models have been trained over $100$ epochs with an Adam optimizer and batch size of $100$. For our DP-SWD we have used $1000$ random projections and the output dimension is the classical $28\times 28=784$.

Table 2 reports some quantitative results on the task. We note that MERF and our DP-SWD perform on par on these problems (with a slight advantage for MERF on FashionMNIST and for DP-SWD on MNIST). Note that our results on MERF are better than those reported in [9]. We also remark that GS-WGAN performs the best at the expense of a model with $20$-fold more parameters and a very expensive training time (few hours just for training the $1000$ discriminators, while our model and MERF’s take less than 10min). Figure 4 and 5 present some examples of generated samples for MNIST and FashionMNIST. We can note that the samples generated by DP-SWD present some diversity and are visually more relevant than those of MERF, although they do not lead to better performance in the classification task. Our samples are a bit blurry compared to the ones generated by the non-private SWD: this is an expected effect of smoothing.

We also evaluate our DP-SWD distance for training generative models on large RGB datasets such as the $64\times 64\times 3$ CelebA dataset. To the best of our knowledge, no DP generative approaches have been experimented on such a dataset. For instance, the GS-WGAN of [9] has been evaluated only on grayscale MNIST-like problems. For training the model, we followed the same approach (architecture and optimizer) as the one described in Nguyen et al. [23]. In that work, in order to reduce the dimension of the problems, distributions are compared in a latent space of dimension $d=8192$. We have used $k=2000$ projections which leads to a ratio $\frac{k}{d}<0.25$. Noise variance $\sigma$ and privacy loss over $100$ iterations have been evaluated using the PyTorch package of [32] and have been calibrated for $\epsilon=10$ and $\delta=10^{-6}$, since the number of training samples is of the order of $170$K. Details are in the appendix. Figure 6 presents some examples of samples generated from our DP-SWD and SWD. We note that in this high-dimensional context, the sensitivity bound plays a key role, as we get a FID score of 97 vs 158 respectively using CLT bound and Bernstein bound, the former being smaller than the latter.

A simulation of the random $Y$ and resulting histogram is depicted in Figure 7.

From the above lemma, we have a probabilistic bound on the sensitivity of the random direction projection and SWD . The lower this bound is the better it is, as less noise needed for achieving a certain $(\varepsilon,\delta)$-DP. Interestingly, the first and last terms in this bound have an inverse dependency on the dimension. Hence, if the dimension of space in which the DP-SWD has to be chosen, for instance, when considering latent representation, a practical compromise has to be performed between a smaller bound and a better estimation. Also remark that if $k<d$, the bound is mostly dominated by the term $\log(1/\delta)$. Compared to other random-projection bounds [30] which have a linear dependency in $k$. For our bound, dimension also help in mitigating this dependency.