WaveCluster with Differential Privacy

Differential privacy [12] is a recent privacy model, which guarantees that an adversary cannot infer an individual’s presence in a dataset from the randomized output, despite having knowledge of all remaining individuals in the dataset.

The parameter $\epsilon$ indicates the level of privacy. Smaller $\epsilon$ provides stronger privacy. When $\epsilon$ is very small, $e^{\epsilon}$ $\approx$ 1+ $\epsilon$. Since the value of $\epsilon$ directly affects the level of privacy, we refer to it as the privacy budget. Appropriate allocation of the privacy budget for a computational process is important for reaching a favorable trade-off between privacy and utility. The most common strategy to achieve $\epsilon$-differential privacy is to add noise to the output of a function. The magnitude of introduced noise is calibrated by the privacy budget $\epsilon$ and the sensitivity of the query function. The sensitivity of a query function is defined as the maximum difference between the outputs of the query function on any pair of neighboring databases.:

There are two common approaches for achieving $\epsilon$-differential privacy: Laplace mechanism [14] and Exponential mechanism [30].

Laplace Mechanism: The output of a query function $f$ is perturbed by adding noise from the Laplace distribution with probability density function $f(x|b)=\frac{1}{2b}\exp(-\frac{|x|}{b})$, $b=\frac{\Delta f}{\epsilon}$. The following randomized mechanism $A_{l}$ satisfies $\epsilon$-differential privacy:

Exponential Mechanism: This mechanism returns an output that is close to the optimum, with respect to a quality function. A quality function $q(D,r)$ assigns a score to all possible outputs $r\in R$, where $R$ is the output range of $f$, and better outputs receive higher scores. A randomized mechanism $A_{e}$ that outputs $r\in R$ with probability

satisfies $\epsilon$-differential privacy, where $S(q)$ is the sensitivity of the quality function.

Differential privacy has two properties: sequential composition and parallel composition. Sequential composition is that given $n$ independent randomized mechanisms $A_{1},A_{2},\ldots,A_{n}$ where $A_{i}$ ($1\leq i\leq n$) satisfies $\epsilon_{i}$-differential privacy, a sequence of $A_{i}$ over the dataset $D$ satisfies $\epsilon$-differential privacy, where $\epsilon=\sum_{1}^{n}(\epsilon_{i})$. Parallel composition is that given $n$ independent randomized mechanisms $A_{1},A_{2},\ldots,A_{n}$ where $A_{i}$ ($1\leq i\leq n$) satisfies $\epsilon$-differential privacy, a sequence of $A_{i}$ over a set of disjoint data sets $D_{i}$ satisfies $\epsilon$-differential privacy.

WaveCluster is an algorithm developed by Sheikholeslami et al. [35, 36] for the purpose of clustering spatial data. It works by using a wavelet transform to detect the boundaries between clusters. A wavelet transform allows the algorithm to distinguish between areas of high contrast (high frequency components) and areas of low contrast (low frequency components). The motivation behind this distinction is that within a cluster there should be low contrast and between clusters there should be an area of high contrast (the border). WaveCluster has the following steps as shown in Figure 3:

Quantization: Quantize the feature space into grids of a specified size, creating a count matrix $M$.

Wavelet Transform: Apply a wavelet transform to the count matrix $M$, such as Haar transform [4] and Biorthogonal transform [28], and decompose $M$ to the average subband that gives the approximation of the count matrix and the detail subband that has the information about the boundaries of clusters. We refer to the average subband as the wavelet-transformed-value matrix ($W$).

Significant Grid Identification: Identify the significant grids from the average subband $W$. WaveCluster constructs a sorted list $L$ of the positive wavelet transformed values obtained from $W$ and compute the $p$th percentile of the values in $L$. The values that are below the $p$th percentile of $L$ are non-significant values. Their corresponding grids are considered as non-significant grids and the data points in the non-significant grids are considered as noise.

Cluster Identification: Identify clusters from the significant grids using connected component labeling algorithm [23] (two grids are connected if they are adjacent), map the clusters back to the original multi-dimensional space, and label the data points based on which cluster the data points reside in.

In WaveCluster, users need to specify four parameters:

num_grid ($g_{1},g_{2},\ldots,g_{n}$): the number of grids that the $n$-
dimensional space is partitioned into along each dimension. For the brevity of description, we simply use $g$ to refer to the partitions of the $n$-dimensional space ($g_{1},g_{2},\ldots,g_{n}$). This parameter controls the scaling of quantization. Inappropriate scaling can cause problems of over-quantization and under-quantization, affecting the accuracy of clustering [36].

density threshold ($p$): a percentage value $p$ that specifies $p$% of the values in $L$ are non-significant values. For ease of presentation, we use $k=(1-p)|L|$ to represent the top $k$ values in $L$ and their corresponding grids are considered as significant grids.

level: a wavelet decomposition level, which indicates how many times a wavelet transform is applied. The larger the level is, the more approximate the result is. In our techniques, we set level to 1 since a smaller level value provides more accurate results [36].

wavelet: a wavelet transform to be applied. Haar transform [4] is one of the simplest wavelet transforms and widely used, which is computed by iterating difference and averaging between odd and even samples of a signal (or a sequence of data points). Other commonly used wavelet transforms include Biorthogonal transform [28], Daubechies transform [11], and so on.

Motivating Scenario. Consider a scenario with two participants: the data owner (e.g. hospitals) and the querier (e.g. data miner). The data owner holds raw data and has the legal obligation to protect individuals’ privacy while the querier is eager to obtain cluster analysis results for further exploration. The goal of our work is to enable the data owner to release cluster analysis results using WaveCluster while not compromising the privacy of any individual who contributes to the raw data. The data owner has a good knowledge of the raw data and it is not difficult for her to pick the appropriate parameters (e.g. num_grid, density threshold, and wavelet) for non-private WaveCluster. For example, the data owner may draw from her past experience on similar data to determine the appropriate parameters for the current dataset. The parameters picked for the non-private setting are directly used for the private setting, and thus the data owner does not need to infer another set of parameters for the private setting.

Problem Statement. Given a raw data set $D$, appropriate WaveCluster parameters for $D$ and a privacy budget $\epsilon$, our goal is to investigate an effective approach $A$ such that $A$ (1) satisfies $\epsilon$-differential privacy, and (2) achieves high utility of the private WaveCluster results with regard to the utility metrics $U$.

A straightforward technique to achieve differential privacy on WaveCluster is as follows: (1) adapt an existing $\epsilon$-differential privacy preserving data publishing method to get the noisy description of the data distribution in some fashion, such as a set of contingency tables or a spatial decomposition tree [40, 39, 10, 33]; (2) generate a synthetic dataset according to the noisy description; (3) apply WaveCluster on the synthetic dataset. We refer to this technique as Baseline, and its pseudocode is shown in Algorithm 1.

Baseline first leverages a $\epsilon$-differential privacy preserving data publishing method to obtain a noisy dataset $D^{\prime}$ (Line 2) and partitions $D^{\prime}$ based on the number of grids $g$ to obtain the noisy count matrix $M^{\prime}$ (Line 3). Baseline then applies a wavelet transform on $M^{\prime}$ to obtain $W^{\prime}$ (Line 4). $W^{\prime}$ is then turned into a list $L^{\prime}$ that keeps only positive values and the values in $L^{\prime}$ is sorted in ascending order (Line 5). With $L^{\prime}$, $k^{\prime}$ is computed based on the specified density threshold $p$ and the size of $L^{\prime}$ (Line 6). Finally, Baseline obtains $d^{\prime}$ as the top $k^{\prime}$th value in $L^{\prime}$ (Line 7), where any value in $L^{\prime}$ greater than $d^{\prime}$ is considered as a significant value, and applies the connected component labeling algorithm to identify clusters of significant grids (Line 8).

Discussion. Baseline achieves differential privacy on WaveCluster through the achievement of differential privacy on data publishing. However, it does not produce accurate WaveCluster results in most cases. The adapted $\epsilon$-differential privacy preserving data publishing method is designed for answering range queries. The noisy descriptions of the data distribution generated by the method may contain negative counts for certain partitions since the noise distribution is Laplacian with zero mean. These negative counts do not affect the range query accuracy too much since zero-mean noise distribution smooths the effect of noise. For example, a partition $p_{1}$ has the true count of 2 and the noisy count of -2, whose noise is canceled by another partition $p_{2}$ having the true count of 10 and the noisy count of 14 when both $p_{1}$ and $p_{2}$ are included in a range query. In particular, when the range query spreads large range of a dataset, a single partition with a noisy negative count does not affect its accuracy too much. However, when the method is used for generating a synthetic dataset, the noisy negative counts are reset as zero counts, causing the data distribution to change radically on the whole and further leading to the severe deviation in differentially private WaveCluster results.

To address the challenge faced by Baseline, we propose techniques that enforce differential privacy on the key steps of WaveCluster. Our first approach, called Private Quantization (PrivQT), introduces independent Laplacian noise in the quantization step to achieve differential privacy. In the quantization step, the data is divided into grids and the count matrix $M$ is computed. To ensure differential privacy in this step, we rely on the Laplace mechanism that introduces independent Laplacian noise to $M$. Clearly, if we change one individual in the input data, such as adding, removing or modifying an individual, there is at most one change in one entry of $M$. According to the parallel composition property of differential privacy, the noise amount introduced to each grid is $Lap(\frac{1}{\epsilon})$, given a privacy budget $\epsilon$. Since the following steps of WaveCluster are carried on using the differentially private count matrix $M^{\prime}$, the clusters derived from these steps are also differentially private. Algorithm 2 shows the pseudocode of PrivQT. Except from the first step that introduces independent Laplacian noise to $M$ (Line 2), the other steps (Lines 3-7) are the same as Baseline.

Selecting the appropriate grid size (reflected by the parameter num_grid $g$) in the quantization step strongly affects the accuracy of WaveCluster results [36], and also the differentially private
WaveCluster results. A small grid size (small $g$) causes more data points to fall into each grid and thus the count of data points for each grid becomes larger, which makes the count matrix $M$ resistant to Laplacian noise. However, the small grid size is not helpful for WaveCluster to detect clusters with accurate shapes and renders the results less useful. On the other hand, although posing a larger grid size on the data captures the density distribution of the data more clearly, it makes each grid’s count too small and thus become sensitive to Laplacian noise, which dramatically affects the identification of significant grids and further the shapes of clusters. Our empirical results show that only when an appropriate grid size is given, differentially private WaveCluster results maintains high utility.

Discussion. Although PrivQT achieves differential privacy on the WaveCluster results, the noisy count matrix $M^{\prime}$ and its resulting noisy $L^{\prime}$ are significantly distorted and consequently the clustering results. The reason is as follows. Given a specified percentage value $p$, PrivQT computes $k^{\prime}$ from the positive values in $W^{\prime}$, where $W^{\prime}$ is derived from $M^{\prime}$, which is perturbed by Laplacian noise. Laplacian distribution is symmetric and has zero-mean. According to its randomness, approximately half of the zero-count grids become noisy positive-count grids due to positive noise while the remaining ones are turned into noisy negative-count grids due to negative noise. These noisy positive-count grids may cause their corresponding wavelet transformed values in $W^{\prime}$ to become positive (depending on the targeted wavelet transform), which will inappropriately participate in the computation of $k^{\prime}$ and further distorts $k^{\prime}$. Due to the dominating errors introduced by approximately half of zero-count grids becoming noisy positive-count grids, our empirical results show that the utility of private WaveCluster results by PrivQT improves marginally even for a large privacy budget.

The limitation of PrivQT lies in the severe distortion of $k^{\prime}$ by Laplacian noise introduced into count matrix $M^{\prime}$. To mitigate the distortion, we propose a technique, PrivTHR, which prunes a portion of noisy positive values in $W^{\prime}$ to refine the computation of $k^{\prime}$. Algorithm 3 shows the pseudocode of PrivTHR.

PrivTHR first introduces random noise to the count matrix $M$, similar to PrivQT, and obtains a noisy count matrix $M^{\prime}$ (Line 2). PrivTHR then applies a wavelet transform on $M^{\prime}$ to obtain $W^{\prime}$ (Line 3). $W^{\prime}$ is then turned into a list $L^{\prime}$ that keeps only positive values and the values in $L^{\prime}$ is sorted in ascending order (Line 4). Thus, only the positive values in $W^{\prime}$ will be used for computing $k^{\prime}$ based on the specified density threshold $p$. To reduce the distortion of $k^{\prime}$, starting from the smallest noisy positive values in $L^{\prime}$, PrivTHR discards the first $\frac{|Z|^{\prime}}{2}$ values (Line 6), where $Z$ represents the non-positive (negative or zero) values in the $W$ and $|Z|^{\prime}$ is a noisy estimate of $|Z|$ (Line 5). The reason why PrivTHR removes $\frac{|Z|^{\prime}}{2}$ values from $L^{\prime}$ is based on the utility analysis (in Section 5.2) that approximately $\frac{|Z|}{2}$ non-positive values in $W$ are turned into positive values due to the randomness of Laplacian noise. Since $|Z|$ partially describes the data distribution and releasing $|Z|$ without protection may leak private information, PrivTHR also introduces Laplacian noise to $|Z|$, ensuring the whole process correctly enforces differentially privacy (Lines 11-17). The noise introduced to $|Z|$ depends on the wavelet transform used to compute $W$. For example, if we use Haar transform for $n$-dimensional data, a value in $W$ is computed by applying average for two neighboring elements along each dimension. Since any single change in the input only causes one entry of the count matrix $M$ to change by 1, the change of $M$ causes at maximum one value in $W$ to change, and thus causes $|Z|$ to change by 1 at maximum, i.e., the sensitivity of $|Z|$ is 1¹¹1For other wavelet transforms that use circular convolutions, such as Biorthogonal transform, the sensitivity of $n$ depends on the count of positive values and the count of negative values in the matrix computed by the coefficient vector [28].. Finally, PrivTHR obtains $d^{\prime}$ as the top $k^{\prime}$th value in $L^{\prime\prime}$ (Line 8), where any value in $L^{\prime\prime}$ greater than $d^{\prime}$ is considered as a significant value, and applies the connected component labeling algorithm to identify clusters of significant grids (Line 9).

Budget Allocation. PrivTHR first introduces Laplacian noise in the quantization step using a privacy budget $\epsilon_{1}=\alpha\epsilon$, where $0<\alpha<1$. In the significant grid identification step, PrivTHR further introduces Laplacian noise to $|Z|$ using the remaining privacy budget $(1-\alpha)\epsilon$. Based on utility analysis in Section 5.2.2, $\epsilon_{1}$ requires a smaller amount of budget than $\epsilon_{2}$. Our empirical results in Section 7 further show in detail the impact of $\alpha$ on clustering accuracy.

Besides pruning noisy positive values in $W^{\prime}$, we propose an alternative technique that employs Exponential mechanism for deriving $k^{\prime}$ from the sorted list of $L$. Algorithm 4 shows the pseudocode of PrivTHR_EM.

PrivTHR_EM first introduces Laplacian noise to the count matrix $M$, which is similar to PrivQT and PrivTHR. After that, we obtain a noisy count matrix $M^{\prime}$ (Line 2) and the corresponding $W^{\prime}$ (Line 3). Different from the previous two techniques that compute $k^{\prime}$ from $W^{\prime}$, PrivTHR_EM derives $k^{\prime}$ from $W$ using Exponential mechanism (Lines 7-15). In this case, although the sorted list derived from $W^{\prime}$ is severely distorted in PrivTHR_EM, the derivation of $k^{\prime}$ from $W$ is not affected by the distorted $W^{\prime}$ at all. Given reasonable privacy budget, $k^{\prime}$ derived from $W$ using Exponential mechanism is reasonably accurate, compared to the case when $k^{\prime}$ is derived from $W^{\prime}$.

The quality function fed into the Exponential mechanism is [10]:

where $L$ represents the sorted positive values in $W$ with $Min$ and $Max$ values (Line 10), and $X$ represents the possible output space, i.e., all the possible values in the range of $(0,Max]$. Given a $W$ with $m$ positive values and their relationships are $x_{1}\geq x_{2}\geq\ldots\geq x_{m}$, these $m$ values divide the range $(0,Max]$ into $m$ partitions: $(0,x_{m}],(x_{m-1},x_{m-2}],$ $\ldots,(x_{2},x_{1}]$, and the ranks for these partitions are $m$, $m-1$, $\ldots$, 2, 1. For any $x\in(x_{i-1},x_{i}]$, its rank is $rank(x_{i})$. For example, if $x\in(x_{2},x_{1}]$, $rank(x)=rank(x_{1})=1$. Similar to PrivTHR, when using Haar transform, any single change in the input causes only one value in $W$ to change. Thus, at maximum one value will be added into or removed from $L$, causing the outcome of $q(L,X)$ to be changed by 1, i.e., the sensitivity of $q(L,X)$ is 1²²2Similar to PrivTHR, for other wavelet transforms that use circular convolutions, the sensitivity of $q(L,X)$ depends on the count of positive values and the count of negative values in the matrix computed by the coefficient vector [28]..

Plugging in the above quality function into Exponential mechanism, we obtain the following algorithm: for any value $x\in(0,Max]$, the Exponential mechanism (EM) returns $x$ with probability
$Pr[EM(L)=x]\propto exp(-\frac{\epsilon|rank(x)-k|}{2})$ (Line 12). Since all the values in a partition have the same probability to be chosen, a random value from the partition $Pt_{i}=(x_{i-1},x_{i}]$ will be chosen with the probability proportional to $|Pt_{i}|*exp(-\frac{\epsilon}{2}|i-k|)$. In other words, once $k^{\prime}$ is chosen, PrivTHR_EM further computes a uniform random value $d^{\prime}$ from $Pt_{i}$ (Line 13), and any value in $L^{\prime}$ greater than $d^{\prime}$ is considered as a significant value.

Budget Allocation. Similar to PrivTHR, the privacy budget is split between two steps: introduction of Laplacian noise in quantization and obtaining $k^{\prime}$ using Exponential mechanism. Previous empirical experiments [10] on splitting budgets between obtaining noisy median and noisy counts suggest that, 30% vs. 70% budget allocation strategy performs best. Specifically, 70% of budget is allocated for obtaining noisy count matrix $M^{\prime}$ (Line 2) and the remaining budget is allocated for computing $k^{\prime}$ (Line 4).

In this part we establish the privacy guarantee of PrivQT, PrivTHR and PrivTHR_EM.

In this section, we present utility guarantees of our algorithms (PrivQT, PrivTHR and PrivTHR_EM) with theoretical analysis. In WaveCluster, the step of significant grid identification determines the clustering results. In the private results of WaveCluster, PrivQT, PrivTHR and PrivTHR_EM return a list of noisy significant grids. To quantify the utility of PrivQT, PrivTHR and PrivTHR_EM, we consider finding significant grids whose wavelet transformed values surpass a threshold to be similar to finding the top-$k$ frequent itemsets whose frequencies surpass a threshold. In significant grid identification, $L$ is the list of positive wavelet transformed values from $W$ sorted in ascending order, $Z$ represents the set of zero values from $W$, and $k$ indicates the threshold position in $L$ and all the top-$k$ values in $L$ correspond to significant grids, where $k=(1-p)|L|$. One parameter to specify $k$ is the density threshold $p$, which remains the same either with or without noise introduction. However, $|L|$, another parameter to determine $k$, will be changed to $|L^{\prime}|$ under differential privacy, where $L^{\prime}$ is the list of positive wavelet transformed values from $W^{\prime}$ sorted in ascending order. $L$ is different from $L^{\prime}$ since noise introduction might result in a portion of zero values in $Z$ becoming positive and a small portion of positive values in $L$ becoming non-positive.