Achilles’ Heels: Vulnerable Record Identification in Synthetic Data Publishing

We consider an entity (e.g., a company) that wants to give a third-party access to a private dataset for analysis. A dataset is a multiset of records $D=\{x_{1},\ldots,x_{n}\}$, where each record relates to a unique individual. In the case of narrow datasets, multiple individuals can however have the same record. We assume the dataset to be sensitive, containing information such as healthcare records or financial transactions. We assume each record to consist of $F$ attributes $x_{i}=(x_{i,1},\ldots,x_{i,F})\in\mathcal{V}_{1}\times\ldots\times\mathcal{V}_{F}$, where $\mathcal{V}_{j}$ denotes the space of values that can be taken by the $j$-th attribute. We denote by $\mathcal{V}=\mathcal{V}_{1}\times\ldots\times\mathcal{V}_{F}$ the universe of possible records, and assume the dataset to be sampled i.i.d. from a distribution $\mathcal{D}$ over $\mathcal{V}$.

An increasingly popular approach to release data while mitigating the privacy risk is to instead generate and publish a synthetic dataset [13]. Synthetic data is generated from a statistical model trained to have similar statistical properties to the real data.

We refer to the statistical model as the synthetic data generator. Formally, this is a randomized function $\Phi:\mathcal{V}^{n}\rightarrow\mathcal{V}^{m}$ mapping a private dataset $D$ to a synthetic dataset $D^{s}=\Phi(D)$ of $m$ records. The synthetic data generator can take the form of a probabilistic model such as a Bayesian Network [42] or a Generative Adversarial Network (GAN) such as CTGAN [39].

Differential Privacy (DP) [12] is a formal privacy guarantee. Originally developed to protect the release of aggregate statistics, it has since been extended to machine learning models and recently to synthetic data generators.

DP relies on a notion of neighboring datasets. Two datasets $D,D^{\prime}\sim\mathcal{D}$ are neighboring, if they differ by only one record. Intuitively, a DP randomized function $\Phi$ ensures that one attacker cannot infer the difference between $\Phi(D)$ and $\Phi(D^{\prime})$ bounded by a certain probability, effectively providing formal guarantees against privacy risks and, in particular, membership inference attacks.

Formally, the definition of differential privacy is the following:

The parameter $\epsilon$ is referred to as the privacy budget. It quantifies the privacy leakage of a data release, with smaller $\epsilon$ providing more protection. Achieving DP with small to reasonable values of $\epsilon$ can however require significant noise to be added to a machine learning model, decreasing its utility. Efficiently achieving DP guarantees for ML models, including synthetic data generation models, is an active area of research. In this paper, we used the DP generator PrivBayes from the work of Zhang et al. [42]. We refer the reader to Sec. 1 for details.

Threat Model. Membership Inference Attacks (MIAs) aim to infer whether a target record $x_{T}$ is part of the original dataset $D$ used to generate a synthetic dataset $D^{s}$, i.e., whether $x_{T}\in D$. The traditional attacker in MIAs is assumed to have access to the published synthetic dataset, $D^{s}$, as well as to an auxiliary dataset, $D_{aux}$, with the same underlying distribution as $D$: $D_{aux}\sim\mathcal{D}$. Additionally, the attacker is assumed to have access to the target record $x_{T}$ and to know the generative model type and its hyperparameters, but not to have access to the model [35, 20].

Privacy Game. MIAs are instantiated as a privacy game. The game consists of a challenger and an attacker and is instantiated on a target record $x_{T}$. The challenger samples datasets $D$ of $n-1$ records from $\mathcal{D}$, such that all records are different from $x_{T}$. With equal probability, the challenger adds $x_{T}$ to $D$ or a random, different record $x_{R}\sim\mathcal{D}$, with $x_{R}\neq x_{T}$, to $D$. The challenger then trains a generator $\Phi_{T}$ on $D$ and uses it to generate a synthetic dataset $D^{s}$. The challenger shares $D^{s}$ with an attacker whose goal is to infer whether or not $x_{T}\in D$. If the attacker correctly infers, they win the game. The privacy risk is estimated by playing the game multiple times and reporting the average.

Shadow Modeling. State-of-the-art MIAs against synthetic data generators rely on the shadow modeling technique [34, 35, 20]. With this technique, the attacker leverages their access to the auxiliary dataset as well as the knowledge of the model, to train multiple instances of $\Phi_{T}$ (called $\Phi_{shadow}$) and evaluate the impact of the presence or absence of a target record on the resulting model. An attacker aiming to perform an MIA will proceed as follows. They will first create (e.g. by sampling) multiple datasets $D_{shadow}$ from $D_{aux}$, such that $|D_{shadow}|=|D|-1$. Note that we ensure $x_{T}\notin D_{aux}$. The attacker then adds $x_{T}$ to half of the $D_{shadow}$ and a random record $x_{R}$, distinct from $x_{T}$, to the other half. Using the completed $D_{shadow}$ (now $|D_{shadow}|=|D|$) and the knowledge of model $\Phi_{T}$, the attacker will now train multiple shadow generators $\Phi_{shadow}$. The attacker will then use the $\Phi_{shadow}$ to produce synthetic datasets $D_{shadow}^{s}$, each labeled with either IN if $x_{T}\in D_{shadow}$ or OUT otherwise. Using the labeled dataset, the attacker can now train a meta-classifier $\mathcal{M}_{meta}$ to distinguish between cases where the target record was and was not part of the training dataset. More specifically, $\mathcal{M}_{meta}$ would be trained using $D_{shadow}^{s}$ and the constructed binary label IN or OUT. At inference time, the attacker would then query the $\mathcal{M}_{meta}$ on the released $D^{s}$ for records of interest, and return its prediction.

Houssiau et al. [20] developed an MIA that instantiates the shadow modeling technique (see Sec. 2.3 for details) by training a random forest classifier on a subset of $k$-way marginal statistics that select the targeted record. The authors evaluated this approach on only one outlier record and found it to outperform other methods. Throughout our experiments, we found the attack to consistently outperform alternative methods and thus confirm it to be the state-of-the-art attack against synthetic tabular data. We refer to it as the query-based attack.

Given a target record $x_{T}$ and a synthetic dataset $D^{s}$ sampled from a generator fitted on a dataset $D$, this attack computes counting queries $Q^{A}$ to determine how many records in the synthetic dataset match a subset $A\subset\{1,\ldots,F\}$ of attribute values of target record $x_{T}$. We denote by $a_{i}$ the $i$-th attribute for every $x\in D^{s}$ in the following: $Q^{A}(x_{T})=\text{{COUNT}}\text{ WHERE }\bigwedge_{i\in A}(a_{i}=x_{T,i})$. Counting queries are equal to $k$-way marginal statistics computed on the attribute values of the target record, multiplied by the dataset size $|D^{s}|$.

Our intuition behind this attack is twofold. First, when these statistics are preserved between the original and synthetic datasets, the answers are larger by 1, on average, when $x_{T}\in D$ compared to when $x_{T}\notin D$, since the queries select the target record. Second, the impact of the target record on the synthetic dataset is likely to be local, i.e., more records matching a subset of its attribute values are likely to be generated when $x_{T}\in D$, leading to statistically different answers to the same queries depending on the membership of $x_{T}$.

Like Houssiau et al. [20], we randomly sample a predetermined number $N$ of attribute subsets $A$ and feed the answers to the associated $Q^{A}$, computed on the synthetic shadow dataset, to a random forest meta-classifier. For computational efficiency reasons, we here use a C-based implementation of the query computation [9] instead of the authors’ Python-based implementation [20].

We further extend the method to also account for continuous columns by considering the less-than-or-equal-to $\leq$ operator on top of the equal operator $=$ that was previously considered exclusively. For each categorical attribute we then use $=$, while using the $\leq$ for each continuous attribute.

We compare our approach against three baselines to identify vulnerable records:

Random: randomly sample target records from the entire population.

Rare Value (Groundhog) [35] sample a target record that either has a rare value for a categorical attribute or a value for a continuous attribute that is larger than the 95th percentile of the respective attribute.

Log-likelihood (TAPAS) [20]: sample target records having the lowest log-likelihood, assuming attribute values are independently drawn from their respective empirical distributions.

Note that this approach is defined by the authors only for categorical attributes $f\in\mathcal{F}_{\text{cat}}$: for each record $x_{i}$, they compute the frequency of the value $x_{i,f}$ in the entire dataset, resulting in $p_{i,f}$. The likelihood of a record is defined by the product of $p_{j,f}$ across all attributes. We here extend this approach to also account for continuous attributes $f\in\mathcal{F}_{\text{cont}}$ by discretising them according to percentiles.

To evaluate the different methods to select vulnerable records, we run the query-based MIA described in Sec. 5.1 on the ten records identified by each of the three methods as well as our own approach.

We evaluate our results against three generative models using the implementation available in the reprosyn repository [1].

SynthPop first estimates the joint distribution of the original, private data to then generate synthetic records. This joint distribution consists of a series of conditional probabilities which are fitted using classification and regression trees. With a first attribute randomly selected, the distribution of each other attribute is sequentially estimated based on both observed variables and previously generated synthetic columns. Initially proposed as an R package [28], the reprosyn repository uses a re-implementation in Python [18].

BayNet uses Bayesian Networks to model the causal relationships between attributes in a given dataset. Specifically, the attributes are represented as a Directed Acyclic Graph, where each node is associated with an attribute and the directed edges model the conditional independence between the corresponding attributes. The corresponding conditional distribution of probabilities $\mathbb{P}[X|Parents(X)]$ is estimated on the data using a GreedyBayes algorithm (for more details we refer to Zhang et al. [42]). Synthetic records can then be sampled from the product of the computed conditionals, i.e. the joint distribution.

PrivBayes is a differentially private version of the BayNet algorithm described above. Here, a first Bayesian Network is trained under an $\epsilon_{1}$-DP algorithm. The conditional probabilities are then computed using an $\epsilon_{2}$-DP technique, leading to a final $\epsilon=(\epsilon_{1}+\epsilon_{2})$-DP mechanism. From this approximate distribution, we can generate synthetic records without any additional cost of privacy budget. Note that for $\epsilon\to\infty$ PrivBayes becomes equivalent to BayNet. Again, we refer the reader to Zhang et al. [42] for a more in-depth description.

We evaluate our method against two publicly available datasets.

UK Census [29] is the 2011 Census Microdata Teaching File published by the UK Office for National Statistics. It contains an anonymised fraction of the 2011 Census from England and Wales File (1%), consisting of 569741 records with 17 categorical columns.

Adult [31] contains an anonymized set of records from the 1994 US Census database. It consists of 48842 records of 15 columns. 9 of those columns are categorical, and 6 are continuous.

Table 1 shows the parameters we use for the attack. We use a given dataset $\Omega$ (e.g. Adult) to select our most at-risk records as the target records. We then partition $\Omega$ randomly into $D_{test}$ and $D_{aux}$. $D_{aux}$ is the auxiliary knowledge made available to the attacker, while $D_{test}$ is used to sample $n_{test}$ datasets used to compute the attack performance. $n_{shadow}$ represents the number of shadow datasets used, sampled from the auxiliary knowledge $D_{aux}$. We considered, in all experiments, the size of the release synthetic dataset ($D^{s}$) to be equal to the size of the private dataset $D$, $|D|=|D^{s}|=1000$. To create the shadow datasets, while ensuring that $D_{aux}$ does not contain the target record, we randomly sample $|D|-1$ records in $D_{aux}$. Then we add the target record $x_{T}$ in half of the datasets, and in the other half, we used another record randomly sampled from $D_{aux}$.

We run the query-based attack using $N=100000$ queries randomly sampled from all possible count queries ($2^{F}$ in total) and a random forest meta-classifier with 100 trees, each with a maximum depth of 10.

In order to measure the attack performance, we compute the Area Under the receiver operating characteristic Curve (AUC) for the binary classification of membership on $n_{test}$ datasets.

Lastly, in line with the auditing scenario, we use $k=5$ neighbors in the private dataset $\Omega$ to select the most vulnerable records using our vulnerability score $V_{k}$.

We evaluate the effectiveness of vulnerable record identification methods across the two datasets UK Census and Adult, and two synthetic data generators Synthpop and BayNet. For each dataset, each of the four record identification methods selects 10 target records. We then perform the MIA, as a privacy game, on each of these records and report its AUC.

Figure 1 show that our vulnerable record identification method consistently outperforms both the random baseline and the two ad-hoc techniques previously used in the literature. The median risk of records identified by our method is indeed consistently higher than all other methods, up to 18.6 p.p. higher on the UK Census dataset using BayNet. On average, the AUC of the records we identified are 7.2 p.p. higher than the AUC of the records selected by other methods (Table 2). Importantly, our method also consistently manages to identify records that are more or equivalently at-risk compared to other methods.

MIAs against synthetic data is an active area of research with a competition [2] now being organised to develop better, more accurate, MIAs. To evaluate the effectiveness of our vulnerable record identification, we developed a new attack which we aimed to be as different as possible to the existing state of the art. We call this new attack target attention.

In contrast with the state-of-the-art method, which first computes features from a synthetic dataset to then use these as input for a meta-classifier, our new attack takes as input (a part of) the synthetic dataset to directly predict membership. As such, the model would be able to extract directly from the synthetic dataset the information that is most useful for the MIA. To our knowledge, it is the first method that is record-focused and allows for trainable feature extraction. For more information about the new method, we refer the reader to Appendix 0.A. We then follow the same methodology as above to evaluate the effectiveness of the four vulnerable record identification methods when using the target attention approach.

Figure 2 shows that, again, our vulnerable record identification method strongly outperforms both the random baseline and the two ad-hoc techniques previously used in the literature. The mean risk of records identified by our method is again consistently higher than all other methods, up to 10.89 p.p. higher on UK Census using BayNet. Across datasets and generators, the records we identify are 5.2 p.p. more at risk than the records selected by other methods (Appendix 0.A).

Throughout the paper, we use $k=5$, meaning that our vulnerability score $V_{k}(x_{i})$ is computed using the 5 closest records to the record of interest. We here evaluate the impact of a specific choice of $k$ for the record identification and associated results. More specifically, we compute the mean distance for each record using values of $k$ ranging from 1 to 50 and report the mean AUC for the ten records selected by the method on the Adult dataset using Synthpop.

Figure 3(a) shows our method to be robust to specific choices of $k$. The mean AUC of the top ten records varies indeed only slightly for different values of $k$. Looking at the top 10 records selected for different values of $k$, we find that only a handful of records are distinct, suggesting our method to not be too sensitive to specific values of $k$.

We here rely on a generalized form of cosine distance across attribute types. We now evaluate the susceptibility of our results to different choices of distance metrics. We do this using several versions of the Minkowski distance between two vectors $a,b\in\mathbb{R}^{L}$. For a given value $p\in\mathbb{N}^{*}$, the Minkowski distance is defined as $d_{\text{Minkowski}}^{p}(a,b)=\sqrt[p]{\sum_{l=1}^{L}|a_{l}-b_{l}|^{p}}$. Similarly to Eq. 5, we generalise this distance metric for two records $x_{i},x_{j}\in D$ to:

Notably, the Minkowski distance with $p=2$ closely relates to the cosine distance. As before, we compute the mean distance for each record using Minkowski distances with $p$ ranging from 1 to 4 and report the mean AUC for the ten records selected by the method on the Adult dataset using Synthpop.

Figure 3(b) shows the attack performance to, again, be robust to different choices of distance metric. In particular, mean AUC is not significantly different for the cosine distance and $p\in\{1,2\}$, while slightly decreasing for higher $p$ values. Cosine being the standard distance metric in the literature, we prefer to use it.

An active line of research aims at developing differentially private (DP) synthetic data generators. As discussed above however, developing models that achieve DP with meaningful theoretical $\epsilon$ while maintaining high utility across tasks remains a challenge. MIAs are thus an important tool to understand the privacy protection offered by DP models for various values of $\epsilon$.

We here evaluate the performance of our vulnerable record identification approach when DP generators are used. Specifically, we report the mean and standard deviation AUC for the 10 records identified as being at risk by our approach and by the three baselines for varying values of $\epsilon$ using PrivBayes [42].

Figure 4 shows that our approach outperforms both the random baseline and the two ad-hoc techniques previously used in the literature. As expected though, the AUC drops when $\epsilon$ decreases. The attack still performs quite well for $\epsilon=100$ and manages to identify at-risk records for $\epsilon=10$. Importantly, the general utility of synthetic data generated by PrivBayes for low value of $\epsilon$ is being debated.