Privacy-Preserving Feature Selection with Secure Multiparty Computation

Assume that we have a set $S$ of $m$ training examples, where each training example consists of an input feature vector $(x_{1},\ldots,x_{p})$ and a corresponding label $y$. Throughout this paper, we assume that there are $n$ possible class labels. We wish to induce an ML model from this training data that can infer, for a previously unseen input feature vector, a label $y$ as accurately as possible. Not all $p$ features may be equally beneficial to this end. In the filter approach to feature selection, all features are first assigned a score that is indicative of their predictive ability. Subsequently only the best scoring features are retained. A well-known feature scoring criterion is Gini impurity, made popular as part of the classification and regression tree algorithm (CART) [7].

If the $j^{th}$ feature $F_{j}$ is a discrete feature that can assume $\ell$ different values, then it induces a partition $S_{1}\cup S_{2}\cup\ldots\cup S_{\ell}$ of $S$ in which $S_{i}$ is the set of instances that have the $i^{th}$ value for the $j^{th}$ feature. The Gini impurity of $S_{i}$ is defined as:

where $p_{c}$ is the probability of a randomly selected instance from $S_{i}$ belonging to the $c^{th}$ class. The Gini score of feature $F_{j}$ is a weighted average of the Gini impurities of the $S_{i}$’s:

Conceptually, $G(F_{j})$ estimates the likelihood of a randomly selected instance to be misclassified based on knowledge of the value of the $j^{th}$ feature. During feature selection, the $k$ features with the lowest Gini scores are retained.

If $F_{j}$ is a feature with continuous values, then $G(F_{j})$ is defined as the weighted average of the Gini impurities of a set $S_{\leq\theta}$ containing all instances for which the $j^{th}$ feature value is smaller than or equal to $\theta$, and a set $S_{>\theta}$ with all instances for which the $j^{th}$ feature value is larger than $\theta$. In the CART algorithm, an optimal threshold $\theta$ is determined based on sorting of all the instances on their feature values. Since privacy-preserving sorting is a time-consuming operation in MPC [6, 20], in Sec. IV-B we propose a more straightforward approach for threshold selection which, as we show in Sec. V, yields desirable improvements in accuracy.

Protocols for MPC enable a set of parties to jointly compute the output of a function over each of the parties’ private inputs, without requiring parties to disclose their input to anyone. MPC is concerned with the protocol execution coming under attack by an adversary which may corrupt parties to learn private information or cause the result of the computation to be incorrect. MPC protocols are designed to prevent such attacks being successful, and use proven cryptographic techniques to guarantee privacy.

Adversarial Model: An adversary $\mathcal{A}$ can corrupt any number of parties. In a dishonest-majority setting, half or more of the parties may be corrupt, while in an honest-majority setting, more than half of the parties are honest (not corrupted). Furthermore, $\mathcal{A}$ can be a semi-honest or a malicious adversary. While a party corrupted by a semi-honest or “passive” adversary follows the protocol instructions correctly but tries to obtain additional information, parties corrupted by malicious or “active” adversaries can deviate from the protocol instructions. The protocols in Sec. IV are sufficiently generic to be used in dishonest-majority as well as honest-majority settings, with passive or active adversaries. This is achieved by changing the underlying MPC scheme to align with the desired security setting. Some of the most efficient MPC schemes have been developed for 3 parties, out of which at most one is corrupted. We evaluate the runtime of our protocols in this honest-majority 3PC setting, which is growing in popularity in the PPML literature, e.g. [14, 24, 31, 34], and we demonstrate how even better runtimes can be obtained with a recently proposed MPC scheme for 4PC with one corruption [13].

In the MPC schemes used in this paper, all computations by the parties (servers) are done over integers in a ring $\mathbb{Z}_{q}$. Raw data in ML applications is often real-valued. As is common in the MPC literature, we convert real numbers to integers using a fixed-point representation [9]. After this conversion, the data owners secret share their values with the parties using a secret sharing scheme and proceed by performing operations over the secret shares.

For the passive 3PC setting, we follow a replicated secret sharing scheme from Araki et al. ([4]). To share a secret value $x\in\mathbb{Z}_{q}$ among parties $P_{1},P_{2}$ and $P_{3}$, the shares $x_{1},x_{2},x_{3}$ are chosen uniformly at random in $\mathbb{Z}_{q}$ with the constraint that $x_{1}+x_{2}+x_{3}=x\mod q$. $P_{1}$ receives $x_{1}$ and $x_{2}$, $P_{2}$ receives $x_{2}$ and $x_{3}$, and $P_{3}$ receives $x_{3}$ and $x_{1}$. Note that it is necessary to combine the shares available to two parties in order to recover $x$, and no information about the secret shared value $x$ is revealed to any single party. For short, we denote this secret sharing by $[\![x]\!]_{q}$. Let $[\![x]\!]_{q}$, $[\![y]\!]_{q}$ be secret shared values and $c$ be a constant, the following computations can be done locally by parties without communication:

• •

  Addition ($z=x+y$): Each party $P_{i}$ gets shares of $z$ by computing $z_{i}=x_{i}+y_{i}$ and $z_{(i+1{\rm\ mod\ }3)}$ $=$ $x_{(i+1{\rm\ mod\ }3)}$ $+$ $y_{(i+1{\rm\ mod\ }3)}$. This is denoted by $[\![z]\!]_{q}\leftarrow[\![x]\!]_{q}+[\![y]\!]_{q}$.

• •

  Subtraction $[\![z]\!]_{q}\leftarrow[\![x]\!]_{q}-[\![y]\!]_{q}$ is performed analogously.

• •

  Multiplication by a constant ($z=c\cdot x$): Each party multiplies its local shares of $x$ by $c$ to obtain shares of $z$. This is denoted by $[\![z]\!]_{q}\leftarrow c\cdot[\![x]\!]_{q}$

• •

  Addition of a constant ($z=x+c$): $P_{1}$ and $P_{3}$ add $c$ to their share $x_{1}$ of $x$ to obtain $z_{1}$, while the parties set $z_{2}=x_{2}$ and $z_{3}=x_{3}$. This will be denoted by $[\![z]\!]_{q}\leftarrow[\![x]\!]_{q}+c$.

The main advantage of replicated secret sharing compared to other secret sharing schemes is that replicated shares enables a very efficient procedure for multiplying secret shared values. To compute $x\cdot y=(x_{1}+x_{2}+x_{3})(y_{1}+y_{2}+y_{3})$, the parties locally perform the following computations: $P_{1}$ computes $z_{1}$ $=$ $x_{1}\cdot y_{1}$ $+$ $x_{1}\cdot y_{2}$ $+$ $x_{2}\cdot y_{1}$, $P_{2}$ computes $z_{2}$ $=$ $x_{2}\cdot y_{2}$ $+$ $x_{2}\cdot y_{3}$ $+$ $x_{3}\cdot y_{2}$ and $P_{3}$ computes $z_{3}$ $=$ $x_{3}\cdot y_{3}$ $+$ $x_{3}\cdot y_{1}$ $+$ $x_{1}\cdot y_{3}$. By doing so, without any interaction, each $P_{i}$ obtains $z_{i}$ such that $z_{1}+z_{2}+z_{3}=x\cdot y\mod q$. After that, the parties are required to convert from this additive secret sharing representation back to the original replicated secret sharing representation (which requires that the parties add a secret sharing of zero and that each party sends one share to one other party for a total communication of three shares). See [4] for more details.

In the active 3PC setting, we use the MPC scheme SYReplicated2k recently proposed by Dalskov et al. ([13]). In this MPC scheme, the parties are prevented from deviating from the protocol and from gaining knowledge from other parties through the use of information-theoretic message authentication codes (MACs). In addition to computations over secret shares of the data, the parties also perform computations required for MACs. See [13] for details. Finally, we use the MPC scheme recently proposed by Dalskov et al. ([13]) for the active 4PC setting, where the computations are outsourced to four servers out of which at most one has been corrupted by a malicious adversary.

Building Blocks: Building on the cryptographic primitives listed above for addition and multiplication of secret shared values, MPC protocols for other operations have been developed in the literature. In this paper, we use:

• •

  Secure matrix multiplication $\pi_{\mathsf{DMM}}$: at the start of this protocol, the parties have secret sharings $[\![A]\!]$ and $[\![B]\!]$ of matrices $A$ and $B$; at the end, the parties have a secret sharing $[\![C]\!]$ of the product of the matrices, $C=A\times B$. $\pi_{\mathsf{DMM}}$ can be constructed as a direct extension of the secure multiplication protocol for two integers, which we will denote as $\pi_{\mathsf{DM}}$ in the remainder of the paper. Similarly, we use $\pi_{\mathsf{DP}}$ to denote the protocol for the secure dot product of two vectors. In a replicated sharing scheme, dot products can be computed more efficiently than the direct extension from $\pi_{\mathsf{DM}}$, and matrix multiplication can use this optimized version of dot products; we refer to Keller ([23]) for details.

• •

  Secure comparison protocol $\pi_{\mathsf{LT}}$ [8]: at the start of this protocol, the parties have secret sharings $[\![x]\!]$ and $[\![y]\!]$ of two integers $x$ and $y$; at the end, they have a secret sharing of $1$ if $x<y$, and a secret sharing of $0$ otherwise.

• •

  Secure argmin protocol $\pi_{\mathsf{ARGMIN}}$: this protocol accepts secret sharings of a vector of integers and returns a secret sharing of the index at which the vector has the minimum value. $\pi_{\mathsf{ARGMIN}}$ is straightforwardly constructed using the above mentioned secure comparison protocol.

• •

  Secure equality test protocol $\pi_{\mathsf{EQ}}$ [9]: at the start of this protocol, the parties have secret sharings $[\![x]\!]$ and $[\![y]\!]$ of two integers $x$ and $y$; at the end, they have a secret sharing of $1$ if $x=y$, and a secret sharing of $0$ otherwise.

• •

  Secure division protocol $\pi_{\mathsf{DIV}}$ [9]: at the start of this protocol, the parties have secret sharings $[\![x]\!]_{q}$ and $[\![y]\!]_{q}$ of two integers $x$ and $y$; at the end, they have a secret sharing $[\![z]\!]_{q}$ of $z=x/y$.

At the start of the Protocol $\pi_{\mathsf{FILTER-FS}}$ for secure feature selection, the parties have secret shares of a data matrix $D$ of size $m\times p$, in which the rows correspond to instances and the columns to features. The parties also have secret shares of a vector $G$ of length $p$ containing a score for each of the features. At the end of the protocol, the parties have a reduced matrix $D^{\prime}$ of size $m\times k$ in which only the columns from $D$ corresponding to the lowest scores in $G$ are retained (note that this protocol can be trivially modified to select the $k$ features with the highest scores). The main ideas behind the protocol (which is described in Protocol 1) are to:

1. 1.

   Determine the indices of the features that need to be selected (these are stored in a secret-shared way in $I$).

2. 2.

   Create a matrix $T$ in which the columns are one-hot-encoded representations of these indices.

3. 3.

   Multiply $D$ with this feature selection matrix $T$.

Example 1. Consider the data matrix $D$ at the left of Equation (3), containing values for $m=5$ instances (rows) and $p=4$ features (columns). Assume that the feature score vector is $G=[65,26,83,14]$ and that we want to select the $k=2$ features with the lowest scores in $G$.

The lowest scores in $G$ are 14 and 26, hence the 4th and the 2nd column of $D$ should be selected. The columns of $T$ in Equation (3) are a one-hot-encoding of 4 and 2 respectively, and multiplying $D$ with $T$ will yield the desired reduced data matrix $D^{\prime}$. This multiplication takes place on Line 9 in Protocol 1. The bulk of Protocol 1 is about how to construct $T$ based on $G$. As explained below, this process involves an auxiliary vector, which, at the end of the protocol, contains the following values for our example: $I=[4,2]$.

In the protocol, vector $[\![I]\!]_{q}$ of length $k$ stores the indices of the $k$ selected features out of the $p$ features of $[\![D]\!]_{q}$ and matrix $[\![T]\!]_{q}$ is a $p\times k$ transformation matrix that eventually holds one-hot-encodings of the indices in $I$. Through executing Lines 1-8 of Protocol 1, the parties construct a feature selection matrix $T$ based on the values in $G$. In Line 2 the index of the $i^{th}$ smallest value in $[\![G]\!]_{q}$ is identified. To this end, the parties run a secure argmin protocol $\pi_{\mathsf{ARGMIN}}$. The inner for-loop serves two purposes, namely constructing the $i^{th}$ column of matrix $T$, and overwriting the score in $G$ of the feature that was selected in Line 2 by the upper bound, so that it will not be selected anymore in further iterations of the outer for-loop (such an upper bound $t$ is passed as input to Protocol 1 and is usually very easy to determine in practice, as most common feature scoring techniques range between $0$ and $1$):

• •

  To construct the $i^{th}$ column of $T$, the parties loop through row $j=1\ldots p$, and on Line 5, update $T[j][i]$ with either a 0 or a 1, depending on the outcome of the secure equality test on Line 4. The outcome of this test will be 1 exactly once, namely when $j$ equals $I[i]$, hence Line 5 results in a one-hot-encoding of $I[i]$ stored in the $i$th column of $T$.

• •

  The flag $flag_{k}$ computed on Line 4 is used again on Line 6 to overwrite $G[I[i]]$ with $t$ in an oblivious manner, where $t$ is a value that is larger than the highest possible score that occurs in $[\![G]\!]_{q}$. This theoretical upper bound $t$ ensures that feature $I[i]$ will not be selected again in later iterations of the outer for-loop.

As is common in MPC protocols, we use multiplication instead of control flow logic for conditional assignments. To this end, a conditional based branch operation as “$\textbf{if }c\textbf{ then }a\leftarrow b$” is rephrased as $a\leftarrow a+c\cdot(b-a)$. In this way, the number and the kind of operations executed by the parties does not depend on the actual values of the inputs, so it does not leak information that could be exploited by side-channel attacks. Such a conditional assignment occurs in Line 6 of Protocol 1, where the value of the condition $c$ itself is computed on Line 4. In the final step, on Line 9, the parties multiply matrix $D$ with matrix $T$ in a secure manner to obtain a matrix $D^{\prime}$ that contains only the feature columns corresponding to the $k$ best features. Throughout this process, the parties are unaware of which features were actually selected. The secret shared matrix $D^{\prime}$ can subsequently be used as input for a privacy-preserving ML model training protocol, e.g. [16].

Protocol $\pi_{\mathsf{FILTER-FS}}$ assumes the availability of a feature score vector $G$ and an upper bound $t$ for the values in $G$. Below we explain how this can be obtained from the data in a secure manner. To this end, we present a protocol $\pi_{\mathsf{MS-GINI}}$ for computation of the score of a feature based on Gini impurity. This protocol is applicable to data sets with continuous features. It is computationally cheaper than previously proposed protocols for Gini impurity that rely on sorting of feature values. Furthermore, as shown in previous work [25] and in Sec. V, the “Mean-Split” GINI score can yield similar accuracy improvements.

Recall that we have a set $S$ of $m$ training examples, where each training example consists of an input feature vector $(x_{1},\ldots,x_{p})$ and a corresponding label $y$. We propose to split the set of values of the $j^{th}$ feature $F_{j}$ based on its mean value as a threshold $\theta$. We denote by $S_{\leq\theta}$ the set of instances that have $x_{j}\leq\theta$, and by $S_{>\theta}$ the set of instances that have $x_{j}>\theta$. Furthermore, for $c=1,\ldots,n$, we denote by $L_{c}$ the set of examples from $S$ that have class label $y=c$. Based on the binary split, we define the MS-GINI (“Mean-Split” GINI) score for feature $F_{j}$ as:

with the Gini impurities of $S_{\leq\theta}$ and $S_{>\theta}$ defined as:

and the probabilities defined as:

Formulas (4), (5) and (6) are consistent with the definition of Gini score given in Sec. II, and presented here in more detail to enhance the readability of our secure protocol $\pi_{\mathsf{MS-GINI}}$ for the computation of the Gini score $G(F)$ of feature $F$ (described in Protocol 2).

At the start of Protocol $\pi_{\mathsf{MS-GINI}}$, the parties have secret shares of a feature column $F$ (think of this as a column from data matrix $D$ in Example 1), as well as secret shares of an one-hot-encoded version of the label vector. The latter is represented as a label-class matrix $[\![L]\!]_{q}$, in which $[\![L[i][j]]\!]_{q}=[\![1]\!]_{q}$ means that the label of the $i^{th}$ instance is equal to the $j^{th}$ class. Otherwise, $[\![L[i][j]]\!]_{q}=[\![0]\!]_{q}$. We note that, while there are $n$ classes, it is sufficient for $L$ to contain only $n-1$ columns: as there is exactly one value 1 per row, the value of the $n^{th}$ column is implicit from the values of the other columns. We indirectly take advantage of this fact by terminating the loop on Line 6-10 at $n-1$, and performing calculations for the $n^{th}$ class separately and in a cheaper manner on Line 13-14, as we explain in more detail below.

On Line 1, the parties compute $[\![\theta]\!]_{q}$ as a threshold to split the input feature $[\![F]\!]_{q}$, as the mean of the feature values in the column. To this end, each party first sums up the secret shares of the feature values, and then multiplies the sum with a known constant $\frac{1}{m}$ locally. Line 2 is to initialize all counters related to $S_{\leq\theta}$ and $S_{>\theta}$ to zero. After Line 14, these counters will contain the following values:

These counters are needed for the probabilities in Equation (6). For each instance, in Line 4 of Protocol 2, the parties perform a secure comparison to determine whether the instance belongs to $S_{>\theta}$. The outcome of that test is added to $b$ on Line 5. Since the total number of instances is $m$, $a$ can be straightforwardly computed as $m-b$ after the outer for-loop, i.e. on Line 12. Lines 7-8 check whether the instance belongs to $S_{>\theta}\cap L_{j}$, in which case $B[j]$ is incremented by 1. The equivalent operation of Line 7-8 for $A[j]$ would be $[\![A[j]]\!]_{q}\leftarrow[\![A[j]]\!]_{q}+\pi_{\mathsf{DM}}((1-[\![flag_{s}]\!]_{q}),[\![L[i][j]]\!]_{q})$. We have simplified this instruction on Line 9, taking advantage of the fact that $\pi_{\mathsf{DM}}([\![flag_{s}]\!]_{q},[\![L[i][j]]\!]_{q})$ has been precomputed as $[\![flag_{m}]\!]_{q}$ on Line 7.

On Line 13-14 the parties compute $[\![A[n]]\!]_{q}$ and $[\![B[n]]\!]_{q}$, leveraging the fact that sum of all values in $[\![A]\!]_{q}$ is $[\![a]\!]_{q}$, and the sum of all values in $[\![B]\!]_{q}$ is $[\![b]\!]_{q}$. All operations on Line 13-14 can be performed locally by the parties, on their own shares. Moving the computation of $[\![A[n]]\!]_{q}$ and $[\![B[n]]\!]_{q}$ out of the for-loop, reduces the number of secure multiplications needed from $m\times n$ to $m\times(n-1)$. In the case of a binary classification problem, i.e. $n=2$, this means that the number of secure multiplications required is cut down by half.

Using the notations for the counters from the pseudocode of Protocol 2, Equation (4) comes down to:

If data are vertically partitioned and all data owners have the label vector, they can compute MS-GINI scores offline without $\pi_{\mathsf{MS-GINI}}$, and the computing servers would only have to do feature selection based on pre-computed MS-GINI scores with Protocol $\pi_{\mathsf{FILTER-FS}}$. In reality, often, it is not reasonable to allow each data owner to have all labels, so we do not assume this scenario in our protocols.

Protocol $\pi_{\mathsf{GINI-FS}}$ (described in Protocol 3) performs secure filter-based feature selection with MS-GINI, used for the experiments in this work. It combines the building blocks presented earlier in the section. By executing the loop on Line 1-3, the parties compute the MS-GINI score of the $i^{th}$ feature from the original data matrix $[\![D]\!]_{q}$ using Protocol $\pi_{\mathsf{MS-GINI}}$, and store it into $[\![G[i]]\!]_{q}$. On Line 4, the parties perform filter-based feature selection using Protocol $\pi_{\mathsf{FILTER-FS}}$ to obtain a $m\times k$ matrix $[\![D^{\prime}]\!]_{q}$ with $k$ selected features from $[\![D]\!]_{q}$. As the standard GINI score is upper bounded by 1, and $\pi_{\mathsf{MS-GINI}}$ ignores the multiplication by $1/m$ for efficiency reasons, it is safe to use $m$ as the upper bound that is passed to Protocol $\pi_{\mathsf{FILTER-FS}}$ on Line 4.