Fidel: Reconstructing Private Training Samples from Weight Updates in Federated Learning

The mathematical foundation of this attack method is formulated by Le Trieu et al. [17] and is repeated in this section as background information. Le Trieu et al. found that neurons in densely connected layers can reconstruct the activation of the previous layer by leveraging the fact that the weight and bias changes associated with this neuron are relative to the activation of the previous layer.

The activation of a neuron $i$ in a densely connected layer $l$ is formulated as $a_{i}=A(W^{l-1}_{i}\cdot a^{l-1}+b_{i})$ in which $b_{i}$ is the bias of neuron $i$, $W^{l-1}_{i}$ is the weight vector connecting layer $l-1$ to neuron $i$ and $A()$ is the activation function. For a small network that connects an input vector to a singular neuron as illustrated in Figure 2, the activation of the previous layer is the actual input values $x$ of a sample thus $a_{i}=A(W\cdot x+b)$. Generally, for a given input $x$ we note that $a^{0}=x$ and $h(x)$ is the activation of the last layer of the network.

Loss on a training sample $(x,y)$ is calculated by applying a loss function $\mathcal{L}$ on the network with its weights $W$ biases $b$. The loss determines the error of the weights based on the distance between the true value $y$ and the output value for $x$ as calculated by the network $h(x)$. For a single sample $(x,y)$ the loss function is $\mathcal{L}(W,b,x,y)$.

Two commonly used loss functions on which this attack works are squared error and cross-entropy loss functions. Using the squared error function as an example, the loss is calculated as $\mathcal{L}(W,b,x,y)=(h(x)-y)^{2}$.

To find the numerical direction in which the weights and biases of the network must move to provide a lower loss value for a specific training sample $(x,y)$, the gradient of each weight and bias are calculated.

where $j$ is the $j^{th}$ input element in the input space of $x$.

We can identify two important characteristics of this derivation. First, having $g_{w}$ and $g_{b}$ allows us to retrieve the entire input $x$ on which the network was trained by dividing $\frac{g_{w}}{g_{b}}=x$. Secondly, if $g_{b}$ is not known, $g_{w}$ allows for a reconstruction of the input $x$ that is linearly related to the true sample $x$. This principle is illustrated in Figure 3 in which three different samples can be derived from their respective training gradients even if the bias gradient is unknown.

After the gradients of a model are calculated (can be aggregation over multiple samples) the weights of the model are updated by $w_{t+1}\leftarrow w_{t}-\eta g$ in which $\eta$ represents the so called learning rate. Having the before and after weights ${w_{t+1},w_{t}}$ or the difference in weight value $\delta=w_{t+1}-w_{t}$ can result in the same reconstruction for $x$ as described above since

The previous section described how a neuron on any dense layer can reconstruct the activation of the previous layer if either the gradients of this layer or any form of weight update is known.

This can be exploited during regular federated learning operation since the server must receive information about the change of weights of the participating clients. This exploitable surface makes it so that federated learning in its basic form is fundamentally unsafe against MITM attack and server-side attacks. Using more training-samples per update or a different network architecture such as CNN is no guarantee against this attack as will be demonstrated later on in Section IV.

Commonly, dense layers have more than a singular neuron, thus multiple reconstructions of the same input can be reconstructed. This induces reconstruction robustness against noise, lossy compression or other model update perturbation. Additionally, communication bandwidth reduction techniques that rely on communicating randomly chosen weights or other partial weight update omittance schemes do not guarantee safety against this attack. Multiple neurons can be employed in order to collaboratively reconstruct the original sample.

In an FCNN, the first layer is densely connected to the input. In this experiment, the first layer consists of 128 neurons. Therefore, 128 partial reconstructions are available to the adversary. After training the network for some rounds on the training-set, the client trains on a singular training sample of the test-set. By executing the Fidel on the weight update of the model, the adversary now has 128 partial reconstructions available. The first 24 partial reconstructions for the MNIST and CIFAR-10 datasets are illustrated in Figure 5 in combination with the private training sample. The illustrated partial reconstructions values are normalized to be displayed properly.

It is clearly visible how such reconstructions reveal the underlying data. Note that some partial reconstructions are fully black. This is because of the application of the ReLu activation function; if the sum of weighted inputs is negative, the weights of this neuron will have no gradient and no reconstruction can be made from this neuron.

Neural networks trained for image recognition often implement convolutional layers followed by max-pooling layers. The partial reconstructions for a CNN are thus the convoluted and subsequently max-pooled input samples. The partial reconstructions for the first 24 filters of the first neuron are illustrated in Figure 6. This small subset is only 24 of the possible 32 * 128 = 4096 (13x13 & 15x15) partial reconstructions available to the adversary.

These partial reconstructions may reveal information outright. The MNIST reconstructions show this particularly well. This is because a convolution operation often retains some spatial information of their input. A larger amount of stacked convolutional layers may reduce the ability to reveal private information in partial reconstructions. Reconstructing the original sample from such partial reconstructions can be done in various ways. In this paper, we use a generative neural network as an example.

For an FCNN, the gradient update of each neuron in the first layer can provide a full reconstruction of the provided input sample. Figure 2 illustrates the first layer of such a network. Commonly such a network has more than one neuron indicating that multiple reconstructions are possible for each weight update.

We use two basic networks for the experiments in this paper as the target victim networks, represented by an FCNN as described in Table I, and a CNN as described in Table II. These networks are used for demonstrative purposes and are not optimized for any specific application.

We demonstrate the experiments on the MNIST [26] and CIFAR-10 [27] datasets which consist of 70k 28x28 gray-scale and 60k 32x32 RGB images, respectively. Among these images, we reserve 10k images as a test set in both datasets. These datasets are intended for image classification tasks with 10 distinct labels.

In this section, we illustrate the first 24 partial reconstructions out of a possible 128 formed from the weight update of a client’s network trained for one epoch on a batch of 10 samples. Applying the attack yields partial reconstructions as illustrated in Figure 9

From these examples, it is possible to see that even for models trained on more than only one sample, information is still revealed. For the CIFAR-10 example, the 3 partial reconstructions annotated by numbers (1, 2 and 3) are particularly revealing of the original private samples. It is interesting to note that in the figure, there is one neuron that did not activate for any of the provided samples and thus gave a fully black reconstruction.

Without any additive noise or other data modification, the partial reconstructions are combinations of the input images individually scaled with some (possibly negative) factor. Therefore, theoretically, these partial reconstructions can be used to retrieve all underlying samples employing blind source separation algorithms. We do not explore this notion further in this paper.

Figure 10 shows the first 24 out of 32 filters for a single neuron on a model training update of 10 samples. Note that the neuron is particularly responsive to the 3rd sample for MNIST and the 7th sample for CIFAR-10.

Using the same generative method as demonstrated above, we show generated reconstructions from a local private data set of 10 samples. Each set of partial reconstructions retrieved associated with a neuron is now used to generate possible reconstructions. In Figure 15, 100 reconstructions for an MNIST sample batch and 120 reconstructions for a CIFAR-10 sample batch are illustrated. Within these reconstructions, nearly all private samples can be identified (albeit with varying degrees of fidelity).

To understand why it is possible to see fully reproduce images when training on a batch of multiple images, we introduce the term negligible gradients. This term is used to identify a gradient of a sample (for a single neuron) which is an order of magnitude smaller than another sample gradient update. A demonstration of this is given in Figure 11. In this figure, a client trained on the FCNN network with 3 local samples (at the top) can leak (among others) the shown two partial reconstructions (at the bottom). Within these partial reconstructions, the number four and five are visible but do not obfuscate any private information about the number nine.

In Figure 9, we showed that for a batch of 10 samples it is relatively likely that in an FCNN a neuron is only activated on a single sample of the whole batch. This phenomenon propagates with the CNN network where a neuron in the first dense layer (after the convolution layers) only activates for a single sample, which in turn allows for reconstruction as shown in Figure 10.

Two noteworthy factors influence this phenomenon to the detriment of privacy in the federated learning process. First, the usage of the ReLu activation function greatly enhances the likelihood that a neuron is only activated on a single sample. The absence of positive value ceilings in ReLu increases the likelihood that the activation of a sample is so large that all other samples have negligible gradient updates on its weights. Second, the usage of dropout after the first dense layer also increases the likelihood that a neuron is only activated for a single sample in the batch. Because dropout forces a samples gradient to zero for randomly chosen neurons.

For image data, evaluating if private information is leaked is often not dependent on the value of single pixels. The context of the rest of the image can often provide enough information to compensate for the erroneous pixels. The reconstruction obtained with the Fidel may be scaled by some unknown (possibly negative) value.

To evaluate if a reconstruction is fully revealing a private sample, we use the Pearson correlation coefficients [28] between the reconstruction and the original sample. This metric is invariant against scaling of all sample values by some constant as well as invariant against adding some constant to all values in the sample. This metric is used to demonstrate that a linear relationship exists between the private sample and the reconstructed sample.

As a threshold for the complete reveal of a sample, we chose a correlation coefficient of 0.98 to indicate that all private information of the input image is fully revealed. This threshold provides some robustness against negligible gradients and floating-point calculation errors. In a practical setting, two images taken right after one another with the same equipment can be expected to have at best such a correlation coefficient [29].

To justify this threshold Figure 12 shows example correlation coefficients between private samples and reconstructions in the batch FCNN setting. As depicted in this figure, correlations lower than 0.98 may still reveal sensitive private information, especially for image data.

For the FCNN MNIST demonstration, we test varying numbers of private samples used to train the local model to see how many of the underlying samples are fully revealed in the weight update. These results are illustrated in Figure 13. Private samples are not counted multiple times for different neurons, so it may very well be that multiple neurons fully reveal the same underlying private sample. The graph describes the average revealed samples per round over 200 measurements. Indicating the average per round reveal over 200 rounds of federated training. Notably, the graph shows the average sample reveal that is greater than one for nearly all scenarios. This means that having a large local dataset does not ensure privacy preservation as data samples can still be revealed with local datasets as large as 200 samples.

The amount of fully revealed private samples is incredibly depend on the employed activation function as well as the usage of dropout. The graph shows that in using the ReLu activation function, the amount of revealed private samples is significantly higher than other activation functions. Using dropout is demonstrably effective in allowing the Fidel attack to succeed. The worst-case scenario in this experiment shows that as much as $\frac{2}{3}$ of a local dataset of 30 samples can be reconstructed.

If the first dense layer has more neurons than in the network employed in this experiment, we expect that even more private samples may be revealed per weight update. This is based on the notion that the neurons may be expected to reveal samples in a non-dependent probabilistic fashion. Additionally, the probabilistic nature of a neuron activating for a singular training sample means that the expected amount of revealed samples will increase cumulatively over the number of training rounds.

We perform the same experiment on the generative method with the CNN network and the CIFAR-10 dataset. The results of this experiment are illustrated in Figure 14. Here too, we find that especially for smaller sized local datasets the weight update may reveal private information that can be reconstructed to near-perfect accuracy.

This graph shows that for the CNN network the local dataset size is of greater influence and the amount of fully revealed samples overall are far less than with the FCNN experiment. We can attribute this, in part, to the generative neural network that was employed for reconstruction. Additionally, the max-pooling layer might remove essential information needed for a full reconstruction.

From these measurements, it seems that under most circumstances the activation function and network design are of great influence to the preservation of privacy. It is important to re-emphasize that the graphs solely show fully revealed images for the chosen threshold of 0.98 correlation or more. As shown in Figure 12, lower correlations may still reveal private information.

Federated learning has been designed specifically for the setting of numerous clients with small datasets. The threat of this attack may, therefore, be assessed among other things on the distribution of the data among clients i.e. the number of samples in a clients local dataset.