Balancing Privacy Protection and Interpretability in Federated Learning

As a typical gradient-based interpretability approach, Gradient-weighted Class Activation Mapping (Grad-CAM) [29] aims to provide visual explanations on the tasks of computer vision by using the gradients to capture the salient regions of feature map in the last convolutional layer of deep neural networks.

In this paper, we use image classification as a running example to introduce the basic idea of Grad-CAM. Let $y^{c}$ be the prediction score (before softmax activation) of class $c$ and $Z^{k}$ denote the feature map of a certain (e.g., last) convolutional layer. The gradient of $y^{c}$ with respect to $Z^{k}$ is denoted by $\frac{\partial y^{c}}{\partial Z^{k}}$. Then we can leverage the gradients via back-propagation to get the following neuron importance weights as:

where $k$ represents the $k$-th channel of feature map $Z$, $M$ represents the product of the width and height of the feature map, $\alpha_{k}^{c}$ denotes the importance of feature map $Z^{k}$ for class $c$, and $Z_{ij}^{k}$ is the data of feature map $Z$ at the coordinate $(i,j)$ in channel $k$.

Next, we can use weighed sum of forward activation maps and the ReLU activation function to obtain the localization map Grad-CAM $L_{Grad-CAM}^{c}$ as follows.

Note that, ReLU is used to remain features that have a positive impact on class $c$.

Federated learning is an effective privacy-preserving technique that collaboratively trains a global model without sharing data with each client. The detailed implementation of FL can be summarized as follows. First, a central server sends a global model with weight initialization $w_{0}$ to local clients. Then $K$ selected clients will train the model with local data, and then each of them, e.g., client $k$, will upload the model parameter $w_{k}^{t}$ to the central server at the $t$-th round of communication. On server side, it will update $w_{t}$ by aggregating the weights from selected clients, yielding

After that, the server will broadcast the global parameter to local clients for the next round of training. In local training, the weights of each client can be updated with

where $\eta$ is the learning rate and $\nabla F_{k}(w_{k}^{t+1})$ is the gradient of client $k$.

As a privacy-preserving method, differential privacy (DP) attempts to maximize the useful information retrieved from a database while minimizing the expenditure of privacy resources [4]. In recent years, DP has been widely used in federated learning to defend against gradient leakage attack. One of the most popular DP techniques is called ($\epsilon$, $\delta$)-differential privacy [11], given by

where $D_{i}$ and $D^{{}^{\prime}}_{i}$ are two neighboring datasets and only one single entry of data differs between them. $\epsilon$ is the privacy budget. The smaller $\epsilon$, the higher privacy protection. In addition, the parameter $\delta$ denotes the upper bound of the output probability that can be changed by adding or removing a record in the dataset, i.e., the privacy protection standard.

In the applications of federated learning, we often achieve differential privacy by adding a moderate amount of Gaussian noise and clipping the gradient.

Since the GradCAM approach mainly relies on the feature map of the last convolutional layer for model interpretability, for simplicity, we will focus on the last two layers of the DNN, including one convolutional layer and one fully connected layer. Specifically, let $H^{(l-2)}$ and $W^{(l-1)}$ denote the input of the convolutional layer and the convolution kernel, respectively. Then we add the independent identical distribution (IID) Gaussian noise, $n_{\epsilon}\sim N(0,\zeta^{2})$, to the convolution kernel $W^{(l-1)}$ and the last fully connected layer, yielding

After injecting noise, the last two layers of the DNN can be written as:

Note that for simplicity, we do not show the injected noise to the bias term in the derivation, but will add the noise to all the parameters in the DNN. Next, we still study how the noise impacts the weights during backpropagation.

Assume that the loss function is denoted as $L$ and the prediction (before softmax) for class $c$ as $Y_{c}$. To obtain the gradient of $Y_{c}$ with respect to the feature map in the $i$-th channel $\frac{\partial Y_{c}}{\partial Z_{i}^{(l-1)}}$, we need to compute the backpropagation in the following steps.

Based on the above derivation, we can obtain the following expression for $\frac{\partial Y_{c}}{\partial Z_{i}^{(l-1)}}$:

The above equation is composed of two terms: the first term is the derivative of the last fully connected layer and the second term is the derivative of convolutional layer. Next, we briefly analyze the influence of noise on the feature map using gradient based interpretability, as introduced in Section 2.1. We can see that if we only add noise to the fully connected layer, it will affect the feature map used for model interpretability. While if we only inject noise to the convolutional layer, it will change the output of activation function, thereby changing the feature map.

The above simple analysis suggests that the regular DP will affect the gradient-based interpretability in federated learning. It motivates us to develop a new differential privacy method to improve the model interpretability in the following section.

We first briefly present the core idea of DP-based Federated Learning. As described in Section 2.2, federated learning collaboratively trains a global model in a central server by aggregating the weights or gradients from selected clients and then sends them back to the clients after aggregation. Different from the regular FL, DP-based federated learning attempts to add Gaussian noise into the weights before aggregation. Then the weights with noisy perturbation in local clients will be uploaded to the central server. As discussed above, injecting noise will negatively affect the gradient-based interpretability. As a result, the lack of model interpretability hinders the widespread deployment of FL in safety-critical applications, such as healthcare and autonomous driving.

To deal with the challenge above, we will explore how to improve model interpretability while preserving privacy in the context of federated learning. As we know, gradient-based interpretability, such as Grad-CAM, will target the salient regions (high weights) in the feature map to interpret the important pixel-level features. The key insight is that when the noise with negative value is injected to the large weights, it will counteract the neuron important weights, leading to poor interpretability of FL. Conversely, if we add large positive noise to less important weights, the insignificant regions become salient ones, misleading the interpretability of prediction results. Inspired by this, we propose to design a simple yet effective adaptive differential privacy (ADP) mechanism that selectively injects noise into model parameters. The main idea is that we only inject noise into the less important weights, such that their values should not be over the neuron important weights. To achieve this, we need to sort the weights in a descending order, and then add noise to the bottom $r$% weights, $\widetilde{W}_{r}$. If $\widetilde{W}_{r}$ exceeds the smallest value of the top $r$% weights, it should be clipped.

We apply the proposed ADP to improve the interpretability of DNN in federated learning. In this work, we mainly focus on improving the trade-off between privacy and interpretability of image classification using ResNet [14] consisting of multiple convolutional layers. In general, the shallow layers of ResNet contain high-resolution features but with low-level semantic information. The deep layers extract some more complex and abstract features with high-level semantics, but at the same time the feature resolution decreases. Considering both “shallow local features” and “deep global features”, we choose to selectively add noise to the weights of the last convolutional layer of each block of ResNet (8 parameters for ResNet18). We will conduct experiments to validate the good performance of the proposed strategy.

The proposed ADP mechanism is summarized in Algorithm 1. The names of the selected convolutional layers and the parameter of each layer are stored in “target layers” and “target parameters”, respectively. Line 2 shows the weights of feature maps of each convolutional layer obtained by GradCAM. Line 4 sorts the weights of each target layer and Lines 14-21 selectively add noise to the less important weights for interpretability. In order to easily add noise to the weights, we create a mask matrix that can generate binary value (0 and 1) by setting a threshold to judge whether perturbing weights or not. This mask matrix can help adaptively add noise to the model parameters in “target parameters”. Note that, the proposed mechanism is able to flexibly choose the ratio of injected noise to the weights in each target layer. It is simple, easy to use, yet effective to achieve a good trade-off between interpretability and privacy.

We evaluate the proposed ADP on three benchmark datasets, MNIST [2], Blood [3], and Animals [1], as described below.

MNIST. This dataset contains a total of 70,000 grayscale images of size 28×28, divided into a training set of 60,000 images (and labels) and a test set of 10,000 images (and labels).

Blood. This dataset contains 8-class categories of microscopic peripheral blood cell images. It contains a total of $17092$ RGB images of individual blood cells, which are split into 13600 and 3492 images for training and testing. In our experiment, we resize the images from the original 3 × 360 × 363 pixels to 3 × 64 × 64 pixels.

Animals. This dataset contains 10 animal species, commonly used in the field of image classification. Then we split the total 26,000 images into 20,800 and 5,200 for training and testing. To keep the same image size, we uniformly resize them to 3 × 64 × 64 pixels.

We use ResNet18 with 17 convolutional layers and 1 fully connected layer for image classification, and choose SGD as the optimizer with a learning rate of 0.001. We set a total of 100 clients for federated training, with 0.1-fraction clients chosen in each communication round. We set the number of communication rounds to 1000, and the local training batch to 100. At each round, we set the epoch to 1 for MNIST and Blood, and 5 for Animals. We implement experiments on both IID and Non-IID scenarios.

In the setting of differential privacy, we follow the common setting of $\delta$ to $10^{-5}$, test the effect of $\epsilon$ on accuracy by setting $\epsilon$ to 100, 10, 5, 1, 0.1, respectively and the effect of gradient clipping parameter $c$ on accuracy by setting $c$ to 5, 10, 20, 40, 80, respectively. Note that the parameter $\epsilon$ is the total privacy budget, while each client’s budget in each communication round is the total privacy budget divided by the product of the fraction and the number of communication rounds.

Since we are the first to study the trade-off between privacy and gradient-based interpretability, it is hard to find appropriate benchmark baselines in the prior works. Nevertheless, we compare the proposed ADP method with the following baselines

No DP. “No DP” means that no differential privacy is applied to the federated averaging model, which will serve as an essential baseline to provide a comparison for DP-based models.

DP. “DP” stands for applying differential privacy on the basis of No DP, which adds randomly 100% of noise into the model parameters.

Random DP. For a fair comparison, we choose a baseline called “Random DP”, which will add the same ratio of noise to the weights as the proposed ADP method. The only difference between this method and ADP is that it randomly adds noise to the weights while ADP selectively adds noises to the less important weights.

Before studying the impact of ADP on model interpretability, we first would like to check whether the proposed method can improve accuracy. In this experiment, we choose to inject 25%, 50%, and 75% noise to model parameters of Random DP and ADP. In addition, we observe that accuracy decreases as the clipping parameter $c$ rises. In order to not affect the accuracy too much, we set parameter $c$ to 5. Regarding privacy budget $\epsilon$, we set it to $1$, $5$, and $5$ for MNIST and the other two datasets by considering the trade-off between accuracy and privacy protection. We compare our methods to the baselines under the IID and Non-IID scenarios.

Tables 1 and 2 illustrate the comparison results of different methods under different ratio of injected noise to model parameters. First of all, it can be seen that as the amount of injected noise increases from 25% to 75%, the classification accuracy drops. This validates that there exists a trade-off between accuracy and privacy. Besides, we can observe that our ADP has a higher accuracy than the DP and Random DP on Blood dataset while achieving comparable accuracy to the other datasets. It suggests that the proposed ADP can improve the classification accuracy to some extent under privacy protection.

Next, we explore the influence of our ADP on the interpretability of classification results using three benchmark datasets. Figures 2 and 3 illustrate the visual explanation of different methods using heatmap. We first take a look at the visualizations on MNIST. We can observe that DP no longer localizes the critical parts of handwritten numbers, leading to poor interpretability. In contrast, our ADP still has the ability to localize the most identifiable parts of each number, indicating that it can achieve higher interpretability and improve the recognition ability. In addition, the localizations of random DP are not as good as our method. We will further compare and explain their visualizations on the Blood dataset.

As shown in Figure 2(b), we can observe from the heatmap that the proposed ADP can better localize the salient regions of feature map compared to Random DP and No DP under IID data. It means that our method helps localize the lesions more precisely in medical examinations, thereby facilitating the understanding of the predicted results by doctors and patients. It thus can promote the deployment of FL in safety-critical applications.

Next, we compare the visual interpretation of our method and the baselines on more complex Animals dataset. We can see from Figure 2 (c) that our proposed method can capture the whole facial features and body features of animals, which is in line with human visual understanding. In contrast, No DP can hardly localize facial features and body features of animals while Random DP can only capture part of farcical features. We can conclude that our method can better explain the classification results on complex data.

What is more, we illustrate the visualizations of heatmap for different methods under the Non-IID scenario, as illustrated in Figure 3. Compared to IID, all the methods will reduce their ability to localize the salient regions of feature map. Nevertheless, we can still observe from Figure 3 that our method can better interpret the prediction results.

The above extensive experiments illustrate that our proposed method can not only enhance interpretability but also improve classification accuracy. Therefore, the proposed ADP can help improve the trade-off between privacy, accuracy, and interpretability.

Finally, we also show the robustness of the proposed ADP to gradient leakage attack. We adopt the well-known deep leakage from gradients (DLG) algorithm [35] to attack our method. We want to verify whether the DLG method can reconstruct the input data based on the shared weights from local clients. Figures 4 and 5 show the reconstruction of input data by the attacker under different training iterations. We can observe that DLG fails to recover the input data from the share weights under ADP. The main reason is that DLG is not very effective when the noise is large as mentioned in [35]. In the proposed ADP, we add sufficient Gaussian noise with a variance ranging from $10^{-4}$ to $10^{-3}$ to the model parameters, which matches the range of noise scale to defend against gradient leakage attack as discussed in the prior work. Based on these experimental results, we can conclude that the proposed ADP is robust to the gradient leakage attack, thus is able to protect private data in federated learning.

In order to protect the sensitive information of users, researchers adopt different privacy-protecting methods to prevent gradient leakage attacks (GLK). These approaches can be categorized into two main groups: secure multiparty computing (SMC) [17, 18, 6] and differential privacy (DP) [11]. SMC mainly adopts cryptographic techniques to protect the private information of input, making attackers hard to see the model updates. For instance, Kanagavelu et al. [17] proposed a two-phase SMC method to protect data privacy in federated learning. However, SMC is computational expensive, limiting its application to energy-aware IoT devices. DP aims to inject noise to the model parameters to defend against GLK. Wei et al. [32] proposed a privacy-protecting federated learning framework that adds noisy perturbation to the gradients of local clients before sending them to the server. While DP is very simple yet effective method for privacy protection, it will lead to an accuracy drop of federated learning [8].

Some studies [13, 7] try to improve the trade-off between privacy and accuracy. For example, Luo et al. [22] combined transfer learning and sparse network finetuning to improve the privacy-utility trade-off in federated learning. Hu et al. [15] developed Fed-SMP, a new differentially private FL framework for ensuring the client-level DP while retaining the model accuracy. In addition, some researchers [16, 5] integrated secure aggregation and distributed DP to improve privacy-accuracy trade-off. These existing works can achieve remarkable performance on the trade-offs between privacy and model accuracy, they do not take into account the interpretability. In real-world applications, such as healthcare, it is crucial to explain and understand the diagnosis results of disease using federated learning [24].

Some recent works focus on the trade-off between accuracy and interpretability in classical machine learning (not federated learning) [27, 26, 24]. This is because trustworthy machine learning system needs to ensure both the privacy protection and interpretability. Harder et al. [13] attempted to produce accurate prediction while interpreting the classification results using several locally linear maps (LLM) per class. However, the proposed approach tries to explain the results based on input features using some simple datasets. Patel et al. [25] designed a privacy-preserving framework to study the minimum privacy budget required for feature-based model explanations. In summary, existing works focus on the trade-off between privacy and feature-based interpretation rather than gradient-based interpretation in regular machine learning. None of existing research works has explored the trade-off between privacy preservation and gradient-based interpretation in federated learning. To our best knowledge, the proposed adaptive differential privacy mechanism in this paper is the first work to fill the gap between them in federated learning.