Efficient Client Contribution Evaluation
for Horizontal Federated Learning

In horizontal FL, the feature space is shared among all client datasets $\mathcal{D}_{i}=\{(\textbf{x}_{k}^{i},y_{k}^{i})\}_{k=1}^{m_{i}}$, $i=1,...,N$. The local model owned by each participant has the same structure $f$ as the global model. In a communication round $t$, first, each client optimizes its local model parameters $\bm{\theta}_{i}^{t}$ with $\mathcal{D}_{i}$ and sends the parameter gradients $\nabla\bm{\theta}_{i}^{t}$ (or $\bm{\theta}_{i}^{t}$, the two are equivalent in FL) to the central server. The central server then collects and aggregates $\bm{\theta}_{i}^{t}$’s to update the global model by

where $\alpha_{\bm{\theta}}$ is the predefined learning rate. Finally, the renewed model parameters $\bm{\theta}_{G}^{t+1}$ are broadcast to all participants. This is the workflow of the basic FederatedAverage (FedAvg) algorithm. In this process, the central server cannot access the raw data held by each client. Therefore, for clients, the value of data is reflected by the gradients they upload to central server. If the contribution of each gradient to the global model can be evaluated, it will indirectly reflect the client’s contribution.

To achieve this goal, a novel method is proposed, named Federated REINFORCE client contribution evaluation (F-RCCE). The F-RCCE is an integrated evaluation algorithm with the horizontal FL system (Fig.1). In the proposed framework, an evaluator module $g_{\bm{\phi}}$ is implemented on the central server. Before aggregating client updates, the central server runs the evaluator to obtain an estimate of the values of the gradients. The gradients are then selected or discarded as decided by the evaluator. Only the selected client gradients are aggregated to renew the global model. In this framework, the evaluator is trained alongside the global task model $f_{\bm{\theta}}$, using a held-out validation set $\mathcal{D}_{v}=\{(\textbf{x}_{k}^{v},y_{k}^{v})\}_{k=1}^{m_{v}}$ owned by the central server. Thus the evaluator is able to perform task-specific evaluation that serves designated data distribution.

Optimization of the evaluator defines a non-differentiable sequential decision-making task. It is difficult to be achieved by end-to-end gradient descent. This problem is modeled as a RL problem for decision-making [17, 18, 19]. Details on the proposed formulation is explained in Section 2.2.

In this paper, we assume a modest security environment for the FL system. That is, all participants of the system are honest-but-curious. Client datasets $\mathcal{D}_{i}$ are private to the respective participants. The validation set $\mathcal{D}_{v}$ is selected to the need of the task model owner and private to the central server. All communications to and from the central server should be encrypted. In the experiments the encryption / decryption is omitted because it is independent of the proposed methods.

In F-RCCE, the evaluator module is formulated as an RL agent that collects client updates $\bm{\theta}_{i}^{t}$, and outputs a selection probability $\bm{\omega}^{t}$, that is, $g_{\bm{\phi}}(\bm{\theta}^{t})=\bm{\omega^{t}}$. The evaluator then samples a selection vector $\textbf{S}^{t}=[s_{1}^{t},...,s_{n}^{t}]$, where $s_{i}^{t}\in\{0,1\}$ represents $\bm{\theta_{i}^{t}}$ being included or discarded in the model aggregation. $P(s_{i}^{t}=1)=\bm{\omega}_{i}^{t}$ is the probability of $\bm{\theta_{i}^{t}}$ being included. After model aggregation, a reward signal $r$ is calculated based on the global model’s performance on $\mathcal{D}_{v}$. In summary, the proposed RL problem is:

• •

  State space: The feasible set of $\bm{\theta}_{G}$.

• •

  Action space: $\textbf{S}^{t}$.

• •

  Reward function: $r(\textbf{S}^{t})$.

In this paper, the reward function is defined to be directly related with the global model’s performance on the validation set. Specifically,

where $L_{v}$ is the loss function of global model on validation set $\mathcal{D}_{v}$, and $\delta$ is a baseline calculated by moving average of previous loss $L_{v}$ with moving averaging window $T>0$.

As opposed to FedAvg, the global model parameters are updated as

To optimize the evaluator, the objective function of $g_{\bm{\phi}}$ is defined as follows:

where $\pi_{\bm{\phi}}(\bm{\theta}^{t},\cdot)$ is a stochastic, parameterized policy defined by $\bm{\phi}$. According to the policy gradient theorem [20], we can obtain the probability of $\textbf{S}^{t}$ by $p(\textbf{S}^{t}|\bm{\phi})=\prod_{i=1}^{n}[{\bm{\omega}_{i}^{t}}^{s_{i}^{t}}\cdot(1-{\bm{\omega}_{i}^{t}})^{1-s_{i}^{t}}]$. With the log-derivative trick, we have

where

Putting it all together, the following policy gradient can be given as

Then, the evaluator’s model parameters $\bm{\phi}$ can be optimized by gradient ascent method with learning rate $\alpha$:

The pseudo-code for the proposed F-RCCE algorithm is illustrated in Algorithm 1.

In this section, a Horizontal FL environment with a large number of clients ($\geq 50$) is simulated to test the proposed F-RCCE method on the SMS Spam dataset [21]. The SMS Spam data set contains 5572 samples, among which 5000 samples are divided into 50 groups according to Dirichlet distribution and distributed to each client, and the remaining 572 samples are stored as a validation set in the central server. The task model $f_{\bm{\theta}}$ classifies SMS as ham or spam. In data preprocessing, the data is tokenized and the text is converted to sequences using Keras with $max\_words=1000$ and $max\_len=150$. After that, data was standardized to have zero mean and one standard deviation using scikit-learn’s Standard Nomalizer. Labels of categorical data are encoded in one-hot embedding.

On the model selection, logistic regression classifier is used as the task model and a four-layer multi-layer-perceptron as evaluator model. Adam optimizer with learning rate $\alpha_{\bm{\phi}}=10^{-5}$ is used to optimize evaluator model and SGD optimizer with learning rate $\alpha_{\bm{\theta}}=0.1$ is used to optimize the local model which each client copied by global model.

To verify the authenticity of the reported contribution factor, the remove-and-retrain scheme is adopted which proposed by Yoon et al [16]. Specifically, given an estimate of contribution values, the lowest- or highest-valued datapoints are discarded. The task model are then re-trained with the remaining data. When the value estimation is correct, removing high-valued datapoints should lead to a decrease in the task model accuracy, vice versa.

In order to reduce the uncertainty caused by random initialization, each experiment is repeated five times. The experimental data is averaged as the presented results. The baseline task model is trained by conventional FedAvg, without any data value estimation. The convergence curves of the FedAvg baseline as well as the co-trained F-RCCE model are presented in Fig.2(a). The two curves are almost identical, demonstrating that the inclusion of the evaluator $\bm{\phi}$ does not have a noticeable impact on the federated model.

As aforementioned, LOO[11] cannot directly measure the value of gradients uploaded by clients for each communication round, a specific client is deleted in each re-training iteration to evaluate the contribution of individual client. Afterwards, part of the highest-/lowest-contribution clients are removed following removing rate in re-training stage.

For F-RCCE, the evaluator and the global task model are trained for 1,000 communication rounds. Then the task model is re-trained with the evaluator fixed. In the re-training process, the selection probability output by the evaluator is taken as the contribution value of the client update. At each round, 30% of the client gradients are removed with the highest (F-RCCE Removing Highest) or lowest (F-RCCE Removing Lowest) contribution values. The task model is updated with the remaining gradients. The validation accuracy of the task model is recorded for each round.

As shown in Fig.2 (a), removing the highest contribution gradients makes the model converge more slowly than the baseline. Removing the lowest contribution gradients, on the other hand, slightly improves the validation accuracy. In comparison, the difference between removing highest or lowest valued clients by the LOO method is not significant.

To verify the ability of F-RCCE to identify data abnormality in horizontal FL, a training set is generated with randomly corrupted data. Noises is added to 20% of the SMS samples by deleting 20% of the words in each text string. The subsequent data preprocessing is same as the previous experiment.

Experimental results are shown as Fig.2 (b). The LOO method again fail to distinguish the highest-/lowest-contribution clients. The F-RCCE method performs significantly better than LOO. It is more sensitive to the removal of high/low contribution gradients. After removing the gradients with highest contribution calculated by F-RCCE, the performance of the global model has an obvious decline. And the performance of the global model is gratifying after removing the gradients with lowest contribution calculated by F-RCCE.

In a business-to-client model, horizontal FL often has a large number of clients. The following experiment investigates the consistency and time complexity of F-RCCE verses increasing number of clients. Horizontal FL systems are simulated with 50 to 500 clients. Considering the convergence of the Evaluator requires a number of iterations, each global model in F-RCCE iterates for 1000 rounds. As a control, each global model iterates for 50 rounds under LOO. The running time of the two algorithms are recorded, as shown in Table 1. With the number of clients increasing from 100 to 500, the time cost of F-RCCE only increases by 6%. However, the time cost of LOO increased almost linearly. Therefore, F-RCCE is more applicable in in practical situations.

The validation accuracies with highest-/lowest-contribution gradients verses different numbers of clients are plotted on Fig.2 (c). It shows that, removing the highest-contribution gradients always degrades task model performance. Removing the lowest-contribution gradients can sometimes benefit the validation accuracy, but sometimes causes destructive consequences. This phenomenon is due to the fact that the datasets are randomly generated. Some of the removed gradients still have positive effects on the model, although its contribution ranks low compared to others. There is no obvious trend in the validation accuracy with the increased number of clients. Result support that F-RCCE performs consistently in both small and large numbers of clients.

A intuitive idea is proposed to measure the clients contribution using F-RCCE. That is, the selection probability calculated by F-RCCE in each iteration is taken as the contribution of the iteration, and the contributions in all iterations are summed and scaled as the client contribution. For clients, when their amount of clean data is large, the calculated gradient is relatively stable, and it is easier to eliminate the influence of outlier. So the amount of clean data is considered to be positively correlated with the client’s contribution. In the experiment, a trained evaluator is used to measure the clients contribution for 50 communication rounds. Clients are sorted by the number of samples they held, and the contributions are plot in Fig. 2 (d). With the increase of the number of samples, the contribution of client also showed an upward trend. Experimental result shows that F-RCCE has the ability to evaluate the contribution of clients.