EasyFL: A Low-code Federated Learning Platform For Dummies

The life cycle of FL comprises two phases: the experimental phase and the production phase. Fig. 1 presents key activities in both phases. Researchers develop novel algorithms to address specific problems in simulation-based settings in the experimental phase. After deployment, they evaluate the algorithms under real-world scenarios in the production phase. We propose this FL life cycle based on in-depth studies of research and industrial practices of FL [19, 5] and traditional machine learning engineering for production (MLOps) [27, 28, 29, 30]. However, different from MLOps that deploys models to perform inference for new requests, FL trains new models with real-world decentralized data in the production phase.

In the experimentation phase, researchers solve problems by iterating these steps: (1) Identify the problems; (2) Setup the simulation environments; (3) Adapt existing algorithms and develop new algorithms; (4) Conduct experiments with different combinations of hyperparameters; (5) Analyze results.

The key step to transit from the experimental phase to the production phase is deployment. The production phase contains the following steps: (1) Collect and transform data in clients; (2) Adapt datasets to FL systems; (3) Train and evaluate the algorithm developed in the experimentation phase; (4) Analyze results; (5) Publish the model for serving.

In both phases, if the training results are not promising, researchers usually consider two ways to improve performance: (1) Starting from the simulation setup again to ensure that simulated scenarios match real-world scenarios; (2) Iterating algorithms if the simulation setup meets the requirement.

Based on the FL life cycle, we present the system requirements of a low-code FL platform by analyzing the challenges of the critical activities in the life cycle.

Ease of Experimentation and Development Fundamentally, an FL platform must allow users with different levels of expertise to easily run FL experiments, develop new algorithms, and implement new applications. To achieve this objective, the platform must resolve the following challenges. Firstly, statistical and system heterogeneity are the most common challenges: simulation of statistical heterogeneity requires methods to partition datasets to become non-IID and unbalanced; simulation of system heterogeneity (tightly related to hardware) currently is complex and resource-demanding.

Secondly, most of the recently innovated federated algorithms, such as FedProx [23] and FedReID [13], change only a few aspects of the existing algorithms (more examples are provided in Appendix C). Replicating the whole codebase for FL training would be a waste of time and effort.

Thirdly, FL metrics collection is not trivial because it needs to support: (1) diverse training methods; (2) metrics in three levels of hierarchy: a training task comprises metrics of rounds where a round contains metrics of clients. Existing logging methods like TensorBoard [17] cannot fulfill these requirements.

To address these challenges, EasyFL provides out-of-the-box heterogeneity simulation to simulate FL challenges with the least hassles, abstracts the training flow to enable minimum coding for new algorithms and applications, and supports comprehensive tracking for in-depth analysis (Section V).

Diverse Training Methods An FL platform must support three training methods: standalone, distributed, and remote training. In the experimental phase, standalone training simulates FL on a hardware device (CPU or GPU), which is the simplest way to start FL experiments. Distributed training increases training speed by training on multiple hardware devices, using communication backend provided by frameworks like TensorFlow [17]. Existing distributed training allocates one client to a GPU [23]. Further optimization is needed because 1) available hardware resources are normally less than the number of simulated clients (tens or hundreds); 2) simulations of unbalanced data or system heterogeneity lead to stragglers, resulting in resource underutilization when fast clients wait for the stragglers.

Moreover, in the production phase, remote training is needed to support remote message transmission as the server and clients are deployed to different locations. An FL platform must integrate remote communication into the training flow, without affecting standalone and distributed training.

EasyFL supports these three training methods by decoupling the communication channels and the training flow. It provides standalone training by default, optimizes distributed training (Section VI), and supports remote training by transmitting messages through remote communication (Section VII).

Seamless and Scalable Deployment Deployment, the last step to run FL in production, is also challenging because of diverse computing environments and the potentially large number of clients. Firstly, the production environment of FL clients could have large variances on the hardware and operating systems [20]. Manually adapting and deploying is time-consuming and unscalable. Moreover, new FL algorithm development usually needs multiple iterations of deployment because the algorithm verified in the experimental environment may not work perfectly well in the production environment. Secondly, to scale up the number of clients in training in production, the server must be aware of the addresses of clients. Using static configuration is a straightforward approach to manually record addresses of clients on the server. However, this method is unstable and unscalable when the number of clients is large because it is common that existing clients would drop out and new clients would join [31].

To tackle these challenges, EasyFL supports seamless and scalable deployment with containerization and service discovery (Section VII).

EasyFL’s architecture comprises of an interface layer and a modularized system layer. The interface layer provides simple APIs for high-level applications and the system layer has complex implementations to accelerate training and shorten deployment time with out-of-the-box functionalities.

Interface Layer The interface layer, providing a common interface across FL applications, is one of the key designs that enable low-code capability. It contains APIs that are designed to encapsulate complex system implementations from users. These APIs decouple application-specific models, datasets, and algorithms such that EasyFL is generic to support a wide range of applications like computer vision and healthcare.

System Layer The system layer supports and manages the FL life cycle. It consists of eight modules to support FL training pipeline and life cycle: 1) The simulation manager initializes the experimental environment with heterogeneous simulations. 2) The data manager loads training and testing datasets, and the model manager loads the model. 3) A server and the clients start training and testing with FL algorithms such as FedAvg [21]. 4) The distribution manager optimizes the training speed of distributed training. 5) The tracking manager collects the evaluation metrics and provides methods to query training results. 6) The deployment manager seamlessly deploys FL and scales FL applications in production.

The system layer embraces coarse-grained plugin architecture with modular design and fine-grained plugin design with training flow abstraction: Modularity provides users with the flexibility to extend to new datasets and models with application-specific plugins; Training flow abstraction allows users to customize federated algorithms by replacing specific training stages in the server and client modules.

Table II summarizes the most important APIs of EasyFL. We introduce below these APIs in three categories (initialization, registration, and execution), and illustrate their usage scenarios.

init(configs): Initialize EasyFL with provided configurations (configs) or default configurations if not specified. It instructs the simulation manager to coordinate with the data manager to set up the simulation environment for experiments. Besides, these configurations determine the training hardware and hyperparameters for both experimental and production scenarios.

register_<module>: Register customized modules to the system. EasyFL supports the registration of customized datasets, models, server, and client, replacing the default modules in FL training. In the experimental phase, users can register newly developed algorithms to understand their performance. In the production phase, users can use it to adapt to real-world datasets.

run, start_<server/client>: The APIs are commands to trigger execution. run(callback) starts FL using standalone training or distributed training, with an optional callback to define execution after training is completed. start_server and start_client start the server and client services to communicate remotely with args variables for configurations specific to remote training, such as the endpoint addresses.

These APIs empower users with various levels of expertise to conduct Fl experiments or build FL prototypes with minimal coding, as shown in Listing 1. For example, beginners can quickly start FL with no more than three lines of code (Example 1); To build prototypes, experienced researchers can customize federated algorithms in the server and client modules by registering new server and client, and start the services to conduct remote training (Example 2). We further explain how to achieve low-code implementations of the server and client in Section V-B.

To minimize researchers’ hassles in setting up the simulation environment to start experiments, EasyFL provides various methods to simulate statistical and system heterogeneity using the simulation manager and the data manager. Users can easily change configurations in initialization to customize simulation methods.

Statistical Heterogeneity To simulate statistical heterogeneity, which is closely related with datasets, EasyFL includes three most commonly used datasets in FL research: FEMNIST [1, 32], Shakespeare [33, 21], and CIFAR-10 [34], the statistics of which are shown in Table III. FEMNIST and Shakespeare datasets simulate the non-IID and unbalanced data with realistic scenarios as described in [1]. CIFAR-10 dataset can be flexibly constructed to different numbers of clients with unbalanced data simulation and two different types of non-IID simulation: (1) dividing the dataset by Dirichlet process $Dir(\alpha)$ [35]; (2) dividing the dataset by class, each client containing $N$ out of 10 classes [22]. EasyFL provides flexibility for researchers to simulate various numbers of clients, applying non-IID data simulation and unbalanced data simulation, separately or in combination.

Besides these out-of-the-box datasets, researchers can also employ register_dataset API to easily integrate new datasets. For datasets that naturally simulate statistical heterogeneity, such as datasets provided in LEAF benchmark [1], EasyFL supports researchers to start training once the datasets are adapted into the system. For datasets that are not simulated yet, such as ImageNet dataset [36], EasyFL provides functionalities to partition the datasets for non-IID and unbalanced data simulations. These three datasets provide a good starting point, we will keep extending to provide more datasets for simulation.

System Heterogeneity Although system heterogeneity is associated with hardware resources, EasyFL simulates it in a lightweight and realistic manner. System heterogeneity is caused by varieties of hardware and network connectivity of clients, resulting in varied training times among clients and network transmission times between the server and clients. As a result, the server receives model updates from clients at different times — the slower ones are the stragglers. A significant system heterogeneity simulation should reveal these stragglers with varied time between server model distribution and client model uploads.

We simulate the time differences using training speed variances of mobile devices from AI-Benchmark [37]. We first calculate the training speed ratio of different mobile devices. Then we assign each client a type of mobile device based on speed ratio. In each round of training, clients not only execute the training but also wait for the time proportional to their speed ratios before uploading updates to the central server. Besides, EasyFL can simulate networking conditions (e.g., latency) for communication with an isolated environment provided by containerization (Section VII). As such, EasyFL simulates the system heterogeneity with stragglers.

EasyFL supports fine-grained plugin design by abstracting the FL training flow in the server and client modules to multiple stages as shown in Fig. 3. It provides researchers with the flexibility to customize any of these stages when developing new FL algorithms.

Each round of training or testing begins from the selection stage, which employs strategies to select a fraction of participated clients in the server. The compression stage implements data compression algorithms to reduce communication costs and the decompression stage decompresses the data. The distribution stage transmits data from the server to the clients. Clients download the data in the download stage and execute training or testing after decompression. The encryption stage encrypts the updates before clients upload them to the server in the upload stage. At the end of each round, the server aggregates these updates in the aggregation stage.

We propose training flow abstraction based on analysis of FL open problems [19] and emerging solutions in top publications, as shown in Table VII. We analyzed 33 papers from recent top publications: 10 out of 33 ($\sim$30%) publications innovate new algorithms by changing only one stage of the training flow; The majority ($\sim$57%) change only two stages of the training flow. However, implementations of these studies mostly need to write the whole FL training process. In contrast, training flow abstraction allows researchers to focus on innovating algorithms, without re-implementing the whole training process. The stages in both the server and client modules serve as templates for the standard FL training process. Users can inherit them and replace one or more stages to implement new algorithms. For example, researchers who focus on improving communication efficiency can develop new compression algorithms to replace the compression related stages. We integrate a compression algorithm [38] as an example with around 80 lines of code, whereas the released implementation requires several hundred lines of code.

Moreover, EasyFL encourages users to publish their new federated algorithms as plugins to the community. Others can easily use these plugins to reproduce results and further develop on them with minimal coding. We believe that EasyFL has a great potential to grow and become an open-source community to facilitate the reproducibility of FL research.

To support comprehensive and easy analysis of FL training results, EasyFL provides a powerful tracking manager. The tracking manager is specially designed to collect and store three levels of FL metrics: training task metrics, round metrics of a task, and client metrics of a round, equipping researchers with in-depth analysis with comprehensive details. Task metrics include information about the whole training such as configurations and hyperparameters. Round metrics record the metrics in the server for each round of training and testing, such as accuracy, communication cost, and training time. Client metrics contain training and testing metrics of selected clients.

EasyFL supports two forms of tracking for diverse training methods: (1) Local tracking logs metrics to local storage, which is efficient for standalone and distributed training; (2) Remote tracking starts a tracking service to collect metrics via API calls, required by remote training.

The tracking manager provides command-line tools to query the metrics. It also exposes simple APIs for gPRC calls and HTTP requests, which are extensible to build a visualization dashboard and real-time performance monitoring.

We implement EasyFL as a standalone Python library of $\approx$ 6000 lines of code (LOC). EasyFL uses PyTorch [44] as the backend for deep learning model training and testing. Remote communication in EasyFL is based on gRPC ¹¹1https://grpc.io/, a high-performance RPC framework. We also provide several other RPC methods to start and stop training. Protocol Buffers ²²2https://developers.google.com/protocol-buffers, a lightweight method for serializing structured data, are used to serialize remote messages.

Targeting two different types of deployment, EasyFL provides different stacks for service discovery. For deployment using Kubernetes [43], EasyFL utilizes Pod, the smallest deployable unit in Kubernetes where containers run on, as the registor. The Service in Kubernetes, which connects client Pods and an internal DNS, serves as the registry. For deployment using only Docker [42] containers, EasyFL uses etcd ³³3https://etcd.io/ as registry because it is a reliable and consistent key-value store for distributed systems. It provides docker image for docker-gen ⁴⁴4https://github.com/jwilder/docker-gen, which serves as registor to get the container metadata including IP addresses.

Datasets and Models We use datasets and models provided by EasyFL shown in Table III. We evaluate the performance under IID and non-IID partition of these datasets.

Algorithm and Hyperparameters EasyFL provides FedAvg [21] as the standard algorithm. By default, we set local epoch to be 10 ($E=10$) and batch size to be 64 ($B=64$) for all the experiments. We use SGD as the optimizer and tune the learning rate for each dataset. For different experiments, we select different numbers of clients to participate in the training. More experiment settings are provided in Appendix B-A.

Evaluation Metrics We evaluate EasyFL both qualitatively and quantitatively. On the one hand, we provide qualitative results of lines of code (LOC) to implement FL applications. On the other hand, we use the evaluation metrics provided by the tracking manager for quantitative results: model accuracy, total training time, processing time each round (round time), and communication cost. To reduce the impact of hardware instability during training or testing, we calculate the round time $T_{round}$ by averaging end-to-end processing time $T_{total}$ of $R$ rounds, $T_{round}=\frac{T_{total}}{R}$.

Experiment Environment All simulation experiments are run on one NVIDIA^® 2080Ti GPU or 64 NVIDIA^® V100 GPUs (8 GPUs per node) with CUDA 10.1 and CuDNN 7.6. We deploy and evaluate EasyFL in production on a Kubernetes cluster with three nodes, where each node has 28 Intel(R) Core(TM) i9-9940X CPUs.

To show the effectiveness of heterogeneity simulation, we present benchmark results on the impact of simulated statistical heterogeneity and system heterogeneity.

Impact of Statistical Heterogeneity on Performance We compare the accuracy of models trained on IID and non-IID settings. For FEMNIST and Shakespeare, we simulate non-IID with realistic partition. For CIFAR-10, we simulate three levels of non-IID with increasing heterogeneity: Dirichlet process $Dir(0.5)$ (10 classes per client), three classes per client, and two classes per client. These experiments are run with 10 selected clients per round.

Table IV shows that the simulated statistical heterogeneity leads to performance degradation. Models trained on simulated non-IID data all perform worse than models trained on the IID data. In particular, the increasing degree of statistical heterogeneity simulated from Dir to two classes per client causes even larger degradation, leading to the largest accuracy gap of 21.25%. Researchers can use these results as a benchmark when they start using EasyFL and have the flexibility to customize the degree of non-IID.

Impact of Heterogeneity on Training Time We simulate unbalanced data by $Dir(0.5)$ and system heterogeneity using the EasyFL simulation manager. Fig. 6 shows that unbalanced data, system heterogeneity, and their combination cause a huge discrepancy in training time of each round among clients in the CIFAR-10 dataset. The fastest client is four times faster than the slowest client because of unbalanced data as shown in Fig. 6. This gap is larger under system heterogeneity in Fig. 6 and their combination in Fig. 6. FL training normally requires hundreds of rounds, further amplifying the impact of the stragglers. We provide the results of training time per round of Shakespeare and CIFAR-10 datasets in Appendix B-B.

In this section, we demonstrate the effectiveness of our proposed GreedyAda and the scalability of distributed training in EasyFL.

Performance of GreedyAda EasyFL accelerates training with GreedyAda (Greedy Allocation with Adaptive Profiling) under resource constraints of $M$ available GPUs and heterogeneous scenarios. The heterogeneous scenarios are simulated with the combination of unbalanced data and system heterogeneity described in Section VIII-C. We run the experiments with 20 selected clients per round using standalone training and distributed training with different client allocation strategies: (1) GreedyAda; (2) random allocation that randomly allocates around $\frac{20}{M}$ clients to a GPU; (3) slowest allocation that allocates around $\frac{20}{M}$ slowest clients to a GPU. Fig. 5 demonstrates that GreedyAda achieves the fastest training speed — up to 1.5x faster than random allocation and up to 2.2x faster than slowest allocation — in all datasets with different number of GPUs.

Scalability We further evaluate the scalability of distributed training with different numbers of GPUs $\{8,16,24,32,64\}$ and varied amount of data $\{5\%,10\%,20\%,40\%,80\%,100\%\}$. Data amount means the percentage of samples used training in clients. We run these experiments with 100 selected clients each round under the IID setting for 100 rounds on FEMNIST and measure the processing time per round (round time).

Fig. 7 demonstrates that the round time is effectively reduced with increasing numbers of GPUs. While compared with 1.84x (optimal 2x) speedup from 8 GPUs to 16 GPUs, the speedup from 8 GPUs to 64 GPUs is only 4.96x (optimal 8x). The underutilization of hardware resources is mainly because the data amount is small (5%) for these experiments, causing the communication overhead among GPUs to outweigh the training time when the number of GPUs is large. We further investigate it by increasing the data amount.

Fig. 7 illustrates that the round time increases as data amount increases, either using 32 or 64 GPUs. However, the increase in round time is much smaller than the increase in data amount. In particular, the data amount increases by 20x from 5% to 100%, but the round time only increases less than 4x. These results suggest that when the data amount is small, the smaller number of GPUs can handle training effectively; when the data amount is large, EasyFL can scale up to the large number of GPUs for training. Besides, we also present the evaluation accuracy of different data amounts in Fig. 7. The accuracy increased from $\sim$80% to $\sim$85% with data amount increased from 5% to 100%.

We evaluate remote training on time to deployment and performance of training in containers.

Deployment Time EasyFL enables sub-linear time to deployment by leveraging the power of containerization and container orchestration engine, Kubernetes. Without containerization, the time to deployment grows linearly as we increase the number of clients, where the deployment of one client would need hours because of diverse and complex computing environments. In contrast, EasyFL only needs a one-time setup of the Docker environment and a Kubernetes cluster. It builds docker images of FL components in seconds and deploys clients to each node in Kubernetes clusters in minutes, dramatically reducing the deployment time and effort.

Accuracy and Distribution Latency EasyFL supports remote training to achieve the same accuracy as experiments in Table IV under the same settings. Besides, Fig. 8 presents the distribution latency of the server when scaling up the number of clients using FEMNIST dataset. Although the distribution latency increases almost linearly using multi-threading, the latency is relatively low compared to the training time needed. To further optimize this distribution latency, we can consider replicating servers to load balance the requests.

EasyFL enables easy implementation of FL to a wide range of applications by allowing seamless migration from existing training codes. We demonstrate its capability by using it to implement three applications: (1) FedProx [23] is an optimization framework to address statistical and system heterogeneity; (2) STC [38] is a compression framework to reduce communication cost; (3) FedReID [13] implements FL to person re-identification, an important computer vision task for video surveillance. Table V compares lines of code (LOC) and average training time per round (round time)⁵⁵5The round times of FedProx and STC are evaluated on the MNIST dataset. of EasyFL implementation with the original implementations. EasyFL greatly reduces the efforts of researchers in writing codes (3.2x to 9.5x less coding) while maintaining and even using less round time. Fewer codes result in fewer bugs and faster iterations of development. Besides, the codes are similar to non-FL implementation that researchers are familiar with. We provide more details of LOC counting and implementation of STC [38] using EasyFL in Appendix A.

Despite that EasyFL reduces the LOC significantly with abstractions, we further investigate whether these abstractions would lead to extra training overhead by comparing with other FL libraries: LEAF [1] and TFF [4]. We conduct the experiments with 10 selected clients per round under IID simulation for $R=100$ rounds on FEMNIST and CIFAR-10 datasets, and $R=20$ rounds on the Shakespeare dataset.

Table VI compares the round time and GPU usage of EasyFL, LEAF, and TFF. We standardize the experimental environment and the versions of PyTorch and TensorFlow to conduct these experiments⁶⁶6These experiments are run with CUDA10.1 and CuDNN 7.6. EasyFL is run on PyTorch 1.7.0; LEAF and TFF are run on TensorFlow 2.3.2. We upgrade LEAF from TensorFlow 1.3 and use TFF v0.17.0 (latest v0.19.0) to be compatible with CUDA and CuDNN versions.. To facilitate comparison across two deep learning frameworks (TensorFlow and PyTorch), we refer to the baseline in PyTorch paper [45] for back-of-the-envelope comparison: PyTorch is 1.06 (using ResNet-50) to 2.14 times faster than TensorFlow on computer vision (CV) tasks and 1.61 times faster than TensorFlow on the natural language processing (NLP) task. EasyFL achieves similar speedups on CIFAR-10 dataset using ResNet-18, 1.38 to 2 times of speedups on FEMNIST dataset (simple CV task), and 4.67 to 32.86⁷⁷7TFF is slow on the Shakespeare dataset because the LSTM model is unable to use CuDNN kernel due to the FL computation. times of speedups on Shakespear dataset (NLP task). These results show that EasyFL enables users to write less code without imposing extra system overhead. It also has better GPU utilization on FEMNIST and Shakespear datasets. Compared with TFF that tends to use all available GPU memory, LEAF and EasyFL are much more efficient in memory usage.

We conduct a case study of EasyFL on developing new federated computer vision applications with FedReID [13]. FedReID implements FL to person re-identification (ReID), which is an important retrieval task in computer vision. Since FedReID is a new application that uses nine heterogeneous datasets to simulate nine clients, we adapt these datasets into EasyFL via register_dataset API. As the training and testing of ReID are different from image classification, we customize the train and test in clients and integrate it with register_client API. The codes for training and testing are almost the same as the ones used for normal person ReID training. With these two registrations, we then initialize EasyFL with init API with configurations like training rounds and learning rates. After these steps, we can start training with run API.

We further leverage distributed training optimization in EasyFL to accelerate FedReID training and achieve near-optimal training speed with 6 fewer GPUs, as shown in Fig. 9. We train FedReID with 9 clients containing unbalanced data. The client with the largest dataset is the bottleneck in training. Instead of training with 9 GPUs by allocating each client to one of them, EasyFL saves hardware resources by achieving similar training speeds with only 3 GPUs.

Moreover, we build a prototype of FedReID using EasyFL with containerization and seamless deployment to Kubernetes. It imitates the real-world scenario that a client is deployed to an edge device, training with data collected from cameras.