Federated Deep AUC Maximization for Heterogeneous Data
with a Constant Communication Complexity

Federated learning (FL) is an emerging paradigm for large-scale learning to deal with data that are (geographically) distributed over multiple clients, e.g., mobile phones, organizations. An important feature of FL is that the data remains at its own clients, allowing for the preservation of data privacy. This feature makes FL attractive not only to internet companies such as Google and Apple but also to conventional industries that provide services to people such as hospitals and banks in the era of big data [rieke2020future, long2020federated]. Data in these industries is usually collected from people who are concerned about data leakage. But in order to provide better service to people, large-scale machine learning from diverse data sources is important for addressing model bias. For example, most patients in hospitals located in urban areas could have dramatic differences in demographic data, lifestyles, and diseases from patients who are from rural areas. Machine learning models (in particular, deep neural networks) trained based on patients’ data from one hospital could dramatically bias towards its major population, which could bring about serious ethical concerns [pooch2020can].

One of the fundamental issues that could cause model bias is data imbalance, where the number of samples from different classes are skewed. Although FL provides an effective framework for leveraging multiple data sources, most existing FL methods still lack the capability to tackle the model bias caused by data imbalance. The reason is that most existing FL methods are developed for minimizing the conventional objective function, e.g., the average of a standard loss function on all data, which are not amenable to optimizing more suitable measures such as area under the ROC curve (AUC) for imbalanced data. It has been recently shown that directly maximizing AUC for deep learning can lead to great improvements on real-world difficult classification tasks [robustdeepAUC]. For example, robustdeepAUC reported the best performance by DAM on the Stanford CheXpert Competition for interpreting chest X-ray images like radiologists [chexpert19].

However, the research on FDAM is still limited. To the best of our knowledge, dist_auc_guo is the only work that was dedicated to FDAM by solving the non-convex strongly-concave min-max problem in a distributed manner. Their algorithm (CODA) is similar to the standard FedAvg method [DBLP:conf/aistats/McMahanMRHA17] except that the periodic averaging is applied both to the primal and the dual variables. Nevertheless, their results on FDAM are not comprehensive. By a deep investigation of their algorithms and analysis, we found that (i) although their FL algorithm CODA was shown to be better than the naive parallel algorithm (NPA) with a small mini-batch for DAM, the NPA using a larger mini-batch at local machines can enjoy a smaller communication complexity than CODA; (ii) the communication complexity of CODA for homogeneous data becomes better than that was established for the heterogeneous data, but is still worse than that of NPA with a large mini-batch at local clients. These shortcomings of CODA for FDAM motivate us to develop better federated averaging algorithms and analysis with a better communication complexity without sacrificing the sample complexity.

This paper aims to provide more comprehensive results for FDAM, with a focus on improving the communication complexity of CODA for heterogeneous data. In particular, our contributions are summarized below:

• •

  First, we provide a stronger baseline with a simpler algorithm than CODA named CODA+, and establish its complexity in both homogeneous and heterogeneous data settings. Although CODA+ has a slight change from CODA, its analysis is much more involved than that of CODA, which is based on the duality gap analysis instead of the primal objective gap analysis.

• •

  Second, we propose a new variant of CODA+ named CODASCA with a much improved communication complexity than CODA+. The key thrust is to incorporate the idea of stochastic controlled averaging of SCAFFOLD [karimireddy2019scaffold] into the framework of CODA+ for correcting the client-drift for both local primal updates and local dual updates. A striking result of CODASCA under a PL condition for deep learning is that its communication complexity is independent of the number of machines and the targeted accuracy level, which is even better than CODA+ in the homogeneous data setting. The analysis of CODASCA is also non-trivial that combines the duality gap analysis of CODA+ for a non-convex strongly-concave min-max problem and the variance reduction analysis of SCAFFOLD. The comparison between CODASCA and CODA+ and the NPA for FDAM is shown in Table 1.

• •

  Third, we conduct experiments on benchmark datasets to verify our theory by showing CODASCA can enjoy a larger communication window size than CODA+ without sacrificing the performance. Moreover, we conduct empirical studies on medical chest X-ray images from different hospitals by showing that the performance of CODASCA using data from multiple organizations can improve the performance on testing data from a single hospital.

We consider federated learning of deep neural networks by maximizing the AUC score. The setting is the same to that was considered as in [dist_auc_guo]. Below, we present some preliminaries and notations, which are mostly the same as in [dist_auc_guo]. In this paper, we consider the following min-max formulation for distributed AUC maximization problem:

where $K$ is the total number of machines, $f_{k}(\mathbf{w},a,b,\alpha)$ is defined below.

where $\mathbf{z}^{k}=(\mathbf{x}^{k},y^{k})\sim\mathbb{P}_{k}$, $\mathbb{P}_{k}$ is the data distribution on machine $k$, $p$ is the ratio of positive data. When $\mathbb{\phi}_{k}=\mathbb{\phi}_{l},\forall k\neq l$, this is referred to as the homogeneous data setting; otherwise heterogeneous data setting.

Notations. We define the following notations:

In the remaining of this paper, we will consider the following distributed min-max problem that can cover the distributed DAM,

where

Assumptions. Similar to [dist_auc_guo], we make the following assumptions throughout this paper.

To quantify the drifts between different clients, we introduce the following assumption.

Remark. $D$ quantifies the drift between the local objectives on clients and the global objective. $D=0$ denotes the homogeneous data setting that all the local objectives are identical. $D>0$ corresponds to the heterogeneous data setting.

In this section, we present a stronger baseline than CODA [dist_auc_guo]. The motivation is that (i) the CODA algorithm uses a step to compute the dual variable from the primal variable by using sampled data from all clients; but we find this step is unnecessary by an improved analysis; (ii) the complexity of CODA for homogeneous data is not given in its original paper. Hence, CODA+ is a simplified version of CODA but with much refined analysis.

We present the steps of CODA+ in Algorithm 1. It is similar to CODA that uses stagewise updates. In $s$-th stage, a strongly convex strongly concave subproblem is constructed as following:

where $\mathbf{v}_{0}^{s}$ is the output of the previous stage.

CODA+ improves upon CODA in two folds. First, CODA+ algorithm is more concise since the output primal and dual variables of each stage can be directly used as input for the next stage, while CODA needs an extra large batch of data after each stage to compute the dual variable. This modification not only reduces the sample complexity, but also makes the algorithm applicable to a boarder family of nonconvex min-max problems. Second, CODA+ has a smaller communication complexity for homogeneous data than that for heterogeneous data while the previous analysis of CODA yields the same communication complexity for homogeneous data and heterogeneous data.

We have the following lemma to bound the convergence for the subproblem in each $s$-th stage.

Remark. Note that the term $A_{1}$ on the RHS is the drift of clients caused by skipping communication. When $D=0$, i.e., the machines have homogeneous data distribution, we need $\eta I=O\left(\frac{1}{K}\right)$, then $A_{1}$ can be merged with the last term. When $D>0$, we need $\eta I^{2}=O\left(\frac{1}{K}\right)$, which means that $I$ has to be smaller in heterogeneous data setting and thus the communication complexity is higher.

Remark. The key difference between the analysis of CODA+ and that of CODA lies at how to handle the term $(\alpha_{0}-\alpha)^{2}$ in Lemma 1. In CODA, the initial dual variable $\alpha_{0}$ is computed from the initial primal variable $\mathbf{v}_{0}$, which then reduces the error term $(\alpha_{0}-\alpha)^{2}$ to one similar to $\|\mathbf{v}_{0}-\mathbf{v}\|^{2}$, which is then bounded by the primal objective gap due to the PL condition. However, since we do not conduct the extra computation of $\alpha_{0}$ from $\mathbf{v}_{0}$, our analysis directly deals with such error term by using the duality gap of $f^{s}$. This technique is originally developed by [yan2020optimal].

Remark. Due to the PL condition, the step size $\eta$ decreases geometrically. Accordingly, $I$ increases geometrically due to Lemma 1, and $I$ increases with a faster rate when the data are homogeneous than that when data are heterogeneous. As a result, the total number of communications in homogeneous setting is much less than that in heterogeneous setting.

Although CODA+ has a significantly reduced communication complexity for homogeneous data, it is still suffering from a high communication complexity for heterogeneous data. Even for the homogeneous data, CODA+ has a worse communication complexity with a dependence on the number of clients $K$ than the NPA algorithm for FDAM with a large batch size.

Can we further reduce the communication complexity for FDAM for both homogeneous and heterogenous data without using a large batch size?

The main reason that causes the degeneration in the heterogeneous data setting is the data difference. Even at global optimum $(\mathbf{v}_{*},\alpha_{*})$, the gradient of local functions in different clients could be different and non-zero. In the homogeneous data setting, different clients still produce different solutions due to stochastic error (cf. the $\eta^{2}\sigma^{2}I$ term of $A_{1}$ in Lemma 1). These together contribute to the client drift.

To correct the client drift, we propose to leverage the idea of stochastic controlled averaging due to [karimireddy2019scaffold]. The key idea is to maintain and update a control variate to accommodate the client drift, which is taken into account when updating the local solutions. In the proposed algorithm CODASCA for FDAM, we apply control variates to both primal and dual variables. CODASCA shares the same stagewise framework as CODA+, where a strongly convex strongly concave subproblem is constructed and optimized in a distributed fashion approximatly in each stage. The steps of CODASCA are presented in Algorithm 3 and Algorithm 4. Below, we describe the algorithm in each stage.

Each stage has $R$ communication rounds. Between two rounds, there are $I$ local updates, each machine $k$ does the local updates as

where $c_{\mathbf{v}}^{k},c_{\mathbf{v}}$ are local and global control variates for the primal variable, and $c^{k}_{\alpha},c_{\alpha}$ are local and global control variates for the dual variable. Note that $\nabla_{\mathbf{v}}F^{s}_{k}(\mathbf{v}^{k}_{r,t},\alpha^{k}_{r,t};\mathbf{z}^{t}_{r,t})$ and $\nabla_{\alpha}F^{s}_{k}(\mathbf{v}^{k}_{r,t},\alpha^{k}_{r,t};\mathbf{z}^{k}_{r,t})$ are unbiased stochastic gradient on local data. However, they are biased estimate of global gradient when data on different clients are heterogeneous. Intuitively, the term $-c_{\mathbf{v}}^{k}+c_{\mathbf{v}}$ and $-c_{\alpha}^{k}+c_{\alpha}$ work to correct the local gradients to get closer to the global gradient. They also play a role of reducing variance of stochastic gradients, which is helpful as well to reduce the communication complexity in the homogeneous data setting.

At each communication round, the primal and dual variables on all clients get aggregated, averaged and broadcast to all clients. The control variates $c$ at $r$-th round get updated as

which is equivalent to

Notice that they are simply the average of stochastic gradients used in this round. An alternative way to compute the control variates is by computing the stochastic gradient with a large batch of extra samples at the same point at each client, but this would bring extra cost and is unnecessary. $c_{\mathbf{v}}$ and $c_{\alpha}$ are averages of $c_{\mathbf{v}}^{k}$ and $c_{\alpha}^{k}$ over all clients. After the local primal and dual variables are averaged, an extrapolation step with $\eta_{g}>1$ is performed. This step will boost the convergence.

In order to establish the convergence of CODASCA, we first present a key lemma below.

Remark. Compared the above bound with that in Lemma 1, in particular the term $A_{2}$ vs the term $A_{1}$, we can see that CODASCA will not be affected by the data heterogeneity $D>0$, and the stochastic variance is also much reduced. As will seen in the next theorem, the value of $\tilde{\eta}$ and $R$ will keep the same in all stages. Therefore, by decreasing local step size $\eta_{l}$ geometrically, the communication window size $I_{s}$ will increase geometrically to ensure $\tilde{\eta}\leq O(1)$.

The main convergence result of CODASCA is presented below.

Remark. (i) The number of communications is $\widetilde{O}\left(\frac{1}{\mu}\right)$, which is independent of number of clients $K$ and the accuracy level $\epsilon$. This is a significant improvement over CODA+, which has a communication complexity of $\widetilde{O}\left(K/\mu+1/({\mu^{3/2}\epsilon^{1/2}})\right)$ in heterogeneous setting. Moreover, $\widetilde{O}\left(1/({\mu})\right)$ is a nearly optimal rate up to a logarithmic factor, since $O(1/\mu)$ is the lower bound communication complexity of distributed strongly convex optimization [karimireddy2019scaffold, ArjevaniS15] and strongly convexity is a stronger condition than the PL condition.

(ii) Each stage has the same number of communication rounds. However, $I_{s}$ increases geometrically. Therefore, the number of iterations and samples in a stage increase geometrically. Theoretically, we can also set $\eta^{s}_{l}$ to the same value as the one in the last stage, correspondingly $I_{s}$ can be set as a fixed large value. But this increases the number of samples to be processed without further speeding up the convergence. Our setting of $I_{s}$ is a balance between skipping communications and reducing sample complexity. For simplicity, we use the fixed setting of $I_{s}$ to compare CODASCA and the baseline CODA+ in our experiment to corroborate the theory.

(iii) The local step size $\eta_{l}$ of CODASCA decreases in a similar way as the step size $\eta$ in CODA+. But the $I_{s}=O(1/(\sqrt{K}\eta_{l}^{s}))$ in CODASCA increases with a faster rate than that $I_{s}=O(1/(\sqrt{K\eta_{s}}))$ in CODA+ on heterogeneous data. It is noticeable that different from CODA+, we do not need Assumption 2 which bounds the client drift. This means CODASCA can be applied to optimizing the global objective even if local objectives arbitrarily deviate from the global function.

In this section, we first verify the effectiveness of CODASCA compared to CODA+ on various datasets, including two benchmark datasets, i.e., ImageNet, CIFAR100 [imagenet_cvpr09, krizhevsky2009cifar] and a constructed large-scale chest X-ray dataset. Then, we demonstrate the effectiveness of FDAM on improving the performance on a single domain (CheXpert) by using data from multiple sources. For notations, $K$ denotes the number of “clients” (# of machines, # of data sources) and $I$ denotes the communication window size.

Chest X-ray datasets. Five medical chest X-ray datasets, i.e., CheXpert, ChestXray14, MIMIC-CXR, PadChest, ChestXray-AD [chexpert19, wang2017chestx, johnson2019mimic, bustos2020padchest, nguyen2020vindr] are collected from different organizations. The statistics of these medical datasets are summarized in Table 2. We construct five binary classification tasks for predicting five popular diseases, Cardiomegaly (C0), Edema (C1), Consolidation (C2), Atelectasis (C3), P. Effusion (C4), as in CheXpert competition [chexpert19] for our experiments. These datasets are naturally imbalanced and heterogeneous due to different patients’ populations, different data collection protocols and etc. We refer to the whole medical dataset as ChestXray-IH.

Imbalanced and Heterogeneous (IH) Benchmark Datasets. For benchmark datasets, we manually construct the imbalanced heterogeneous dataset. For ImageNet, we first randomly select 500 classes as positive class and 500 classes as negative class. To increase data heterogeneity, we further split all positive/negative classes into $K$ groups so that each split only owns samples from unique classes without overlapping with that of other groups. To increase data imbalance level, we randomly remove some samples from positive classes for each machine. Please note that due to this operation, the whole sample set for different $K$ is different. We refer to the proportion of positive samples in all samples as imbalance ratio ($imratio$). For CIFAR100, we follow similar steps to construct imbalanced heterogeneous data. We keep the testing/validation set untouched and keep them balanced. For imbalance ratio (imratio), we explore two ratios: 10% and 30%. We refer to the constructed datasets as ImageNet-IH (10%), ImageNet-IH (30%), CIFAR100-IH (10%), CIFAR100-IH (30%). Due to the limited space, we only report imratio=10% with DenseNet121 and defer the other results to supplement.

Parameters and Settings. We train Desenet121 on all datasets. For the parameters in CODASCA/CODA+, we tune $1/\gamma$ in [500, 700, 1000] and $\eta$ in [0.1, 0.01, 0.001]. For learning rate schedule, we decay the step size by 3 times every $T_{0}$ iterations, where $T_{0}$ is tuned in [2000, 3000, 4000]. We experiment with a fixed value of $I$ selected from [1, 32, 64, 128, 512, 1024]. We tune $\eta_{g}$ in [1.1, 1, 0.99, 0.999]. The local batch size is set to 32 for each machine. We run a total of 20000 iterations for all experiments.

In this work, we have conducted comprehensive studies of federated learning for deep AUC maximization. We analyzed a stronger baseline for deep AUC maximization by establishing its convergence for both homogeneous data and heterogeneous data. We also developed an improved variant by adding control variates to the local stochastic gradients for both primal and dual variables, which dramatically reduces the communication complexity. Besides a strong theory guarantee, we also exhibit the power of FDAM on real world medical imaging problems. We have shown that our FDAM method can improve the performance on medical imaging classification tasks by leveraging data from different organizations that are kept locally.