Decentralized Federated Averaging

We answer the above questions affirmatively by proposing the decentralized FedAvg with momentum (DFedAvgM). To further reduce the communication cost between clients, we also integrate quantization with DFedAvgM. Our contributions in this paper are elaborated below in threefold.

• •

  Algorithmically, we extend FedAvg to the decentralized setting, where all clients are connected by an undirected graph. We motivate DFedAvgM from the decentralized SGD (DSGD) algorithm. In particular, we use SGD with momentum to train ML models on each client. To reduce the communication cost between each client, we further introduce a quantized version of DFedAvgM, in which each client will send and receive a quantized model.
  

• •

  Theoretically, we prove the convergence of (quantized) DFedAvgM. Our theoretical results show that the convergence rate of (quantized) DFedAvgM is not inferior to that of SGD or DSGD. More specifically, we show that the convergence rates of both DFedAvgM and quantized DFedAvgM depend on the local training and the graph that connects all clients. Besides the convergence results under nonconvex assumptions, we also establish their convergence guarantee under the Polyak-Łojasiewicz (PŁ) condition, which has been widely studied in nonconvex optimization. Under the PŁ condition, we establish a faster convergence rate for (quantized) DFedAvgM. Furthermore, we present a sufficient condition to guarantee reducing communication costs.
  

• •

  Empirically, we perform extensive numerical experiments on training deep neural networks (DNNs) on various datasets in both IID and Non-IID settings. Our results show the effectiveness of (quantized) DFedAvgM for training ML models, saving communication costs, and protecting training data’s membership privacy.

We briefly review three lines of work that are most related to this paper, i.e., federated learning, decentralized training, and decentralized federated learning.

Federated Learning. Many variants of FedAvg have been developed with theoretical guarantees. [8] uses the momentum method for local clients in FedAvg. [9] proposes the adaptive FedAvg, whose central parameter server uses the adaptive learning rate ti aggregate local models. Lazy and quantizatized gradients are used to reduce communications [10, 11]. [12] proposes a Newton-type scheme for FL. The convergence analysis of FedAvg on heterogeneous data is discussed by [13, 14]. The advances and open problems in FL is available in two survey papers [15, 16].

Decentralized Training. Decentralized algorithms are originally developed to calculate the mean of data that are distributed over multiple sensors [17, 18, 19, 20]. Decentralized (sub)gradient descents (DGD), one of the simplest and efficient decentralized algorithms, have been studied in [21, 22, 23, 24, 25]. In DGD, the convexity assumption is unnecessary [26], which makes DGD useful for nonconvex optimization. A provably convergent DSGD is proposed in [27, 28, 4]. [27] provides the complexity result of a stochastic decentralized algorithm. [28] designs a stochastic decentralized algorithm with the dual information and provide the theoretical convergence guarantee. [4] proves that DSGD outperforms SGD in communication efficiency. Asynchronous DSGD is analyzed in [29]. DGD with momentum is proposed in [30, 31]. Quantized DSGD has been proposed in [32].

Decentralized Federated Learning. Decentralized FL is a learning paradigm of choice when the edge devices do not trust the central server in protecting their privacy [33]. The authors in [34] propose a novel FL framework without a central server for medical applications, and the new method offers a highly dynamic peer-to-peer environment. [6] considers training an ML model with a connected network whose nodes take a Bayesian-like approach by introducing a belief of the parameter space.

We organize this paper as follows: in section 2, we present a mathematical formulation of our problem and some necessary assumption. In section 3, we present the DFedAvgM and its quantized algorithms. We present the convergence of the proposed algorithm in section 4. We provide extensive numerical verification of DFedAvgM in section 6. This paper ends up with concluding remarks. Technical proofs and more experimental details are provided in the appendix.

We denote scalars and vectors by lower case and lower case boldface letters, respectively, and matrices by upper case boldface letters. For a vector ${\bf x}=(x_{1},\cdots,x_{d})\in\mathbb{R}^{d}$, we denote its $\ell_{p}$ norm ($p\geq 1$) by $\|{\bf x}\|_{p}={\small(\sum_{i=1}^{d}|x_{i}|^{p})^{1/p}}$, and denote the $\ell_{\infty}$ norm of ${\bf x}$ by $\|{\bf x}\|_{\infty}=\max_{i=1}^{d}|x_{i}|$ and denote $\ell_{2}$ norm as $\|{\bf x}\|$. For a matrix ${\bf A}$, we denote its transpose as ${\bf A}^{\top}$. Given two sequences $\{a_{n}\}$ and $\{b_{n}\}$, we write $a_{n}=\mathcal{O}(b_{n})$ if there exists a positive constant $0<C<+\infty$ such that $a_{n}\leq Cb_{n}$, and we write $a_{n}=\Theta(b_{n})$ if there exist two positive constants $C_{1}$ and $C_{2}$ such that $a_{n}\leq C_{1}b_{n}$ and $b_{n}\leq C_{2}a_{n}$. $\widetilde{\mathcal{O}}(a_{n})$ hides the logarithmic factor of $a_{n}$. For a function $f({\bf x}):\mathbb{R}^{d}\rightarrow\mathbb{R}$, we denote its gradient as $\nabla f({\bf x})$ and its Hessian as $\nabla^{2}f({\bf x})$, and denote its minimum as $\min f$. We use $\mathbb{E}[\cdot]$ to denote the expectation with respect to the underlying probability space.

We first briefly review the previous work on decentralized training, which carries out in the following fashion:

1. 1.

   client $i$ holds an approximate copy of the parameters ${\bf x}(i)\in\mathbb{R}^{d}$ and calculate an unbiased estimate of $\nabla f_{i}:={\bf g}(i)$ at ${\bf x}(i)$. $({\bf x}(i))_{1\leq i\leq m}$ can be non-consensus;

2. 2.

   (communication) client $i$ updates its local parameters ${\bf x}(i)$ as the weighted average of its neighbors: $\tilde{{\bf x}}(i)=\sum_{l\in\mathcal{N}(i)}w_{i,l}{\bf x}(l)$;

3. 3.

   (training) client $i$ updates its parameters as ${\bf x}(i)\leftarrow\tilde{{\bf x}}(i)-\eta{\bf g}(i)$ with a learning rate $\eta>0$.

The traditional decentralization can be described in Figure 1 (a), in which, a communication step is needed after each training iteration. This indicate that the above vanilla decentralized algorithm is different from FedAvg, and the later performs multiple local training step before communication. To this end, we have to slightly modify the scheme of the decentralized algorithm. For simplicity, we consider modifying DSGD to motivate our decentralized FedAvg algorithm. Note that when the original DGD is applied to solve (1), we end up with the following iteration

where we used the fact that $\sum_{l\in\mathcal{N}(i)}w_{i,l}=1$. In (2), if we replace ${\bf x}^{t}(l)$ by ${\bf x}^{t}(i)$, the algorithm then iterates as

In (3), clients communicate with their neighbors after one training iteration, which is then possible to generalize to the federated optimization setting. We replace the single SGD iteration in (3) with multiple SGD with heavy-ball [38] iterations. Therefore, the DFedAvgM can be presented as follows: In each $t\in\mathbb{Z}^{+}$, for each client $i\in\{1,2,\ldots,m\}$, let ${\bf y}^{t,-1}(i)={\bf y}^{t,0}(i)={\bf x}^{t}(i)$. The inner iteration in each node then performs as

where $\mathbb{E}\tilde{{\bf g}}^{t,k}(i)=\nabla f_{i}({\bf y}^{t,k}(i))$. After $K$ inner iterations in each local client, the resulting parameters ${\bf z}^{t}(i)\leftarrow{\bf y}^{t,K}(i)$ is sent to its neighbors ($\mathcal{N}(i)$). Every client then updates its parameters by taking the local averaging as follows

The procedure of DFedAvgM can be illustrated as Figure 1 (b). It is seen that DFedAvgM plays the tradeoff between local computing and communications. It is well-known that the communication costs are usually much more expensive than the computation costs [39], which indicates DFedAvgM can be more efficient than DSGD.

In DFedAvgM, client $i$ needs to send ${\bf x}^{t}(i)$ to its neighbours $\mathcal{N}(i)$. Thus, when the number of neighbours $\mathcal{N}(i)$ grows, client-client communications become the major bottleneck on algorithms’ efficiency. We leverage the quantization trick to reduce the communication cost [40, 41]. In particular, we consider the following quantization procedure: Given a constant $s>0$ and the limited bit number $b\in\mathbb{Z}^{+}$, the representable range is then $\{-2^{b-1}s,-(2^{b-1}-1)s,\ldots,0,s,2s,\ldots,(2^{b-1}-1)s\}$. For any $a\in\mathbb{R}$ with $-2^{b-1}s\leq a<(2^{b-1}-1)s$, we can find an integer $k\in\mathbb{Z}$ such that $ks\leq a<(k+1)s$ and we then use $ks$ to replace $a$. The above quantization scheme is deterministic, which can be written as $q(a):=\lfloor\frac{a}{s}\rfloor s$ for $a\in\mathbb{R}$; Besides the deterministic rule, the stochastic quantization uses the following scheme

It is easy to see that the stochastic quantization is unbiased, i.e., $\mathbb{E}[q(a)]=a$ for any $a\in\mathbb{R}$. When $s$ is small, deterministic and stochastic quantization schemes perform very similarly. For a vector ${\bf x}\in\mathbb{R}^{d}$ whose coordinates are all stored with $32$ bits, we consider quantizing all coordinates of ${\bf x}=[{x}_{1},{x}_{2},\ldots,{x}_{d}]\in\mathbb{R}^{d}$. The multi-dimension quantization operator is then defined as

For both deterministic and stochastic quantization schemes, we have $\mathbb{E}\|Q({\bf x})-{\bf x}\|^{2}\leq\frac{d}{4}s^{2}$ if ${x}_{i}\in[-2^{b-1}s,(2^{b-1}-1)s]$ for $i\in\{1,2,\ldots,d\}$. In this paper, we consider a quantization operator with the following assumption, which hold for the two quantization schemes mentioned above.

Directly quantize the parameters is feasible for sufficiently smooth loss functions, but may be impossible for DNNs. To this end, we consider quantizing the difference the difference of parameters. Quantized DFedAvgM can be summarized as: After running (4) $K$ times, client $i$ quantizes ${\small{\bf q}^{t}(i)\leftarrow Q({\bf y}^{t,K}(i)-{\bf x}^{t}(i))}$ and send it to ${\small\mathcal{N}(i)}$. After receiving ${\small[{\bf q}^{t}(j)]_{j\in\mathcal{N}(i)}}$, every client updates its local parameters as

In each communication, client $i$ just needs to send the pair $(s,{\bf q}^{t}(i))$ to $\mathcal{N}(i)$, whose representation requires $(32+db)\textrm{deg}(\mathcal{N}(i))$ bits rather than $32d\textrm{deg}(\mathcal{N}(i))$ bits for sending the unquantized version. If $d$ is large and $b<32$, the communications can be significantly reduced.

We define ${\bf 1}:=[1,1,\ldots,1]^{\top}\in\mathbb{R}^{m}$ and

For a matrix ${\bf A}$, we denote its spectral norm as $\|{\bf A}\|_{\textrm{op}}$. We also define ${\bf X}:=\begin{bmatrix}{\bf x}(1),{\bf x}(2),\ldots,{\bf x}(m)\end{bmatrix}^{\top}\in\mathbb{R}^{m\times d}$.

In [Proposition 1, [21]], the author also proved that $\|W^{k}-{\bf P}\|_{\textrm{op}}\leq C\lambda^{k}$ for some $C>0$ that depends on the matrix.

With the fact that ${\bf y}^{t,K}(i)={\bf z}^{t}(i)$, Lemma 3 also holds for ${\bf z}^{t}(i)$.