Linear Convergent Decentralized Optimization with Compression

Distributed optimization solves the following optimization problem

with $n$ computing agents and a communication network. Each $f_{i}({\bm{x}}):\mathbb{R}^{d}\rightarrow\mathbb{R}$ is a local objective function of agent $i$ and typically defined on the data $\mathcal{D}_{i}$ settled at that agent. The data distributions $\{\mathcal{D}_{i}\}$ can be heterogeneous depending on the applications such as in federated learning. The variable ${\bm{x}}\in\mathbb{R}^{d}$ often represents model parameters in machine learning. A distributed optimization algorithm seeks an optimal solution that minimizes the overall objective function $f({\bm{x}})$ collectively. According to the communication topology, existing algorithms can be conceptually categorized into centralized and decentralized ones. Specifically, centralized algorithms require global communication between agents (through central agents or parameter servers). While decentralized algorithms only require local communication between connected agents and are more widely applicable than centralized ones. In both paradigms, the computation can be relatively fast with powerful computing devices; efficient communication is the key to improve algorithm efficiency and system scalability, especially when the network bandwidth is limited.

In recent years, various communication compression techniques, such as quantization and sparsification, have been developed to reduce communication costs. Notably, extensive studies (Seide et al., 2014; Alistarh et al., 2017; Bernstein et al., 2018; Stich et al., 2018; Karimireddy et al., 2019; Mishchenko et al., 2019; Tang et al., 2019b; Liu et al., 2020) have utilized gradient compression to significantly boost communication efficiency for centralized optimization. They enable efficient large-scale optimization while maintaining comparable convergence rates and practical performance with their non-compressed counterparts. This great success has suggested the potential and significance of communication compression in decentralized algorithms.

While extensive attention has been paid to centralized optimization, communication compression is relatively less studied in decentralized algorithms because the algorithm design and analysis are more challenging in order to cover general communication topologies. There are recent efforts trying to push this research direction. For instance, DCD-SGD and ECD-SGD (Tang et al., 2018a) introduce difference compression and extrapolation compression to reduce model compression error. (Reisizadeh et al., 2019a; b) introduce QDGD and QuanTimed-DSGD to achieve exact convergence with small stepsize. DeepSqueeze (Tang et al., 2019a) directly compresses the local model and compensates the compression error in the next iteration. CHOCO-SGD (Koloskova et al., 2019; 2020) presents a novel quantized gossip algorithm that reduces compression error by difference compression and preserves the model average. Nevertheless, most existing works focus on the compression of primal-only algorithms, i.e., reduce to DGD (Nedic & Ozdaglar, 2009; Yuan et al., 2016) or P-DSGD (Lian et al., 2017). They are unsatisfying in terms of convergence rate, stability, and the capability to handle heterogeneous data. Part of the reason is that they inherit the drawback of DGD-type algorithms, whose convergence rate is slow in heterogeneous data scenarios where the data distributions are significantly different from agent to agent.

In the literature of decentralized optimization, it has been proved that primal-dual algorithms can achieve faster converge rates and better support heterogeneous data (Ling et al., 2015; Shi et al., 2015; Li et al., 2019; Yuan et al., 2020). However, it is unknown whether communication compression is feasible for primal-dual algorithms and how fast the convergence can be with compression. In this paper, we attempt to bridge this gap by investigating the communication compression for primal-dual decentralized algorithms. Our major contributions can be summarized as:

• •

  We delineate two key challenges in the algorithm design for communication compression in decentralized optimization, i.e., data heterogeneity and compression error, and motivated by primal-dual algorithms, we propose a novel decentralized algorithm with compression, LEAD.

• •

  We prove that for LEAD, a constant stepsize in the range $(0,2/(\mu+L)]$ is sufficient to ensure linear convergence for strongly convex and smooth objective functions. To the best of our knowledge, LEAD is the first linear convergent decentralized algorithm with compression. Moreover, LEAD provably works with unbiased compression of arbitrary precision.

• •

  We further prove that if the stochastic gradient is used, LEAD converges linearly to the $O(\sigma^{2})$ neighborhood of the optimum with constant stepsize. LEAD is also able to achieve exact convergence to the optimum with diminishing stepsize.

• •

  Extensive experiments on convex problems validate our theoretical analyses, and the empirical study on training deep neural nets shows that LEAD is applicable for nonconvex problems. LEAD achieves state-of-art computation and communication efficiency in all experiments and significantly outperforms the baselines on heterogeneous data. Moreover, LEAD is robust to parameter settings and needs minor effort for parameter tuning.

Decentralized optimization can be traced back to the work by Tsitsiklis et al. (1986). DGD (Nedic & Ozdaglar, 2009) is the most classical decentralized algorithm. It is intuitive and simple but converges slowly due to the diminishing stepsize that is needed to obtain the optimal solution (Yuan et al., 2016). Its stochastic version D-PSGD (Lian et al., 2017) has been shown effective for training nonconvex deep learning models. Algorithms based on primal-dual formulations or gradient tracking are proposed to eliminate the convergence bias in DGD-type algorithms and improve the convergence rate, such as D-ADMM (Mota et al., 2013), DLM (Ling et al., 2015), EXTRA (Shi et al., 2015), NIDS (Li et al., 2019), $D^{2}$ (Tang et al., 2018b), Exact Diffusion (Yuan et al., 2018), OPTRA(Xu et al., 2020), DIGing (Nedic et al., 2017), GSGT (Pu & Nedić, 2020), etc.

Recently, communication compression is applied to decentralized settings by Tang et al. (2018a). It proposes two algorithms, i.e., DCD-SGD and ECD-SGD, which require compression of high accuracy and are not stable with aggressive compression. Reisizadeh et al. (2019a; b) introduce QDGD and QuanTimed-DSGD to achieve exact convergence with small stepsize and the convergence is slow. DeepSqueeze (Tang et al., 2019a) compensates the compression error to the compression in the next iteration. Motivated by the quantized average consensus algorithms, such as (Carli et al., 2010), the quantized gossip algorithm CHOCO-Gossip (Koloskova et al., 2019) converges linearly to the consensual solution. Combining CHOCO-Gossip and D-PSGD leads to a decentralized algorithm with compression, CHOCO-SGD, which converges sublinearly under the strong convexity and gradient boundedness assumptions. Its nonconvex variant is further analyzed in (Koloskova et al., 2020). A new compression scheme using the modulo operation is introduced in (Lu & De Sa, 2020) for decentralized optimization. A general algorithmic framework aiming to maintain the linear convergence of distributed optimization under compressed communication is considered in (Magnússon et al., 2020). It requires a contractive property that is not satisfied by many decentralized algorithms including the algorithm in this paper.

We first introduce notations and definitions used in this work. We use bold upper-case letters such as ${\mathbf{X}}$ to define matrices and bold lower-case letters such as ${\bm{x}}$ to define vectors. Let ${\bm{1}}$ and ${\bm{0}}$ be vectors with all ones and zeros, respectively. Their dimensions will be provided when necessary. Given two matrices ${\mathbf{X}},~{}{\mathbf{Y}}\in\mathbb{R}^{n\times d}$, we define their inner product as $\langle{\mathbf{X}},{\mathbf{Y}}\rangle=\text{tr}({\mathbf{X}}^{\top}{\mathbf{Y}})$ and the norm as $\|{\mathbf{X}}\|=\sqrt{\langle{\mathbf{X}},{\mathbf{X}}\rangle}$. We further define $\langle{\mathbf{X}},{\mathbf{Y}}\rangle_{{\mathbf{P}}}=\text{tr}({\mathbf{X}}^{\top}{\mathbf{P}}{\mathbf{Y}})$ and $\|{\mathbf{X}}\|_{{\mathbf{P}}}=\sqrt{\langle{\mathbf{X}},{\mathbf{X}}\rangle}_{{\mathbf{P}}}$ for any given symmetric positive semidefinite matrix ${\mathbf{P}}\in\mathbb{R}^{n\times n}$. For simplicity, we will majorly use the matrix notation in this work. For instance, each agent $i$ holds an individual estimate ${\bm{x}}_{i}\in\mathbb{R}^{d}$ of the global variable ${\bm{x}}\in\mathbb{R}^{d}$. Let ${\mathbf{X}}^{k}$ and $\nabla{\mathbf{F}}({\mathbf{X}}^{k})$ be the collections of $\{{\bm{x}}_{i}^{k}\}_{i=1}^{n}$ and $\{\nabla f_{i}({\bm{x}}_{i}^{k})\}_{i=1}^{n}$ which are defined below:

We use $\nabla{\mathbf{F}}({\mathbf{X}}^{k};\xi^{k})$ to denote the stochastic approximation of $\nabla{\mathbf{F}}({\mathbf{X}}^{k})$. With these notations, the update ${\mathbf{X}}^{k+1}={\mathbf{X}}^{k}-\eta\nabla{\mathbf{F}}({\mathbf{X}}^{k};\xi^{k})$ means that ${\bm{x}}_{i}^{k+1}={\bm{x}}_{i}^{k}-\eta\nabla f_{i}({\bm{x}}_{i}^{k};\xi_{i}^{k})$ for all $i$. In this paper, we need the average of all rows in ${\mathbf{X}}^{k}$ and $\nabla{\mathbf{F}}({\mathbf{X}}^{k})$, so we define $\mkern 1.5mu\overline{\mkern-1.5mu{\mathbf{X}}\mkern-1.5mu}\mkern 1.5mu^{k}=({{\bm{1}}^{\top}{\mathbf{X}}^{k}})/{n}$ and $\mkern 1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern 1.5mu{\mathbf{F}}({\mathbf{X}}^{k})=({{\bm{1}}^{\top}\nabla{\mathbf{F}}({\mathbf{X}}^{k})})/{n}$. They are row vectors, and we will take a transpose if we need a column vector. The pseudoinverse of a matrix ${\mathbf{M}}$ is denoted as ${\mathbf{M}}^{\dagger}$. The largest, $i$th-largest, and smallest nonzero eigenvalues of a symmetric matrix ${\mathbf{M}}$ are $\lambda_{\max}({\mathbf{M}})$, $\lambda_{i}({\mathbf{M}})$, and $\lambda_{\min}({\mathbf{M}})$.

Assumption 1 implies that $-1<\lambda_{n}({\mathbf{W}})\leq\lambda_{n-1}({\mathbf{W}})\leq\cdots\lambda_{2}({\mathbf{W}})<\lambda_{1}({\mathbf{W}})=1$ and ${\mathbf{W}}{\bm{1}}={\bm{1}}$ (Xiao & Boyd, 2004; Shi et al., 2015). The matrix multiplication ${\mathbf{X}}^{k+1}={\mathbf{W}}{\mathbf{X}}^{k}$ describes that agent $i$ takes a weighted sum from its neighbors and itself, i.e., ${\bm{x}}_{i}^{k+1}=\sum_{j\in\mathcal{N}_{i}\cup\{i\}}{w}_{ij}{\bm{x}}^{k}_{j}$, where $\mathcal{N}_{i}$ denotes the neighbors of agent $i$.

In this section, we show the convergence rate for the proposed algorithm LEAD. Before showing the main theorem, we make some assumptions, which are commonly used for the analysis of decentralized optimization algorithms. All proofs are provided in Appendix E.

We consider three machine learning problems – $\ell_{2}$-regularized linear regression, logistic regression, and deep neural network. The proposed LEAD is compared with QDGD (Reisizadeh et al., 2019a), DeepSqueeze (Tang et al., 2019a), CHOCO-SGD (Koloskova et al., 2019), and two non-compressed algorithms DGD (Yuan et al., 2016) and NIDS (Li et al., 2019).

Setup. We consider eight machines connected in a ring topology network. Each agent can only exchange information with its two 1-hop neighbors. The mixing weight is simply set as $1/3$. For compression, we use the unbiased $b$-bits quantization method with $\infty$-norm

where $\cdot$ is the Hadamard product, $|{\bm{x}}|$ is the elementwise absolute value of ${\bm{x}}$, and ${\bm{u}}$ is a random vector uniformly distributed in $[0,1]^{d}$. Only sign$({\bm{x}})$, norm $\|{\bm{x}}\|_{\infty}$, and integers in the bracket need to be transmitted. Note that this quantization method is similar to the quantization used in QSGD (Alistarh et al., 2017) and CHOCO-SGD (Koloskova et al., 2019), but we use the $\infty$-norm scaling instead of the $2$-norm. This small change brings significant improvement on compression precision as justified both theoretically and empirically in Appendix C. In this section, we choose $2$-bit quantization and quantize the data blockwise (block size = $512$).

For all experiments, we tune the stepsize $\eta$ from $\{0.01,0.05,0.1,0.5\}$. For QDGD, CHOCO-SGD and Deepsqueeze, $\gamma$ is tuned from $\{0.01,0.1,0.2,0.4,0.6,0.8,1.0\}$. Note that different notations are used in their original papers. Here we uniformly denote the stepsize as $\eta$ and the additional parameter in these algorithms as $\gamma$ for simplicity. For LEAD, we simply fix $\alpha=0.5$ and $\gamma=1.0$ for all experiments since we find LEAD is robust to parameter settings as we validate in the parameter sensitivity analysis in Appendix D.1. This indicates the minor effort needed for tuning LEAD. Detailed parameter settings for all experiments are summarized in Appendix D.3.

Linear regression. We consider the problem: $f({\bm{x}})=\sum_{i=1}^{n}(\|{\mathbf{A}}_{i}{\bm{x}}-{\bm{b}}_{i}\|^{2}+\lambda\|{\bm{x}}\|^{2})$. Data matrices ${\mathbf{A}}_{i}\in\mathbb{R}^{200\times 200}$ and the true solution ${\bm{x}}^{\prime}$ is randomly synthesized. The values ${\bm{b}}_{i}$ are generated by adding Gaussian noise to ${\mathbf{A}}_{i}{\bm{x}}^{\prime}$. We let $\lambda=0.1$ and the optimal solution of the linear regression problem be ${\bm{x}}^{*}$. We use full-batch gradient to exclude the impact of gradient variance. The performance is showed in Fig. 1. The distance to ${\bm{x}}^{*}$ in Fig. 1a and the consensus error in Fig. 1c verify that LEAD converges exponentially to the optimal consensual solution. It significantly outperforms most baselines and matches NIDS well under the same number of iterations. Fig. 1b demonstrates the benefit of compression when considering the communication bits. Fig. 1d shows that the compression error vanishes for both LEAD and CHOCO-SGD while the compression error is pretty large for QDGD and DeepSqueeze because they directly compress the local models.

Logistic regression. We further consider a logistic regression problem on the MNIST dataset. The regularization parameter is $10^{-4}$. We consider both homogeneous and heterogeneous data settings. In the homogeneous setting, the data samples are randomly shuffled before being uniformly partitioned among all agents such that the data distribution from each agent is very similar. In the heterogeneous setting, the samples are first sorted by their labels and then partitioned among agents. Due to the space limit, we mainly present the results in heterogeneous setting here and defer the homogeneous setting to Appendix D.2. The results using full-batch gradient and mini-batch gradient (the mini-batch size is $512$ for each agent) are showed in Fig. 2 and Fig. 3 respectively and both settings shows the faster convergence and higher precision of LEAD.

Neural network. We empirically study the performance of LEAD in optimizing deep neural network by training AlexNet ($240$ MB) on CIFAR10 dataset. The mini-batch size is $64$ for each agents. Both the homogeneous and heterogeneous case are showed in Fig. 4. In the homogeneous case, CHOCO-SGD, DeepSqueeze and LEAD perform similarly and outperform the non-compressed variants in terms of communication efficiency, but CHOCO-SGD and DeepSqueeze need more efforts for parameter tuning because their convergence is sensitive to the setting of $\gamma$. In the heterogeneous cases, LEAD achieves the fastest and most stable convergence. Note that in this setting, sufficient information exchange is more important for convergence because models from different agents are moving to significantly diverse directions. In such case, DGD only converges with smaller stepsize and its communication compressed variants, including QDGD, DeepSqueeze and CHOCO-SGD, diverge in all parameter settings we try.

In summary, our experiments verify our theoretical analysis and show that LEAD is able to handle data heterogeneity very well. Furthermore, the performance of LEAD is robust to parameter settings and needs less effort for parameter tuning, which is critical in real-world applications.

In this paper, we investigate the communication compression in decentralized optimization. Motivated by primal-dual algorithms, a novel decentralized algorithm with compression, LEAD, is proposed to achieve faster convergence rate and to better handle heterogeneous data while enjoying the benefit of efficient communication. The nontrivial analyses on the coupled dynamics of inexact primal and dual updates as well as compression error establish the linear convergence of LEAD when full gradient is used and the linear convergence to the $\mathcal{O}(\sigma^{2})$ neighborhood of the optimum when stochastic gradient is used. Extensive experiments validate the theoretical analysis and demonstrate the state-of-the-art efficiency and robustness of LEAD. LEAD is also applicable to non-convex problems as empirically verified in the neural network experiments but we leave the non-convex analysis as the future work.

Xiaorui Liu and Dr. Jiliang Tang are supported by the National Science Foundation (NSF) under grant numbers CNS-1815636, IIS-1928278, IIS-1714741, IIS-1845081, IIS-1907704, and IIS-1955285. Yao Li and Dr. Ming Yan are supported by NSF grant DMS-2012439 and Facebook Faculty Research Award (Systems for ML). Dr. Rongrong Wang is supported by NSF grant CCF-1909523.