Echo-CGC: A Communication-Efficient Byzantine-tolerant Distributed Machine Learning Algorithm in Single-Hop Radio Network

Single-Hop Radio Network   We consider the standard radio network model in the literature, e.g., [1, 3, 14]. In particular, the underlying communication layer ensures the reliable local broadcast property [3]. In other words, the channel is perfectly reliable, and a local broadcast is correctly received by all neighbors. As noted in [1, 3], this assumption does not typically hold in the current deployed wireless networks, but it is possible to realize such a property with high probability in practice with the help from the MAC layer [2] or physical layer [17].

In our system, nodes can be uniquely identified, i.e., each node has a unique identifier. We assume that a faulty node may not spoof another node’s identity. The communication network is assumed to be single-hop; that is, each pair of nodes are within the communication range of each other. Moreover, time is divided into slots, and each node proceeds synchronously. Message collision is not possible because of the nodes follow a pre-determined TDMA schedule that determine the transmitting node in each slot and the transmission protocol is jam-resistant. Each slot is assumed to be large enough so that it is possible for a node to transmit a gradient. We also assume that each communication round (or communication step) is divided into $n$ slots, and the TDMA schedule assigns each node to a unique slot. For ease of discussion, node $i$ is scheduled to transmit at slot $i$.

Stochastic Gradient Descent and Parameter Server   In this work, we focus on the Byzantine-tolerant distributed Stochastic Gradient Descent (SGD) algorithms, which are popular in the optimization and machine learning literature [4, 8, 11, 5]. Given a cost function $Q$, the (sequential) SGD algorithm outputs an optimal parameter $w^{*}$ such that

An SGD algorithm executes in an iterative fashion, where in each round $t$, the algorithm computes the gradient of the cost function $Q$ at parameter $w^{t}$ and updates the parameter with the gradient.

Synchronous Parameter Server Model   Computation of gradients is typically expensive and slow. One popular framework to speed up the computation is the parameter server model, in which the parameter server distributes the computation tasks to $n$ workers and aggregates their computed gradients to update the parameter in each round. Following the convention, we will use node and worker interchangeably.

We assume a synchronous system, i.e., the computation and communication delays are bounded, and the server and workers know the bound. Consequently, if the server does not receive a message from worker $i$ by the end of some round, then the server identifies that worker $i$ is faulty.

Formally speaking, a distributed SGD algorithm in the parameter server model proceeds in synchronous rounds, and executes the following three steps in each round $t$:

1. 1.

   The parameter server broadcasts parameter $w^{t}$ to the workers.

2. 2.

   Each worker $j$ randomly chooses a random data batch $\xi_{j}^{t}$ from the dataset (shared by all the workers) and computes an estimate, $g_{j}^{t}$, of the gradient $\nabla Q(w^{t})$ of the cost function $Q$ using $\xi_{j}^{t}$ and $w^{t}$.

3. 3.

   The server aggregates estimated gradients from all workers and updates the parameter using the gradient descent approach with step size $\eta$:

   $$w^{t+1}=w^{t}-\eta\sum_{j=1}^{n}g_{j}^{t}$$

   (2)

Fault Model and Byzantine SGD   Following [11, 4, 6], our system consists of $n$ workers, up to $f$ of which might be Byzantine faulty. We assume that the central parameter server is always fault-free.

Byzantine workers may be controlled by an omniscient adversary which has the knowledge of the current parameter (at the server) and the local gradient of all the other workers, and may have arbitrary behaviors. They can send arbitrary messages. However, due to the reliable local broadcast property of the radio network model, they cannot send inconsistent messages to the server and other workers. They also cannot spoof another node’s identity. Our goal is therefore to design a distributed SGD algorithm that solves Equation (1) in the presence of up to $f$ Byzantine workers.

Workers that are not Byzantine faulty are called fault-free workers. These workers follow the algorithm specification faithfully. For a given execution of the algorithm, we denote $\mathcal{H}$ as the set of fault-free workers and $\mathcal{B}$ as the set of Byzantine workers. For brevity, we denote $h=|\mathcal{H}|$ and $b=|\mathcal{B}|$; hence, we have $b\leq f$ and $h\geq n-f$.

Communication Complexity   We are interested in minimizing the total number of bits that need to be transmitted from workers to the parameter server in each round. Prior algorithms [11, 4] transmit $n$ gradients in a $d$-dimensional space in each round, since each node needs to transmit its local gradient to the centralized server. Typically, each gradient consists of $d$ floats or doubles (i.e., a single primitive floating point data structure for each dimension).

We assume that the cost function $Q$ satisfies some standard properties used in the literature [4, 8, 6], including convexity, differentiability, Lipschitz smoothness, and strong convexity. Following the convention, we use $\left<a,b\right>$ to represent the dot product of two vectors $a$ and $b$ in the $d$-dimensional space $\mathbb{R}^{d}$.

We also assume that the random data batches are independently and identically distributed from the dataset. Before stating the assumptions, we formally introduce the concept of randomness in the framework. Similar to typical stochastic gradient descent algorithms, the only randomness is due to the random data batches $\xi_{j}^{t}$ sampled by each fault-free worker $j\in\mathcal{H}$ in each round $t$, which further makes $g_{j}^{t}$ as well as $w^{t+1}$ non-deterministic. In the case when a worker uses the entire dataset to train model, $g_{j}^{t}=\nabla Q(w^{t})$. Hence, the result is deterministic, i.e., each fault-free worker derives the same gradient. In practice, data batch is a small sample of the entire data set.¹¹1Reference [11] works on a different formulation in which each worker may have a different local cost function.

Formally speaking, we denote an operator $\mathbb{E}_{\Xi^{t}}(\cdot\mid w^{t},\mathcal{G}_{\mathcal{B}}^{t})$ as the conditional expectation operator over the set of random batches $\Xi^{t}=\{\xi_{j}^{t},j=1,2,\ldots,n\}$ in round $t$ given (i) the parameter $w^{t}$, and (ii) the set of Byzantine gradients $\mathcal{G}_{\mathcal{B}}^{t}=\{g_{j}^{t}:j\in\mathcal{B}\}$. This conditional expectation operator allows us to treat $w^{t}$, $Q(w^{t})$, and $\nabla Q(w^{t})$ as constants, as well as the Byzantine gradients. This is reasonable because (i) we have the knowledge about $Q$ and $w^{t}$ given an execution, and (ii) the Byzantine gradients are arbitrary, and do not depend on the data batches. From now on, without further specification, we abbreviate the operator $\mathbb{E}_{\Xi^{t}}(\cdot\mid w^{t},\mathcal{G}_{\mathcal{B}}^{t})$ as $\mathbb{E}$.

Below we present two further assumptions of local stochastic gradient $g^{t}_{j}$ at each fault-free worker $j$. Similar to [4, 8], we rely on the two following assumptions for correctness proof.

Notation   We list the most important notations and constants used in our algorithm and analysis in the following table.

In this part, we first analyze the convergence rate $\rho$, which is a constant defined later in Equation (13). Recall a few notations that $h=|\mathcal{H}|$ and $b=|\mathcal{B}|$, where given the execution, $\mathcal{H}$ is the set of fault-free workers and $\mathcal{B}$ is the set of Byzantine workers. Also recall that $L$ and $\mu$ are the constants defined in the Assumption 2 and 3, respectively; $\sigma$ defined in Assumption 5; and $r$ is the deviation ratio used in Echo-CGC. To derive $\rho$, we need to define series of constants based on the given parameters of $n,f,h,b,L,\mu,r$, and $\sigma$.

We first define a constant $\beta$ as

where $k_{x}$ is defined as

We then define a constant $\gamma$ as

where

Finally, we define the convergence rate $\rho$ using $\beta$ and $\gamma$ as follows:

We will prove that under some standard assumptions, the convergence rate $\rho$ is in the interval $[0,1)$. We first present several auxiliary lemmas. Due to page limit, most proofs are presented in Appendix A.

Lemma 4.3 implies that we need to bound $\sigma$ for convergence. In general, Echo-CGC is correct if $\sigma=o(\log{n})$. For brevity, we make the following assumption to simplify the proof of convergence and the analysis of communication complexity. We stress that this assumption can be relaxed using basically the same analysis with a denser mathematical manipulation.

Under Assumption 6, we can narrow down the bound of $r$ in Lemma 4.3 to loosen our assumption on fault tolerance.

Next, we prove the convergence of our algorithm. That is, Echo-CGC converges to the optimal point $w^{*}$ of the cost function $Q$. We prove the convergence under the assumption that $n\mu-(3+k^{*})fL>0$. Due to page limit, we present key proofs here, and the rest can be found in Appendix B.

Recall our definition of the conditional expectation $\mathbb{E}=\mathbb{E}_{\Xi^{t}}(\cdot\mid w^{t},G_{\mathcal{B}}^{t})$ introduced in Section 2.2. Before proving the main theorem, we introduce some preliminary lemmas.

The proof of Lemma 4.7 is based on Lemma 4.6 and the following prior results: Gumbel [10] and Hartley and David [12] proved that given identical means and variances $(\mu,\sigma^{2})$, the upper bound of the expectation of the largest random variable among $n$ independent random variables is $\mu+\frac{\sigma(n-1)}{\sqrt{2n-1}}$.

The proof of Lemma 4.8 is based on Lemma 4.6 and the following result: Papadatos [18] proved that for $n$ i.i.d. random variables $X_{1}\leq X_{2}\leq\dotsm\leq X_{n}$ with finite variance $\sigma^{2}$, the maximum variance of $X_{n}$ is bounded above by $n\sigma^{2}$.

Lemma 4.7 and Lemma 4.8 provide upper bounds on $\mathbb{E}\lVert\hat{g}_{j}^{t}\rVert$ and $\mathbb{E}\lVert\hat{g}_{j}^{t}\rVert^{2}$. These two bounds allow us to bound the impact of bogus gradients transmitted by a faulty node $j$. If $j$ transmitted an extreme gradient, it would be dropped by the CGC filter; otherwise, these two bounds essentially imply that the filtered gradient $\hat{g}_{j}^{t}$ has some nice property even if $j$ is faulty. For fault-free gradients, Lemma 4.6 provides a better bound.

We analyze the communication complexity of the Echo-CGC algorithm, and show that under suitable conditions, it effectively reduces communication complexity compared to prior algorithms [4, 11]. First consider a ball in $\mathbb{R}^{d}$ whose center is the true gradient $\nabla Q(w^{t})$:

where $r>0$ is the deviation ratio. For a slight abuse of notations, we abbreviate the ball as $B$. This should not be confused with $\mathcal{B}$, the set of Byzantine workers. We present only the main results, and the proofs can be found in Appendix C.

Given Lemma 4.11, we compute the probability that an arbitrary gradient $g_{j}^{t}$ is in the ball $B$. By Markov’s Inequality,

By Assumption 5, $\mathbb{E}\lVert g_{j}^{t}-\nabla Q(w^{t})\rVert^{2}\leq\sigma^{2}\lVert\nabla Q(w^{t})\rVert^{2}$, so we conclude that , where $p$ is the lower bound defined as $p=1-(1+2/r)^{2}\sigma^{2}$.

Denote $n_{B}=|\{j:g_{j}^{t}\in B\}|$ and $n^{*}$ as the number of workers that send the “echo message” in a round. By Lemma 4.11, $n^{*}\geq n_{B}-1$. Since each event $\{g_{j}^{t}\in B\}$ is independent and has a fixed probability, $n^{*}$ follows a Binomial distribution with success probability which is bounded below by $p$. Therefore,

For $n\gg 1$, we assume that $1/n\approx 0$. Also in practice, $d\gg n$, so the message complexity of each echo message (in $O(n)$ bits) is negligible compared to raw gradients (in $O(d)$ bits). Hence, the ratio of bit complexity of our algorithm and prior algorithms (e.g., [4, 11]) can be approximately bounded above as follows:

We denote the upper bound of ratio of reduced complexity to complexity of prior algorithms as $C=1-p=(1+2/r)^{2}\sigma^{2}$.

Analysis  By Equation (4.3) and Lemma 4.2, $C$ can be expressed as

where $x=f/n$ is the fault-tolerance factor.

As Equation (29) shows, the ratio $\mathcal{C}$ is related to four non-trivial variables: (i) bound of variance $\sigma\geq 0$; (ii) resilience $x=f/n$ satisfying the assumption in Lemma 4.3, i.e.,

(iii) constant $L/\mu$, which is determined by the cost function $Q$ and satisfies $0<L/\mu<1$ by Lemma 4.1; and (iv) number of workers $n>0$.