Basil: A Fast and Byzantine-Resilient Approach for Decentralized Training

To overcome the aforementioned limitations of prior graph-based Byzantine-robust algorithms, we propose Basil, an efficient Byzantine mitigation algorithm, that achieves Byzantine robustness in a decentralized setting by leveraging the sequential training over a logical ring. To highlight some of the benefits of Basil, Fig. 1(a) illustrates a sample result that demonstrates the performance gains and the cost reduction compared to the state-of-the-art Byzantine-robust algorithm UBAR that leverages the parallel training over a graph-based setting. We observe that Basil retains a higher accuracy than UBAR with an absolute value of $\sim 16\%$. Additionally, we note that while UBAR achieves its highest accuracy in $\sim 500$ rounds, Basil achieves UBAR’s highest accuracy in just $\sim 100$ rounds. This implies that for achieving UBAR’s highest accuracy, each client in Basil uses $5\times$ lesser computation and communication resources compared to that in UBAR confirming the gain attained from the sequential training nature of Basil.

In the following, we further highlight the key aspects and performance gains of Basil:

• •

  In Basil, the defense technique to filter out Byzantine nodes is a performance-based strategy, wherein each node evaluates a received set of models from its counterclockwise neighbors by using its own local dataset to select the best candidate.

• •

  We theoretically show that Basil for convex loss functions in the IID data setting has a linear convergence rate with respect to the product of the number of benign nodes and the total number of training rounds over the ring. Thus, our theoretical result demonstrates scalable performance for Basil with respect to the number of nodes.

• •

  We empirically demonstrate the superior performance of Basil compared to UBAR, the state-of-the-art Byzantine-resilient decentralized learning algorithm over graph, under different Byzantine attacks in the IID setting. Additionally, we study the performance of Basil and UBAR with respect to the wall-clock time in Appendix H showing that the training time for Basil is comparable to UBAR.

• •

  For extending the superior benefits of Basil to the scenario when data distribution is non-IID across devices, we propose Anonymous Cyclic Data Sharing (ACDS) to be applied on top of Basil. To the best of our knowledge, no prior decentralized Byzantine-robust algorithm has considered the scenario when the data distribution at the nodes is non-IID. ACDS allows each node to share a random fraction of its local non-sensitive dataset (e.g., landmarks images captured during tours) with all other nodes, while guaranteeing anonymity of the node identity. As highlighted in Section I-B, there are multiple real-world use cases where anonymous data sharing is sufficient to meet the privacy concerns of the users.

• •

  We experimentally demonstrate that using ACDS with only $5\%$ data sharing on top of Basil provides resiliency to Byzantine behaviors, unlike UBAR which diverges in the non-IID setting (Fig. 1(c)).

• •

  As the number of participating nodes in Basil scales, the increase in the overall latency of sequential training over the logical ring topology may limit the practicality of implementing Basil. Therefore, we propose a parallel extension of Basil, named Basil+, that provides further scalability by enabling Byzantine-robust parallel training across groups of logical rings, while retaining the performance gains of Basil through sequential training within each group.

Many Byzantine-robust strategies have been proposed recently for the distributed training setup (federated learning) where there is a central server to orchestrate the training process [12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22]. These Byzantine-robust optimization algorithms combine the gradients received by all workers using robust aggregation rules, to ensure that training is not impacted by malicious nodes. Some of these strategies [15, 16, 19, 17, 18] are based on distance-based approaches, while some others are based on performance-based criteria [12, 13, 14, 21]. The key idea in distance-based defense solutions is to filter the updates that are far from the average of the updates from the benign nodes. It has been shown that distance-based solutions are vulnerable to the sophisticated Hidden attack proposed in [23]. In this attack, Byzantine nodes could create gradients that are malicious but indistinguishable from benign gradients in distance. On the other hand, performance-based filtering strategies rely on having some auxiliary dataset at the server to evaluate the model received from each node.

Compared to the large number of Byzantine-robust training algorithms for distributed training in the presence of a central server, there have been only a few recent works on Byzantine resiliency in the decentralized training setup with no central coordinator. In particular, to address Byzantine failures in a decentralized training setup over a random graph under the scenario when the data distribution at the nodes is IID, the authors in [11] propose using a trimmed mean distance-based approach called BRIDGE to mitigate Byzantine nodes. However, the authors in [10] demonstrate that BRIDGE is defeated by the hidden attack proposed in [23]. To solve the limitations of the distance-based approaches in the decentralized setup, [10] proposes an algorithm called UBAR in which a combination of performance-based and distance-based stages are used to mitigate the Byzantine nodes, where the performance-based stage at a particular node leverages only its local dataset. As demonstrated numerically in [10], the combination of these two strategies allows UBAR to defeat the Byzantine attack proposed in [23]. However, UBAR is not suitable for the training over resource-constrained edge devices, as the training is carried out in parallel and nodes remain active all the time. In contrast, Basil is a fast and computationally efficient Byzantine-robust algorithm, which leverages a novel sequential, memory-assisted and performance-based criteria for training over a logical ring while filtering the Byzantine users.

Data heterogeneity in the decentralized setting has been studied in some recent works (e.g., [24]) in the absence of Byzantine nodes. In particular, the authors of TornadoAggregate [24] propose to cluster users into groups based on an algorithm called Group-BY-IID and CLUSTER where both use EMD (earth mover distance) that can approximately model the learning divergences between the models to complete the grouping. However, EMD function relies on having a publicly shared data at each node which can be collected similarly as in [25]. In particular, to improve training on non-IID data in federated learning, [25] proposed sharing of small portions of users’ data with the server. The parameter server pools the received subsets of data thus creating a small subset of the data distributed at the clients, which is then globally shared between all the nodes to make the data distribution close to IID. However, the aforementioned data sharing approach is considered insecure in scenarios where users are fine with sharing some of their datasets with each other but want to keep their identities anonymous, i.e., data shares should not reveal who the data owners are.

There are multiple real-world use cases where anonymous data sharing is sufficient for privacy concerns. For example, mobile users maybe fine with sharing some of their own text data, which does not contain any personal and sensitive information with others, as long as their personal identities remain anonymous. Another example is sharing of some non-private data (such as landmarks images) collected by a person with others. In this scenario, although data itself is not generated at the users, revealing the identity of the users can potentially leak private information such as personal interests, location, or travel history. Our proposed ACDS strategy is suitable for such scenarios as it guarantees that the owner identity of the shared data points are kept hidden.

As a final remark, we point out that for anonymous data sharing, [26] proposed an approach which is based on utilizing a secure sum operation along with anonymous ID assignment (AIDA). This involves computational operations at the nodes such as polynomial evaluations and some arithmetic operations such as modular operations. Thus, this algorithm may fail in the presence of Byzantine faults arising during these computations. Particularly, computation errors or software bugs can be present during the AIDA algorithm thus leading to the failure of anonymous ID assignment, or during the secure sum algorithm which can lead to distortion of the shared data.

We consider a decentralized learning setting in which a set $\mathcal{N}=\{1,\dots,N\}$ of $|\mathcal{N}|=N$ nodes collaboratively train a machine learning (ML) model $\mathbf{x}\in\mathbb{R}^{d}$, where $d$ is the model size, based on all the training data samples ${\cup_{n\in\mathcal{N}}}\mathcal{Z}_{n}$ that are generated and stored at these distributed nodes, where the size of each local dataset is $|\mathcal{Z}_{i}|=D_{i}$ data points.

In this work, we are motivated by the edge IoT setting, where users want to collaboratively train an ML model, in the absence of a centralized parameter server. The communication in this setting leverages the underlay communication fabric of the internet that connects any pair of nodes directly via overlay communication protocols. Specifically, we assume that there is no central parameter server, and consider the setup where the training process is carried out in a sequential fashion over a clockwise directed ring. Each node in this ring topology takes part in the training process when it receives the model from the previous counterclockwise node. In Section III-A, we propose a method in which nodes can consensually agree on a random ordering on a logical ring at the beginning of the training process, so that each node knows the logical ring positions of the other nodes. Therefore, without loss of generality, we assume for notation simplification that the indices of nodes in the ring are arranged in ascending order starting from node $1$. In this setup, each node can send its model update to any set of users in the network.

In this decentralized setting, an unknown $\beta$-proportion of nodes can be Byzantine, where $\beta\in(0,1)$, meaning they can send arbitrary and possibly malicious results to the other nodes. We denote $\mathcal{R}$ (with cardinality $|\mathcal{R}|=r$) and $\mathcal{B}$ (with cardinality $|\mathcal{B}|=b$) as the sets of benign nodes and Byzantine nodes, respectively. Furthermore, Byzantine nodes are uniformly distributed over the ring due to consensus-based random order agreement. Finally, we assume nodes can authenticate the source of a message, so no Byzantine node can forge its identity or create multiple fake ones [27].

Each node in the set $\mathcal{R}$ of benign nodes uses its own dataset to collaboratively train a shared model by solving the following optimization problem in the presence of Byzantine nodes:

where $\mathbf{x}$ is the optimization variable, and $f_{i}(\mathbf{x})$ is the expected loss function of node $i$ such that $f_{i}(\mathbf{x})=\mathbb{E}_{\zeta_{i}\sim\mathcal{P}_{i}}[l_{i}(\mathbf{x},\zeta_{i})]$. Here, $l_{i}(\mathbf{x},\zeta_{i})\in\mathbb{R}$ denotes the loss function for model parameter $\mathbf{x}\in\mathbb{R}^{d}$ for a given realization $\zeta_{i}$, which is generated from a distribution $\mathcal{P}_{i}$.

The general update rule in this decentralized setting is given as follows. At the $k$-th round, the current active node $i$ updates the global model according to:

where $\bar{\mathbf{x}}^{(i)}_{k}{=}\mathcal{A}(\mathbf{x}^{(j)}_{v},j\in\mathcal{N},v=1,\dots,k)$ is the selected model by node $i$ according to the underlying aggregation rule $\mathcal{A}$, $g(\bar{\mathbf{x}}^{(i)}_{k})$ is the stochastic gradient computed by node $i$ by using a random sample from its local dataset $\mathcal{Z}_{i}$, and $\eta_{k}^{(i)}$ is the learning rate in round $k$ used by node $i$.

Our goal is to design an algorithm for the decentralized training setup discussed earlier, while mitigating the impact of the Byzantine nodes. Towards achieving this goal, we propose Basil that is described next.

Our proposed Basil algorithm that is given in Algorithm 1 consists of two phases; initialization phase and training phase which are described below.
Phase 1: Order Agreement over the Logical Ring. Before the training starts, nodes consensually agree on their order on the ring by using the following simple steps. 1) All users first share their IDs with each other, and we assume WLOG that nodes’ IDs can be arranged in ascending order. 2) Each node locally generates the order permutation for the users’ IDs by using a pseudo random number generator (PRNG) initialized via a common seed (e.g., $N$). This ensures that all nodes will generate the same IDs’ order for the ring.

We now describe how the aggregation rule $\mathcal{A}_{\text{{Basil} }}$ in Basil, that node $i$ implements for obtaining the model $\bar{\mathbf{x}}^{(i)}_{k}$ for carrying out the update in (2), works. Node $i$ stores the $S$ latest models from its previous $S$ counterclockwise neighbors. As highlighted above, the reason for storing $S$ models is to make sure that each stored set of models at node $i$ contains at least one good model. When node $i$ is active, it implements our proposed performance-based criteria to pick the best model out of its $S$ stored models. In the following, we formally define our model selection criteria:

In practice, node $i$ can sample a mini-batch from its dataset and leverage it as validation data to test the performance (i.e., loss function value) of each model of the neighboring $S$ models, and set $\bar{\mathbf{x}}^{(i)}_{k}$ to be the model with the lowest loss among the $S$ stored models. As demonstrated in our experiments in Section VI, this practical mini-batch implementation of the Basil criteria in Definition 3 is sufficient to mitigate Byzantine nodes in the network, while achieving superior performance over state-of-the-art.

In the following, we characterize the communication, computation, and storage costs of Basil.

The complexity of prior graph-based Byzantine-robust decentralized algorithms UBAR [10] and Bridge [11] is $\mathcal{O}({H}_{i}d)$, where ${H}_{i}$ is the number of neighbours (e.g., connectivity parameter) of node $i$ on the graph. So we conclude that Basil maintains the same per round complexity as Bridge and UBAR, but with higher performance as we will show in Section VI.

The costs in Proposition 1 can be reduced by relaxing the connectivity parameter $S$ to $S<b+1$ while guaranteeing the success of Basil (benign subgraph connectivity) with high probability, as formally presented in the following.

We derive the convergence guarantees of Basil under the following standard assumptions.

Assumption 1 (IID data distribution). Local dataset $\mathcal{Z}_{i}$ at node $i$ consists of IID data samples from a distribution $\mathcal{P}_{i}$, where $\mathcal{P}_{i}=\mathcal{P}$ for $i\in\mathcal{R}$. In other words, $f_{i}(\mathbf{x})=\mathbb{E}_{\zeta_{i}\sim\mathcal{P}_{i}}[l(\mathbf{x},\zeta_{i})]=\mathbb{E}_{\zeta_{j}\sim\mathcal{P}_{j}}[l(\mathbf{x},\zeta_{i}))]=f_{j}(\mathbf{x})\,\forall i,j\in\mathcal{R}$. Hence, the global loss function $f(\mathbf{x})=\mathbb{E}_{\zeta_{i}\sim\mathcal{P}_{i}}[l(\mathbf{x},\zeta_{i})]$.

Assumption 2 (Bounded variance). Stochastic gradient $g_{i}(\mathbf{x})$ is unbiased and variance bounded, i.e., $\mathbb{E}_{\mathcal{P}_{i}}[g_{i}(\mathbf{x})]=\nabla f_{i}(\mathbf{x})=\nabla f(\mathbf{x})$, and $\mathbb{E}_{\mathcal{P}_{i}}||g_{i}(\mathbf{x})-\nabla f_{i}(\mathbf{x})||^{2}\leq\sigma^{2}$, where $g_{i}({\mathbf{x}})$ is the stochastic gradient computed by node $i$ by using a random sample $\zeta_{i}$ from its local dataset $\mathcal{Z}_{i}$.

Assumption 3 (Smoothness). The loss functions $f_{i}^{\prime}s$ are L-smooth and twice differentiable, i.e., for any $\mathbf{x}\in\mathbb{R}^{d}$, we have $||\nabla^{2}f_{i}(\mathbf{x})||_{2}\leq L$.

Let $b^{i}$ be the number of counterclockwise Byzantine neighbors of node $i$. We divide the set of stored models $\mathcal{N}_{k}^{i}$ at node $i$ in the $k$-th round into two sets. The first set $\mathcal{G}^{i}_{k}=\{\mathbf{y}_{1},\dots,\mathbf{y}_{r^{i}}\}$ contains the benign models, where $r^{i}=S-b^{i}$. We consider scenarios with $S=b+1$, where $b$ is the total number of Byzantine nodes in the network. Without loss of generality, we assume the models in this set are arranged such that the first model is the closest benign node in the neighborhood of node $i$, while the last model is the farthest node. Similarly, we define the second set $\mathcal{B}_{k}^{i}$ to be the set of models from the counterclockwise Byzantine neighbors of node $i$ such that $\mathcal{B}_{k}^{i}\cup\mathcal{G}_{k}^{i}=\mathcal{N}_{k}^{i}$.

Leveraging the results in Theorem 1 and based on the discussion in Remark 1, we prove the linear convergence rate for Basil, under the additional assumption of convexity of the loss functions.

To extend Basil to be robust against software/hardware faults in the non-IID setting, i.e., when the local dataset $\mathcal{Z}_{i}$ at node $i$ consists of data samples from a distribution $\mathcal{P}_{i}$ with $\mathcal{P}_{i}\neq\mathcal{P}_{j}$ for $i,j\in\mathcal{N}$ and $i\neq j$, we present our Anonymous Cyclic Data Sharing algorithm (ACDS) in the following section.

The ACDS procedure has three phases which are formally described next. The overall algorithm has been summarized in Algorithm 2 and illustrated in Fig. 3.

Phase 1: Initialization. ACDS starts by first clustering the $N$ nodes into $G$ groups of rings, where the set of nodes in each group $g\in[G]$ is denoted by $\mathcal{N}_{g}=\{1_{g},\dots,n_{g}\}$. Here, node $1_{g}$ is the starting node in ring $g$, and without loss of generality, we assume all groups have the same size $n=\frac{N}{G}$. Node $i_{g}\in\mathcal{N}_{g}$ for $g\in[G]$ divides its dataset $\mathcal{Z}_{i_{g}}$ into sensitive ($\mathcal{Z}_{i_{g}}^{s}$) and non-sensitive ($\mathcal{Z}_{i_{g}}^{ns}$) portions, which can be done during the data processing phase by each node. Then, for a given hyperparameter $\alpha\in(0,1)$, each node selects $\alpha D$ points from its local non-sensitive dataset at random, where $|\mathcal{Z}_{i_{g}}|=D$, and then partitions these data points into $H$ batches denoted by $\{c_{i_{g}}^{1},\dots,c_{i_{g}}^{H}\}$, where each batch has $M=\frac{\alpha D}{H}$ data points.

Phase 2: Within Group Anonymous Data Sharing. In this phase, for $g\in[G]$, the set of nodes $\mathcal{N}_{g}$ in group $g$ anonymously share their data batches $\{c^{j}_{1_{g}},\dots,c^{j}_{n_{g}}\}_{j\in[H]}$ with each other. The anonymous data sharing within group $g$ takes $H+1$ rounds. In particular, as shown in Fig. 3, within group $g$ and during the first round $h=1$, node $1_{g}$ sends the first batch $c^{1}_{1_{g}}$ to its clockwise neighbor, node $2_{g}$. Node $2_{g}$ then stores the received batch. After that, node $2_{g}$ augments the received batch with its own first batch $c^{1}_{2_{g}}$ and shuffles them together before sending them to node $3_{g}$. More generally, in the intra-group cyclic sharing over the ring, node $i_{g}$ stores the received set of shuffled data points from batches $\{c^{1}_{1_{g}},\dots,c^{1}_{(i-1)_{g}}\}$ from its counterclockwise neighbor node $(i-1)_{g}$. Then, it adds its own batch $c^{1}_{i_{g}}$ to the received set of data points, and shuffles them together, before sending the combined and shuffled dataset to node $(i+1)_{g}$.

For round $1<h\leq H$, as shown in Fig. 3-(round 2), node $1_{g}$ has the data points from the set of batches $\{c^{(h-1)}_{1_{g}},\dots,c^{(h-1)}_{n_{g}}\}$ which were received from node $n_{g}$ at the end of round $(h-1)$. It first removes its old batch of data points $c^{(h-1)}_{1_{g}}$ and then stores the remaining set of data points. After that, it adds its $h$-th batch, $c^{(h)}_{1_{g}}$ to this remaining set, and then shuffles the entire set of data points in the new set of batches $\{c^{h}_{1_{g}},c^{{h-1}}_{2_{g}},\dots,c^{h-1}_{n_{g}}\}$, before sending them to node $2_{g}$. More generally, in the $h$-th round for $1<h\leq H$, node $i_{g}$ first removes its batch $c^{h-1}_{i_{g}}$ from the received set of batches and then stores the set of remaining data points. Thereafter, node $i_{g}$ adds its $c^{h}_{i_{g}}$ to the set $\{c^{h}_{1_{g}},\ldots,c^{h}_{(i-1)_{g}},c^{h-1}_{(i)_{g}},\dots,c^{h-1}_{n_{g}}\}\backslash\{c^{h-1}_{i_{g}}\}$, and then shuffles the set of data points in the new set of batches $\{c^{h}_{1_{g}},\dots,c^{h}_{i_{g}},c^{h-1}_{(i+1)_{g}},\dots,c^{h-1}_{n_{g}}\}$ before sending them to node $(i+1)_{g}$.

After $H$ intra-group communication iterations within each group as described above, all nodes within each group have completely shared their $H-1$ batches with each other. However, due to the sequential nature of unicast communications, some nodes are still missing the batches shared by some clients in the $H^{th}$ round. For instance, in Fig. 3, after the completion of the second round, node $2_{g}$ is missing the last batches $c^{2}_{3_{g}}$ and $c^{2}_{4_{g}}$. Therefore, we propose a final dummy round, in which we repeat the same procedure adopted in rounds $1<h\leq H$, but with the following slight modification: node $i_{g}$ replaces its batch $c^{H}_{i_{g}}$ with a dummy batch $c^{\text{dummy}}_{i_{g}}$. This dummy batch could be a batch of public data points that share the same feature space that is used in the learning task. This completes the anonymous cyclic data sharing within group $g\in[G]$.

Phase 3: Global Sharing. For each $g\in[G]$, node $1_{g}$ shares the dataset $\{c^{j}_{1_{g}},\dots,c^{j}_{n_{g}}\}_{j\in[H]}$, which it has gathered in phase 2, with all other nodes in the other groups.

We note that implementation of ACDS only needs to be done once before training. As we demonstrate later in Section VI, the one-time overhead of the ACDS algorithm dramatically improves convergence performance when data is non-IID. In the following proposition, we describe the communication cost/time of ACDS.

The proof of Proposition 3 is presented in Appendix D.

In the following, we discuss the anonymity guarantees of ACDS.

In the first round of the scheme, node $2_{g}$ will know that the source of the received batch $c^{1}_{1_{g}}$ is node $1_{g}$. Similarly and more general, node $i_{g}$ will know that each data point in the received set of batches $\{c^{1}_{1_{g}},\dots,c^{1}_{(i-1)_{g}}\}$ is generated by one of the previous $i-1$ counterclockwise neighbors. However, in the next $H-1$ rounds, each received data point by any node will be equally likely generated from any one of the remaining $n-1$ nodes in this group. Hence, the size of the candidate pool from which each node could take a guess for the owner of each data point is small specially for the first set of nodes in the ring. In order to provide anonymity for the entire data and decrease the risk in the first round of the ACDS scheme, the size of the batch can be reduced to just one data point. Therefore, in the first round node $2_{g}$ will only know one data point from node $1_{g}$. This comes on the expense of increasing the number of rounds. Another key consideration is that the number of nodes in each group trades the level of anonymity with the communication cost. In particular, the communication cost per node in the ACDS algorithm is $\mathcal{O}(n)$, while the anonymity level, which we measure by the number of possible candidates for a given data point, is $(n-1)$. Therefore, increasing $n$, i.e., decreasing the number of groups $G$, will decrease the communication cost but increase the anonymity level.

At a high level, Basil+ divides nodes into $G$ groups such that each group in parallel performs sequential training over a logical ring using Basil algorithm. After $\tau$ rounds of sequential training within each group, a robust circular aggregation strategy is implemented to have a robust average model from all the groups. Following the robust circular aggregation stage, a final multicasting step is implemented such that each group can use the resulting robust average model. This entire process is repeated for $K$ global rounds.

We now formalize the execution of Basil+ through the following four stages.

We compare between the training time of Basil and Basil+ in the following proposition.

In the following section, we discuss the random clustering method used in the Stage 1 of Basil+.

In practice, nodes can agree on a random clustering by using similar approach as in Section III-A by the following simple steps. 1) All nodes first share their IDs with each other, and we assume WLOG that nodes’ IDs can be arranged in ascending order, and Byzantine nodes cannot forge their identities or create multiple fake ones [27]. 2) Each node locally random splits the nodes into $G$ subsets by using a pseudo random number generator (PRNG) initialized via a common seed (e.g., N). This ensures that all nodes will generate the set of nodes. To know the nodes order within each local group, the method in Section III-A can be used.

We will consider different scenarios for the connectivity parameter $S$ while evaluating the success of Basil+. Case 1: $S=\min(n-1,b+1)$ We set the connectivity parameter for Basil+ to $S=\min(n-1,b+1)$. By setting $S=b+1$ when $n>b+1$, this ensures the connectivity of each ring (after removing the Byzantine nodes) along with the global ring. On the other hand, by setting $S=n-1$ if $n\leq b+1$, Basil+ would only fail if at least one group has a number of Byzantine nodes of $n$ or $n-1$. We define $B_{j}$ to be the failure event in which the number of Byzantine nodes in a given group is $n$ or $n-1$. The failure event $B_{j}$ follows a Hypergeometric distribution with parameters $(N,b,n)$, where $N$, $b$, and $n$ are the total number of nodes, total number of Byzantine nodes, number of nodes in each group, respectively. The probability of failure is given as follows

where (a) follows from the union bound.

In order to further illustrate the impact of choosing the group size $n$ when setting $S=n-1$ on the probability of failure given in (17), we consider the following numerical examples. Let the total number of nodes in the system be $N=100$, where $b=33$ of them are Byzantine. By setting $n=20$ nodes in each group, the probability in (17) turns out to be $\sim{5\times 10^{-10}}$, which is negligible. For the case when $n=10$, the probability of failure becomes $\sim{1.2\times 10^{-4}}$, which remains reasonably small.

Case 2: $S<n-1$ Similar to the failure analysis of Basil given in Proposition 2, we relax the connectivity parameter $S$ as stated in the following proposition.

The proof of Proposition 5 is presented in Appendix F.

Furthermore, we can observe in Fig. 5(b) that Basil with $\alpha=0$ gives low performance. This confirms that non-IID data distribution degraded the convergence behavior. For UBAR, the performance is completely degraded, since in UBAR each node selects the set of models which gives a lower loss than its own local model, before using them in the update rule. Since performance-based is not meaningful in this setting, each node might end up only with its own model. Hence, the model of each node does not completely propagate over the graph, as also demonstrated in Fig. 5(b), where UBAR fails completely. This is different from the ring setting, where the model is propagated over the ring.

In Fig. 5, we showed that UBAR performs quite poorly for non-IID data setting, when no data is shared among the clients. We note that achieving anonymity in data sharing in graph based decentralized learning in general and UBAR in particular is an open problem. Nevertheless, in Fig. 6, we further show that even $5\%$ data sharing is done in UBAR, performance remains quite low in comparison to Basil+ACDS.

Now, we compute the communication cost overhead due to leveraging ACDS for the experiments associated with Fig. 5. By considering the setting discussed in Remark 5 for ACDS with $G=4$ groups for data sharing and each node sharing $\alpha=5\%$ fraction of its local dataset, we can see from Fig. 5 that Basil takes $500$ rounds to get $\sim 50\%$ test accuracy. Hence, given that the model used in this section is of size $3.6$ Mbits ($117706$ trainable parameters each represented by $32$ bits), the communication cost overhead resulting from using ACDS for data sharing is only $4\%$.

Fig. 7(a) demonstrates that, unlike UBAR, Basil performance scales with the number of nodes $N$. This is because in any given round, the sequential training over the logical ring topology accumulates SGD updates of clients along the logical ring, as opposed to parallel training over the graph topology in which an update from any given node only propagates to its neighbors. Hence, Basil has better accuracy than UBAR. Additionally, as described in Section V, one can also leverage Basil+ to achieve further scalability for large $N$ by parallelizing Basil. We provide the experimental evaluations corresponding to Basil+ in Section VI-B.

To study the effect of different number of Byzantine nodes in the system, we conduct experiments with different $b$. Fig. 7(b) demonstrates that Basil is quite robust to different number of Byzantine nodes.

Fig. 7(c) demonstrates the impact of the connectivity parameter $S$. Interestingly, the convergence rate decreases as $S$ increases. We posit that due to the noisy SGD based training process, the closest benign model is not always selected, resulting in loss of some intermediate update steps. However, decreasing $S$ too much results in larger increase in the connectivity failure probability of Basil. For example, the upper bound on the failure probability when $S=6$ is less than $0.09$. However, for an extremely low value of $S=2$, we observed consistent failure across all simulation trials, as also illustrated in Fig. 7(c). Hence, a careful choice of $S$ is important.

Finally, to study the relationship between privacy and accuracy when Basil is used alongside ACDS, we carry out numerical analysis by varying $\alpha$ in the non-IID setting described previously. Fig. 7(d) demonstrates that as $\alpha$ is increased, i.e., as the amount of shared data is increased, the convergence rate increases as well. Furthermore, we emphasize that even having $\alpha=0.5\%$ gives reasonable performance when data is non-IID, unlike UBAR which fails completely.

In this section, we demonstrate the achievable gains of Basil+ in terms of its scalability, Byzantine robustness, and superior performance over UBAR.