Adaptive Transmission Scheduling in Wireless Networks for Asynchronous Federated Learning

Here, we introduce typical learning strategies to update the central and local parameters in asynchronous FL [15, 14, 13, 12], which address the challenges due to the stragglers. To this end, we consider one access point (AP) that plays a role of a central server in FL and $U$ edge devices. The set of edge devices is defined as $\mathcal{U}=\{1,2,...,U\}$. In asynchronous FL, an artificial neural network (ANN) model composed of multiple parameters is trained in a distributed manner to minimize an empirical loss function $l({\mathbf{w}})$, where ${\mathbf{w}}$ denotes the parameters of the ANN. Asynchronous FL proceeds over a discrete-time horizon composed of multiple rounds. The set of rounds is defined as $\mathcal{T}=\{1,2,...\}$ and the index of rounds is denoted by $t$. Then, we can formally define the problem of asynchronous FL with a given local loss function at device $u$, $f_{u}$, as follows:

where $K_{u}$ denotes the number of data samples of device $u$,¹¹1For simple presentation, we omit the word “edge” from edge device in the rest of the paper. $K=\sum_{u=1}^{U}K_{u}$, and $f_{u}({\mathbf{w}},k)$ is an empirical local loss function defined by the parameters ${\mathbf{w}}$ at $k$-th data sample of device $u$. To solve this problem, in asynchronous FL, every device trains parameters by using its locally available data in an online manner for efficient learning on each device and avoiding too much pileup of the local data. Then, partial devices are scheduled to transmit the trained parameters to the AP. By using the received parameters from the scheduled devices, the AP updates its parameters. We call the parameters at the AP central parameters and those at each device local parameters. The details of how to update the local and central parameters in asynchronous FL are provided in the following.

We consider a WDLN consisting of one AP and $U$ devices that cooperatively operate FL over the discrete-time horizon composed of multiple rounds that have an identical time duration. In the WDLN, the local data stochastically and continually arrives at each device and the device trains its local parameters in each round by using the arrived data. Due to the scarce radio resources of the WDLN, it is impractical that all devices transmit their local parameters to the AP simultaneously. Hence, FL in the WDLN becomes asynchronous FL and the devices who will transmit their local parameters in each round should be carefully scheduled by the AP for training the central parameters effectively.

We now propose an asynchronous FL procedure for the WDLN in which the central and local parameter update strategies introduced in the previous subsection are adopted. In the procedure, the characteristics of the WDLN, such as time-varying channel conditions and system bandwidth, are considered as well. This allows us to implement asynchronous FL in the WDLN while addressing the challenges due to the scarcity of radio resources in the WDLN. We denote the channel gain of device $u$ in round $t$ by $h^{t}_{u}$. We assume that the channel gain of device $u$ in each round follows a parameterized distribution and denote the corresponding system parameters by ${\boldsymbol{\uptheta}}_{u}^{C}$. For example, for Rician fading channels, ${\boldsymbol{\uptheta}}_{u}^{C}$ represents the parameters of the Rician distribution, $\Omega$ and $K$. The vector of the channel gain in round $t$ is defined by ${\mathbf{h}}^{t}=\{h_{u}^{t}\}_{u\in\mathcal{U}}$.

In the procedure, each round $t$ is composed of three phases: transmission scheduling phase, local parameter update phase, and parameter aggregation phase. In the transmission scheduling phase, the AP observes the state information of the devices which can be used to determine the transmission scheduling. In specific, we define the state information in round $t$ as ${\mathbf{s}}^{t}=({\mathbf{h}}^{t},{\mathbf{n}}^{t})$, where ${\mathbf{n}}^{t}=\{n_{u}^{t}\}_{u\in\mathcal{U}}$ is the vector of the number of aggregated samples of each device from its latest successful local update transmission to the beginning of round $t$. It is worth noting that in the following sections, we consider a partially observable WDLN as well, in which the AP can observe the channel gain ${\mathbf{h}}^{t}$ only, to address more restrictive environments in practice. Then, the AP determines the transmission scheduling for asynchronous FL based on the state information. In the local parameter update phase, the AP broadcasts its central parameters ${\mathbf{w}}^{t}$ to all devices. We do not consider the transmission failure for ${\mathbf{w}}^{t}$ at each device as in the related works [9, 7, 8, 10], i.e., all devices receive ${\mathbf{w}}^{t}$ successfully. This is reasonable because the probability of the transmission failure for ${\mathbf{w}}^{t}$ is small in practice since the AP can reliably broadcast the central parameters by using much larger transmission power than that of the devices and robust transmission methods considering the worst-case device.²²2It is worth noting that this assumption is only to avoid unnecessarily complicated modeling. Besides, our proposed method can be easily used even without this assumption as well by allowing each device to maintain its previous local parameters when its reception of the central parameters is failed. Each device $u$ replaces its local parameters with the received central parameters (i.e., ${\mathbf{w}}_{u}^{t}={\mathbf{w}}^{t}$). Then, it trains the parameters using its local data samples which have arrived since the local parameter update phase in the previous round and calculates the local update $\Delta_{u}^{t}$ in (4) as described in Section II-A1. Finally, in the parameter aggregation phase, each scheduled device transmits its local update to the AP. Then, the AP updates the central parameters by averaging them as in (7). In the following, we describe the asynchronous FL procedure in the WDLN in more detail.

First, the AP schedules the devices to transmit their local updates based on the state information to effectively utilize the limited radio resources in each round. We define the maximum number of devices that can be scheduled in each round as $W$. It is worth noting that $W$ is restricted by the bandwidth of the WDLN and typically much smaller than the number of devices $U$ (i.e., $W\ll U$). We define a transmission scheduling indicator of device $u$ in round $t$ as $a_{u}^{t}\in\{0,1\}$, where 1 represents device $u$ is scheduled to transmit its gradient in round $t$ and 0 represents it is not. The vector of the transmission scheduling indicators in round $t$ is defined as ${\mathbf{a}}^{t}=\{a_{u}^{t}\}_{u\in\mathcal{U}}$. In round $t$, the AP schedules the devices to transmit their local updates satisfying the following constraint:

We then define the successful transmission indicator of the local update of device $u$ in round $t$ as a Bernoulli random variable $x_{u}^{t}$, where 1 represents the successful transmission and 0 represents the transmission failure. Since the probability of the successful transmission of device $u$ in round $t$ is determined according to the channel gain of device $u$, we can model it by using the approximation of the packet error rate (PER) with the given signal-to-interference-noise ratio (SINR) [23, 24]. Moreover, we can use the PER provided in [25, 26] to model it considering the HARQ or ARQ retransmissions. For example, the PER of an uncoded packet can be given by $\mathbb{P}[x=0|\Phi]=1-(1-b(\Phi))^{n}$, where $\Phi$ is the SINR with the channel gain $h$, $b(\cdot)$ is the bit error rate for the given SINR, and $n$ is the packet length in bits. We refer the readers to [24] for more examples of the PER approximations with coding. The vector of the successful transmission indicators in round $t$ is defined as ${\mathbf{x}}^{t}=\{x_{u}^{t}\}_{u\in\mathcal{U}}$.

We assume the number of the samples that have arrived at device $u$ during round $t$, $m_{u}^{t}$, is an independently and identically distributed (i.i.d.) Poisson random variable with a system parameter $\theta_{u}$.³³3It is worth noting that the assumption of Poisson data arrivals is only for simple implementation of the Bayesian approach in our proposed method. We can easily generalize our proposed method for any other i.i.d. data arrivals by using sampling algorithms such as Gibbs sampling. At the end of round $t$ (i.e., at the beginning of round $t+1$), the number of aggregated samples of device $u$ from the latest successful local update transmission to the beginning of round $t+1$,⁴⁴4For brevity, to denote $n_{u}^{t+1}$, we simply write “the number of aggregated samples of device $u$ at the beginning of round $t+1$” in the rest of this paper. $n_{u}^{t+1}$, and the total number of samples from device $u$ used for the central updates before the beginning of round $t+1$, $N_{u}^{t+1}$, should be calculated depending on the local update transmission as follows:

and

With the scheduling and successful transmission indicators, $a_{u}^{t}$’s and $x_{u}^{t}$’s, we can rewrite the equation for the centralized parameter update at the AP in (7) as follows:

We summarize the asynchronous FL procedure in the WDLN in Algorithm 1 and illustrate in Fig. 1.

We analyze the convergence of the asynchronous FL procedure proposed in the previous subsection. We first introduce some definitions and an assumption on the objective function of asynchronous FL in (1) for the analysis.

It is worth noting that this assumption is a typical one used in literature [27, 15, 13]. With this assumption, we can show the convergence of the asynchronous FL procedure in the WDLN in the following theorem.

Theorem 1 takes consideration of the asynchronous FL procedure in the WDLN compared with the convergence analysis of asynchronous FL in [13]. As a result, the theorem implies that the faster convergence can be expected as a larger expected number of successful transmissions in the WDLN, $\sum_{u\in\mathcal{U}}a_{u}^{t}\mathbb{P}[x_{u}^{t}=1]$. Besides, it theoretically shows the convergence of the asynchronous FL procedure in the WDLN as $T\rightarrow\infty$ if the expected number of successful transmissions is non-zero since $0<2\xi\underline{\eta}\epsilon^{\prime}<1$. However, various aspects in learning, such as a learning speed and a robustness to stragglers, are not directly shown in this theorem, and they may appear differently according to transmission scheduling strategies used in the asynchronous FL procedure as will be shown in Section V.

Before developing algorithms, we first define a parameterized Markov decision process (MDP) based on the ALS problem in the previous section. Since the system uncertainties in the WDLN, such as the channel statistics and the arrival rates of the samples, depend on the system parameters ${\boldsymbol{\uptheta}}$ in (17), the MDP is parameterized by system parameters ${\boldsymbol{\uptheta}}$. In specific, it is defined as $M_{\boldsymbol{\uptheta}}=(\mathcal{S},\mathcal{A},r^{\boldsymbol{\uptheta}},P^{\boldsymbol{\uptheta}})$, where $\mathcal{S}$ and $\mathcal{A}$ are the state space and action space of the ALS problem, respectively, $r^{\boldsymbol{\uptheta}}$ is the reward function, and $P^{\boldsymbol{\uptheta}}$ is the transition probability such that $P^{\boldsymbol{\uptheta}}({\mathbf{s}}^{\prime}|{\mathbf{s}},{\mathbf{a}})=\mathbb{P}({\mathbf{s}}^{t+1}={\mathbf{s}}^{\prime}|{\mathbf{s}}^{t}={\mathbf{s}},{\mathbf{a}}^{t}={\mathbf{a}},{\boldsymbol{\uptheta}})$. We define the reward function $r^{\boldsymbol{\uptheta}}$ as the effectivity score in (15). For theoretical results, we assume that the state space $\mathcal{S}$ is finite and the reward function is bounded as $r^{\boldsymbol{\uptheta}}:\mathcal{S}\times\mathcal{A}\rightarrow[0,1]$.⁵⁵5We can easily implement such a system by quantizing the channel gain and truncating $n_{u}^{t}$ and $m_{u}^{t}$ with a large constant. Then, the state space becomes finite and we can use the reward function as the normalized version of $F({\mathbf{s}},{\mathbf{a}})$ with the maximum expected reward that can be derived from the constant. In practice, this system is more realistic to be implemented since such variables in a real system are typically finite. The average reward per round of a stationary policy $\pi$ is defined as

The optimal average reward per round $J^{*}({\boldsymbol{\uptheta}})=\max_{\pi}J_{\pi}({\boldsymbol{\uptheta}})$ satisfies the Bellman equation:

where $v({\mathbf{s}},{\boldsymbol{\uptheta}})$ is the value function at state ${\mathbf{s}}$. We can define the corresponding optimal policy, $\pi^{*}({\mathbf{s}},{\boldsymbol{\uptheta}})$, as the maximizer of the above optimization. Then, the policy $\pi^{*}({\mathbf{s}},{\boldsymbol{\uptheta}}^{*})$ becomes the optimal solution to the problem in (16).

We now develop ALSA-PI to solve the ALS problem when ${\boldsymbol{\uptheta}}^{*}$ is perfectly known at the AP in advance. Typically, the true system parameters ${\boldsymbol{\uptheta}}^{*}$ is unknown, but here we introduce ALSA-PI since it will be used to develop BALSA in Section IV-C. Besides, in some cases, ${\boldsymbol{\uptheta}}^{*}$ might be precisely estimated by using past experiences. Since the true system parameters ${\boldsymbol{\uptheta}}^{*}$ are known, the problem in (16) can be solved by finding the optimal policy to ${\boldsymbol{\uptheta}}^{*}$. However, in general, computing the optimal policy $\pi^{*}$ to given parameters requires a large computational complexity which exponentially increases with the number of devices, which is often called a curse of dimensionality. Hence, even if the true system parameters ${\boldsymbol{\uptheta}}^{*}$ are perfectly known in advance, it is hard to compute the corresponding optimal policy $\pi^{*}({\mathbf{s}},{\boldsymbol{\uptheta}}^{*})$. However, for the ALS problem, a greedy policy, which myopically chooses the action to maximize the expected reward in the current round, becomes the optimal policy as the following theorem.

With this theorem, we can easily develop ALSA-PI by adopting the greedy policy with the known system parameters ${\boldsymbol{\uptheta}}^{*}$ thanks to the structure of the reward. This also implies that we can significantly reduce the computational complexity to solve the ALS problem with the known parameters compared with the DP methods that are typical ones to solve MDPs. The computational complexity will be analyzed in Section IV-E.

In round $t$, the greedy policy for given system parameters ${\boldsymbol{\uptheta}}$ chooses the action by solving the following optimization problem:

where $\mathbb{E}[\cdot|{\boldsymbol{\uptheta}}]$ represents the expectation over the probability distribution of the given system parameters ${\boldsymbol{\uptheta}}$. With the scheduling and successful transmission indicators, $a_{u}^{t}$’s and $x_{u}^{t}$’s, we rearrange the objective function in (15) as

Then, we can reformulate the problem as

since the last term in (21) does not depend on both transmission scheduling indicator ${\mathbf{a}}$ and uncertainty on the successful transmission ${\mathbf{x}}^{t}$. In addition, the conditional expectation in the problem depends only on the system parameters for the arrival rates of the data samples, ${\boldsymbol{\uptheta}}^{P}=\{\theta_{1}^{P},...\theta_{U}^{P}\}$, because $x_{u}^{t}$ is solely determined by the current channel gain. To solve the problem, the AP estimates the expected number of samples of each device $u$, $\mathbb{E}[x_{u}^{t}(n_{u}^{t}+m_{u}^{t})|{\boldsymbol{\uptheta}}^{P}]$, by using its channel gain $h_{u}^{t}$ and the system parameter $\theta_{u}^{P}$. Let the devices be sorted in descending order of their expected number of samples $\mathbb{E}[x_{u}^{t}(n_{u}^{t}+m_{u}^{t})|{\boldsymbol{\uptheta}}^{P}]$ and indexed by $(1),(2),...,(U)$. Then, the greedy policy in round $t$ is easily obtained as

In ALSA-PI, the AP schedules the devices according to the greedy policy with the true system parameter ${\boldsymbol{\uptheta}}^{*,P}$ (i.e., $\pi^{g}$ in (23) with ${\boldsymbol{\uptheta}}^{*,P}$) in each round.

We now develop BALSA that solves the parameterized MDP without requiring any a priori information. To this end, it adopts a Bayesian approach to learn the unknown system parameters ${\boldsymbol{\uptheta}}$. We define the policy determined by BALSA as $\phi=(\phi^{1},\phi^{2},...)$ each of which $\phi^{t}({\mathbf{g}}^{t})$ chooses an action according to the history of states, actions, and rewards, ${\mathbf{g}}^{t}=({\mathbf{s}}^{1},...,{\mathbf{s}}^{t},{\mathbf{a}}^{1},...,{\mathbf{a}}^{t-1},r^{1},...,r^{t-1})$. To apply the Bayesian approach, we assume that the system parameters that belong to ${\boldsymbol{\uptheta}}$ are independent to each other. We denote a prior distribution for the system parameters ${\boldsymbol{\uptheta}}$ by $\mu^{1}$. For the prior distribution, the non-informative prior can be typically used if there is no prior information about the system parameters. On the other hand, if there is any prior information, then it can be defined by considering the information. For example, if we know that a parameter belongs to a certain interval, then the prior distribution can be defined as a truncated distribution whose probability measure outside of the interval is zero. We then define the Bayesian regret of BALSA up to time $T$ as

where the expectation in the above equation is over the prior distribution $\mu^{1}$ and the randomness in state transitions. This Bayesian regret has been widely used in the literature of Bayesian RL as a metric to quantify the performance of a learning algorithm [32, 33, 34].

In BALSA, the system uncertainties are estimated by a Bayesian approach. Formally, in the Bayesian approach, the posterior distribution of ${\boldsymbol{\uptheta}}$ in round $t$, which is denoted by $\mu^{t}$, is updated by using the observed information about ${\boldsymbol{\uptheta}}$ according to the Bayes’ rule. Then, the posterior distribution is used to estimate or sample the system parameters for the algorithm. We will describe how the posterior distribution is updated in detail later. We define the number of visits to any state-action pair $({\mathbf{s}},{\mathbf{a}})$ before time $t$ as

where $|\cdot|$ denotes the cardinality of the set.

BALSA operates in stages each of which is composed of multiple rounds. We denote the start time of stage $k$ of BALSA by $t_{k}$ and the length of stage $k$ by $T_{k}=t_{k+1}-t_{k}$. With the convention, we set $T_{0}=1$. Stage $k$ ends and the next stage starts if $t>t_{k}+T_{k-1}$ or $M^{t}({\mathbf{s}},{\mathbf{a}})>2M^{t_{k}}({\mathbf{s}},{\mathbf{a}})$ for some $({\mathbf{s}},{\mathbf{a}})\in\mathcal{S}\times\mathcal{A}$. This balances the trade-off between exploration and exploitation in BALSA. Thus, the start time of stage $k+1$ is given by

and $t_{1}=1$. This stopping criterion allows us to bound the number of stages over $T$ rounds. At the beginning of stage $k$, system parameters ${\boldsymbol{\uptheta}}_{k}$ are sampled from the posterior distribution $\mu^{t_{k}}$. Then, the action is chosen by the optimal policy corresponding to the sampled system parameter ${\boldsymbol{\uptheta}}_{k}$, $\pi^{*}({\mathbf{s}},{\boldsymbol{\uptheta}}_{k})$, until the stage ends. It is worth noting that this posterior sampling procedure, in which the system parameters sampled from the posterior distribution are used for choosing actions, has been widely applied to address the exploration-exploitation dilemma in RL [32, 33, 34, 35]. Since the greedy policy, $\pi^{g}({\mathbf{s}},{\boldsymbol{\uptheta}}_{k})$, is the optimal policy to the ALS problem as shown in the previous subsection, we can easily implement BALSA by using it. Besides, this makes the AP not have to estimate the posterior distribution of the parameters for the channel gains, ${\boldsymbol{\uptheta}}^{C}=\{{\boldsymbol{\uptheta}}_{1}^{C},...,{\boldsymbol{\uptheta}}_{U}^{C}\}$, since the greedy policy do not use them.

Here, we describe BALSA in more detail. In round $t$, the AP observes the states ${\mathbf{s}}^{t}=({\mathbf{h}}^{t},{\mathbf{n}}^{t})$. Then, the AP can obtain the numbers of arrived samples of all devices during round $t-1$, $m_{u}^{t-1}$’s, as follows: for each device whose local model was transmitted to the AP in round $t-1$ (i.e., $u\in\bar{\mathcal{U}}^{t-1}$), it can be known because $n_{u}^{t-1}+m_{u}^{t-1}$ is transmitted to the AP in round $t-1$ for the local update, and for the other devices (i.e., $u\in\mathcal{U}\setminus\bar{\mathcal{U}}^{t-1}$), it can be obtained by subtracting $n_{u}^{t-1}$ from $n_{u}^{t}$. Then, the AP updates the posterior distribution of ${\boldsymbol{\uptheta}}^{P}$, $\mu^{t}_{P}$, according to Bayes’ rule. We denote the posterior distribution for a system parameter $\theta$ that belongs to ${\boldsymbol{\uptheta}}$ in round $t$ by $\mu^{t}_{\theta}$. Then, the posterior distribution of ${\boldsymbol{\uptheta}}^{P}$ is updated by using $m_{u}$’s. For example, if the non-informative Jeffreys prior for the Poisson distribution is used for the system parameter $\theta_{u}^{P}$, then its posterior distribution in round $t$ is derived as follows [36]:

where $S=\sum_{\tau=1}^{t-1}m_{u}^{\tau}$. In this paper, we implement BALSA based on the non-informative prior. In each stage $k$ of the algorithm, the AP samples the system parameters ${\boldsymbol{\uptheta}}^{P}_{k}$ by using the posterior distribution in (27) and schedules the local update transmission according to the greedy policy in (23) using the sampled system parameters ${\boldsymbol{\uptheta}}^{P}_{k}$. We summarize BALSA in Algorithm 2.

We can prove the regret bound of BALSA as the following theorem.

The regret bound in Theorem 3 is sublinear in $T$. Thus, in theory, it is guaranteed that the average reward of BALSA per round converges to that of the optimal policy (i.e., $\lim_{T\rightarrow\infty}R(T)/T=0$), which implies the optimality of BALSA in terms of the long-term average reward per round.

In the WDLN, the state, ${\mathbf{s}}=({\mathbf{h}},{\mathbf{n}})$, can be reported to the AP from the devices. In typical wireless networks, reporting $n_{u}^{t}$’s to the AP needs data transmissions of the devices, which require exchanging control messages between the AP and the devices individually. Besides, it is expected that the number of devices participating in FL is typically much larger than the capacity of wireless networks that represents the number of devices that can transmit the local updates simultaneously [2]. Hence, it might be impractical (at least inefficient) that all individual devices report their numbers of the aggregated samples ${\mathbf{n}}$ to the AP in each round because of the excessive exchanges of the control messages between the AP and the individual devices for reporting ${\mathbf{n}}$. On the other hand, the channel gains may not require such data transmissions in an uplink case since the AP can estimate the channel gains of the devices by using the pre-defined reference signals transmitted from the devices [37]. In this context, in this section, we consider a WDLN in which in each round, the AP observes the channel gain ${\mathbf{h}}$ only and the numbers of the aggregated samples ${\mathbf{n}}$ are not reported to the AP. We can model this as a partially observable MDP (POMDP) based on the description in Section III, but the POMDP is often hard to solve because of the intractable computation. In this section, we develop an algorithm to solve the ALS problem in the partially observable WDLN by slightly modifying BALSA, which is called BALSA-PO.

In the fully observable WDLN, the AP observes the numbers of the aggregated samples, $n_{u}^{t}$’s, from all devices in each round $t$. In BALSA, $n_{u}^{t}$’s are used to choose the action according to the greedy policy in (23) and to obtain the numbers of the arrived samples of all devices during round $t-1$, $m_{u}^{t-1}$’s, for updating the posterior distribution of $\theta_{u}^{P}$. Hence, in the partially observable WDLN, where $n_{u}^{t}$’s are not observable, the AP cannot choose the action according to the greedy policy as well as obtain $m_{u}^{t-1}$’s. To address these issues, first, in BALSA-PO, the AP approximates $n_{u}^{t}$ by using the sampled system parameter of $\theta_{u}^{P}$ as $\tilde{n}_{u}^{t}=(T_{u}^{n}-1)\theta_{u,k}^{P}$, where $T_{u}^{n}$ is the number of the rounds from the latest successful local update transmission to the current round (i.e., $T_{u}^{n}=t-\max\{\tau<t:x_{u}^{\tau}=1\textrm{ and }a_{u}^{\tau}=1\}$) and $\theta_{u,k}^{P}$ is the sampled system parameter of $\theta_{u}^{P}$ in stage $k$. Then, the AP can choose the action according to the greedy policy by using the approximated $n_{u}^{t}$’s, i.e., $\tilde{n}_{u}^{t}$. However, the chosen action will be meaningful only if the posterior distribution keeps updated correctly since $n_{u}^{t}$’s are approximated based on the sampled system parameters. In the partially observable WDLN, it is a challenging issue updating the posterior distribution correctly since the AP cannot obtain $m_{u}^{t}$’s which are required to update the posterior distribution. However, fortunately, even in the partially observable WDLN, the AP can still observe the information about the number of samples $n_{u}^{t}+m_{u}^{t}$ in every successful local update transmission since the information is included in the local update transmission (line 10 of Algorithm 1). Note that $n_{u}^{t}+m_{u}^{t}$ denotes the sum of $m_{u}^{t}$’s over the rounds from the latest successful local update transmission to the current round. This sum of $m_{u}^{t}$’s in the partially observable WDLN is less informative than all individual values of $m_{u}^{t}$’s. Nevertheless, it is informative enough to update the posterior distribution because the update of the posterior distribution of $\theta_{u}^{P}$ requires only the sum of $m_{u}^{t}$’s as in (27). Hence, in the partially observable WDLN, the AP can update the posterior distribution of $\theta_{u}^{P}$ when device $u$ successfully transmits its local update. Then, BALSA-PO can be implemented by substituting the state in BALSA, ${\mathbf{s}}$, to $\tilde{{\mathbf{s}}}=({\mathbf{h}},\tilde{{\mathbf{n}}})$, where $\tilde{{\mathbf{n}}}=\{\tilde{n}_{1},...,\tilde{n}_{U}\}$ and changing the posterior distribution update procedure from BALSA in line 9 of Algorithm 2 as follows: “Update $\mu_{\theta_{u}^{P}}^{t+1}$ as in (27) using $n_{u}^{t}+m_{u}^{t}$ for device $u\in\bar{\mathcal{U}}^{t}$”.

We compare the computational complexities of the DP-based algorithms with/without a priori information, ALSA-PI, and BALSAs. The DP-based algorithm with perfect a priori information requires to run one of the standard DP methods once to compute the optimal policy for the system parameters in the perfect information. For the complexity analysis to compute the optimal policy, we consider the well-known value iteration whose computational complexity is given by $O(S^{2}A)$, where $S=|\mathcal{S}|=(|\mathcal{H}||\mathcal{N}|)^{U}$, $A=|\mathcal{A}|=\frac{U!}{W!(U-W)!}$, $\mathcal{H}$ is the set of the possible channel gain, and $\mathcal{N}$ is the set of the possible number of the aggregated samples. Hence, the computational complexity of the DP-based algorithm with perfect a priori information is given by $O(S^{2}A)$. The DP-based algorithm without a priori information denotes a learning method based on the posterior sampling in [34]. It operates over multiple stages as in BALSAs but the optimal policy for the sampled system parameters ${\boldsymbol{\uptheta}}_{k}$ in stage $k$ is computed based on the standard DP methods. Thus, its upper-bound complexity up to round $T$ is given by $O(S^{2}AT)$ if the policy is computed in all rounds. Since the system parameters are sampled once for each stage, we can also derive the lower bound of its computational complexity up to round $T$ as $O(\sqrt{S^{3}AT(\log T)^{-1}})$ from Lemma 3 in Appendix C.

Contrary to the DP-based algorithms, ALSA-PI and BALSAs use the greedy policy that finds the devices with the $W$ largest expected number of samples in each round. Thus, their computational complexity up to round $T$ is given by $O(UT\log U)$, where $U$ is the number of devices, according to the complexity of typical sorting algorithms. It is worth emphasizing that the complexity of BALSAs grows according to the rate of $U\log U$ while the complexity of the DP algorithms exponentially increases according to $U$. Besides, the DP-based algorithms have much larger computational complexities than BALSAs in terms of not only the asymptotic behavior but also the complexity coefficients in the Big-O notation since the DP-based algorithms require a complex computations based on the Bellman equation while BALSAs require only the computations to update posteriors in (27) and to sort the expected number of samples. From the analyses, we can see that the complexity of BALSAs is significantly lower than that of the DP-based algorithms. The computational complexities of the algorithms are summarized in Table IV.

We first provide the effectivity scores of the algorithms, which are the objective function of the ALS problem in (16), in Fig. 2. Note that Bench is not provided in the figure since in Bench, all devices transmit their local updates and all the transmissions succeed. From the figure, we can see that our BALSAs achieve the similar effectivity score to that of ALSA-PI, which is the optimal one. In particular, as the round proceeds, the effectivity scores of BALSAs converge to that of ALSA-PI. This clearly shows that our BALSAs can effectively learn the uncertainties in the WDLN. On the other hand, RR and $W$-max achieve the much lower effectivity scores compared with BALSAs. Especially, the effectivity score of $W$-max decreases as the round proceeds since it fails not only to maximize the number of the data samples used in training and but also to minimize the adverse effect from the stragglers. To show the validity of the effectivity score for effective learning in asynchronous FL, in the following subsections, it will be clearly shown that the algorithms with the higher effectivity scores achieve better trained model performances such as training loss, accuracy, robustness against stragglers, and learning speed.

To compare the performance of asynchronous FL with respect to the transmission scheduling algorithms, which is our ultimate goal, we provide the training loss and test accuracy of the algorithms. Fig. 3 provides the training loss and test accuracy with MNIST and CIFAR-10 datasets. From Figs. 3a and 3c, we can see that ALSA-PI, BALSA, and BALSA-PO achieve the similar training loss to Bench. Compared with them, RR and $W$-max achieve the larger training loss. The larger training loss of a model typically implies the lower accuracy of the model. This is clearly shown in Figs. 3b and 3d. From the figures, we can see that the models trained with RR and $W$-max achieve the significantly lower accuracy than the models trained with the other algorithms and their variances are much larger than those of the other algorithms. These results imply that RR and $W$-max fail to effectively gather the local updates while addressing the stragglers because they do not have a capability to consider the characteristics of asynchronous FL and the uncertainties in the WDLN. On the other hand, our BALSAs gather the local updates which are enough to train the model while effectively addressing the stragglers due to asynchronous FL by considering the effectivity score.

In Fig. 3, the unstable learning of the algorithms due to the stragglers is not clearly shown since the fluctuations of the training loss and accuracy in the figure are wiped out while averaging the results from the multiple simulation instances. Hence, in Fig. 4, we show the robustness of the algorithms against the stragglers more clearly through the training loss and test accuracy results of a single simulation instance. First of all, it is worth emphasizing that Bench is the most stable one in terms of the stragglers because it has no straggler. From Figs. 4a and 4c, we can see that the training losses of the algorithms considering asynchronous FL based on the effectivity score (i.e., ALSA-PI and BALSAs) are quite stable as much as that of Bench. Accordingly, their corresponding test accuracies are also stable. On the other hand, for the algorithms not considering asynchronous FL (i.e., RR and $W$-max), a lot of spikes (i.e., short-lasting peaks) appear in their training losses, and accordingly, their test accuracies are also unstable. Moreover, the training loss and test accuracy of $W$-max is significantly unstable compared with RR. This is because the scheduling strategy of $W$-max is highly biased according to the average channel gain while RR sequentially schedules all the devices. Such biased scheduling in $W$-max raises much more stragglers than RR. This clearly shows that a transmission scheduling algorithm may cause unstable learning if its scheduling strategy is biased without considering the stragglers.

In Fig. 5, we provide the target accuracy satisfaction rate and the average required rounds for satisfying the target accuracy of each algorithm. For the MNIST dataset, we vary the target accuracy from 0.7 to 0.95, and for the CIFAR-10 dataset, we vary it from 0.48 to 0.72. The test accuracy satisfaction rate of each algorithm is obtained as the ratio of the simulation instances, where the test accuracy of the corresponding trained model exceeds the target accuracy at the end of the simulation, to the total simulation instances. For the average required rounds for satisfying the target accuracy, we find the minimum required rounds of each simulation instance in which the target accuracy is satisfied. To avoid the effect of the spikes, we set a criteria of satisfying the target accuracy as follows: all test accuracies in three consecutive rounds exceed the target accuracy. From Figs. 5a and 5c, we can see that the satisfaction rates of ALSA-PI and BALSAs are similar to that of Bench. On the other hand, the satisfaction rates of RR and $W$-max significantly decrease as the target accuracy increases. Figs. 5b and 5d provide the average required rounds of the algorithms to satisfy the target accuracy. From the figures, we can see that the algorithms considering asynchronous FL (ALSA-PI and BALSAs) require the smaller rounds to satisfy the target accuracy. This clearly shows that the algorithms that consider the effective score of asynchronous FL (i.e., ALSA-PI and BALSAs) have a faster learning speed than the algorithms that do not consider it (i.e., RR and $W$-max).