Deep Reinforcement Learning Based Vehicle Selection for Asynchronous Federated Learning Enabled Vehicular Edge Computing

Vehicle $i$ uses its local data to train a local model, thus the local training delay $T_{l}^{i}$ of vehicle $i$ can be calculated as:

where $C_{0}$ is the number of CPU cycles required to process one unit of data, ${{\mu}_{i}}$ is the computing resources of vehicle $i$, i.e., CPU cycles frequency.

Denote ${{P}_{i}}\left(t\right)$ as the position of vehicle $i$ at time slot $t$, ${{d}_{ix}}\left(t\right)$ and $d_{y}$ as the distance between vehicle $i$ and the antenna of RSU at time slot $t$ along the $x$-axis and $y$-axis. Thus ${{P}_{i}}\left(t\right)$ can be expressed as $\left({{d}_{ix}}\left(t\right),{{d}_{y}},0\right)$. Here, $d_{y}$ is a fixed value, and ${{d}_{ix}}\left(t\right)$ can be denoted as:

where $d_{i0}$ is the initial position of vehicle $i$ along $x$-axis.

We set the height of RSU’s antenna as $H_{r}$, and thus the position of antenna of RSU can be expressed as ${{P}_{r}}=\left(0,0,{{H}_{r}}\right)$. Then the distance between vehicle $i$ and the antenna of RSU at time slot $t$ can be expressed as:

We set the transmission rate of vehicle $i$ at time slot $t$ to be $t{r_{i}}\left(t\right)$. According to Shannon’s theorem, it can be expressed as:

where $B$ is the transmission bandwidth, $p_{0}$ is the transmission power of each vehicle, ${{h}_{i}}\left(t\right)$ is the channel gain of vehicle $i$ at time slot $t$, $\alpha$ is the path loss exponent, ${{\sigma}^{2}}$ is the power of noise.

We use autoregressive model to formulate the relationship between $h_{i}\left(t\right)$ and $h_{i}\left(t-1\right)$:

where ${{\rho}_{i}}$ is the normalized channel correlation coefficient between consecutive time slots, $e\left(t\right)$ is the error vector following a complex Gaussian distribution and is related to ${{h}_{i}}\left(t\right)$. According to Jake’s fading spectrum, ${{\rho}_{i}}={{J}_{0}}\left(2\pi f_{d}^{i}t\right)$, where ${J_{0}}\left({\rm{\cdot}}\right)$ is the zeroth-order Bessel function of the first kind and $f_{d}^{i}$ is the Doppler frequency of vehicle $i$ which can be calculated as:

where $\Lambda$ is the wavelength, $\theta$ is the angle between the moving direction ${x_{0}}=\left({1,0,0}\right)$ and the uplink communication direction ${P_{r}}-{P_{i}}\left(t\right)$. Thus $\cos\theta$ can be calculated as:

The transmission delay of vehicle $i$ for uploading its local model $T_{u}^{i}\left(t\right)$ can be denoted as:

where $\left|w\right|$ is the size of the local model of each vehicle.

Considering that the vehicle mobility can be reflected by its position while the local training delay and transmission delay of the vehicle are related to the vehicle’s time-varying available computing resources and current channel condition, so we define the state at time slot $t$ as:

where $Tr\left(t\right)$ is the set of the transmission rates of each vehicle at time slot $t$, i.e., $Tr\left(t\right)=\left(t{{r}_{1}}\left(t\right),t{{r}_{2}}\left(t\right),\ldots t{{r}_{K}}\left(t\right)\right)$, $\mu\left(t\right)$ is the set of available computing resources of all vehicles at time slot $t$, i.e., $\mu\left(t\right)=\left({{\mu}_{1}}\left(t\right),{{\mu}_{2}}\left(t\right),\ldots{{\mu}_{K}}\left(t\right)\right)$, ${{d}_{x}}\left(t\right)$ is the set of all vehicles’ positions along the $x$-axis at time slot $t$, i.e., ${{d}_{x}}\left(t\right)=\left({{d}_{1x}}\left(t\right),{{d}_{2x}}\left(t\right),\ldots{{d}_{Kx}}\left(t\right)\right)$, $a\left(t-1\right)$ is the action at time slot $t-1$.

Since the purpose of DRL is to select the better vehicles for AFL according to the current state, we define the system action at time slot $t$ as:

where ${\lambda_{i}}\left(t\right),i\in\left[{1,K}\right]$ is the probability of selecting vehicle $i$, and we define that ${\lambda_{1}}\left(0\right)={\lambda_{2}}\left(0\right)=\ldots={\lambda_{K}}\left(0\right)=1$.

We denote a new set ${a_{d}}\left(t\right)=\left({{a_{d1}}\left(t\right),{a_{d2}}\left(t\right),\ldots{a_{dK}}\left(t\right)}\right)$ in order to select specific vehicles. After we normalize the action, if the value of ${{\lambda}_{i}}\left(t\right)$ is greater than or equal to 0.5, ${{a}_{di}}\left(t\right)$ is recorded as 1, otherwise 0. Then we get the set that is composed of of 0 and 1 where the binary value stands for if a vehicle is selected or not.

We aim to select a vehicle with better performance for AFL to obtain a more accurate global model at the RSU where both the local training delay, transmission delay and the accuracy of the global model are critical metrics. Thus, we define the system reward at time slot $t$ as:

where ${\omega}_{1}$ and ${\omega}_{2}$ are the non-negative weighting factors, $Loss\left(t\right)$ is the loss computed by AFL, which will be discussed later.

The expected long-term discount reward of the system can be expressed as:

where $\gamma\in\left(0,1\right)$ is the discounted factor, $N$ is the total number of time slots, $\mu$ is the policy of system. In this paper we aim to find an optimal policy to maximize the expected long-term discounted reward of the system.

Considering the state and action spaces are continuous and DDPG is suitable for solving DRL problems under continuous state and action spaces, we employ DDPG to solve our problem.

DDPG algorithm is based on actor-critic network architecture. The actor network is used for policy promotion, and the critic network is used for policy evaluation. Here, both actor and critic network are constructed by deep neural network (DNN). Specifically, actor network is used to approximate policy $\mu$, and the approximated policy is expressed as ${\mu}_{\delta}$. The actor network observes state and output the action based on policy ${\mu}_{\delta}$.

We improve and evaluate the policy iteratively in order to obtain the optimal policy. To ensure the stability of the algorithm, the target network composed of the target actor network and the target critic network are also employed in the DDPG, where the architectures are the same as the original actor and critic networks, respectively. The proposed algorithm is shown in Algorithm 1.

Let $\delta$ be the actor network parameter, $\xi$ be the critic network parameter, ${{\delta}^{*}}$ be the optimized actor network parameter, ${{\xi}^{*}}$ be the optimized critic network parameter, ${{\delta}_{1}}$ be the target actor network parameter and ${{\xi}_{1}}$ be the target critic network parameter. $\tau$ is the update parameter of target network and ${{\Delta}_{t}}$ is the exploration noise at time slot $t$. $I$ is the size of mini-batch. Now, we will describe our algorithm in detail.

First, we initialize $\delta$ and $\xi$ randomly, and initialize $\delta_{1}$ and $\xi_{1}$ in the target network as $\delta$ and $\xi$ respectively. At the same time, we initialize the replay buffer $R_{b}$.

Then our algorithm will be executed for $E_{max}$ episodes. In the first episode, we first initialize the positions of all vehicles, the channel states and computing resources of vehicles, and set ${{\lambda}_{1}}\left(0\right)\text{=}{{\lambda}_{2}}\left(0\right)\text{=}\ldots\text{=}{{\lambda}_{K}}\left(0\right)\text{=}1$. Then at time slot 1, the system can get the state, i.e., $s\left(1\right)=\left(Tr\left(1\right),\mu\left(1\right),{{d}_{x}}\left(1\right),a\left(0\right)\right)$. Meanwhile, the convolutional neural network (CNN) is employed as the global model ${{w}_{0}}$ at the RSU.

Our algorithm will execute from time slot 1 to time slot $N$. At time slot 1, actor network can get the output ${{\mu}_{\delta}}\left(s|\delta\right)$ according to the state. Note that we add a random noise ${{\Delta}_{t}}$ to the action, and the system can get the action as $a\left(1\right)={{\mu}_{\delta}}\left(s\left(1\right)|\delta\right)+{{\Delta}_{t}}$. Thus we get ${{a}_{d}}\left(1\right)$ based on the action and determine the selected vehicles at this time slot. The selected vehicles will conduct AFL. That is, all the selected vehicles train their local models according to their local data, then upload them to RSU asynchronously for global model updating. Given the action, we can get the reward at time slot 1. After that we update the positions of vehicles according to Eq. (2), recalculate the channel states and the available computing resources of the vehicles, and update the transmission rates of the vehicles according to Eq. (4). Then it can get the next state $s\left(2\right)$. The related sample $\left(s\left(1\right),a\left(1\right),r\left(1\right),s\left(2\right)\right)$ will be stored in $R_{b}$. Note that the system will iteratively calculate and store the samples into $R_{b}$, until reaching the capacity of $R_{b}$.

When the number of tuples is bigger than $I$ in $R_{b}$, the parameters $\delta$, $\xi$, $\delta_{1}$ and $\xi_{1}$ of actor network, critic network and target networks respectively will be trained to maximize $J\left({{\mu}_{\delta}}\right)$. Here, $\delta$ is updated towards the gradient direction of $J\left({{\mu}_{\delta}}\right)$, i.e., ${{\nabla}_{\delta}}J\left({{\mu}_{\delta}}\right)$. We set ${{Q}_{{{\mu}_{\delta}}}}\left(s\left(t\right),a\left(t\right)\right)$ as the action-value function which obeys policy ${{\mu}_{\delta}}$ under $s\left(t\right)$ and $a\left(t\right)$, it can be expressed as:

It represents the long term expected discount reward at time slot $t$.

The existing paper has proved that the solution to ${{\nabla}_{\delta}}J\left({{\mu}_{\delta}}\right)$ can be replaced by solving the gradient of ${{Q}_{{{\mu}_{\delta}}}}\left(s\left(t\right),a\left(t\right)\right)$, i.e., ${{\nabla}_{\delta}}{{Q}_{{{\mu}_{\delta}}}}\left(s\left(t\right),a\left(t\right)\right)$ [46]. Due to the continuous action space of ${{Q}_{{{\mu}_{\delta}}}}\left(s\left(t\right),a\left(t\right)\right)$, it cannot be solved by Bellman Equation [47]. In order to solve this problem, critic network uses $\xi$ to approximate ${{Q}_{{{\mu}_{\delta}}}}\left(s\left(t\right),a\left(t\right)\right)$ by ${{Q}_{\xi}}\left(s\left(t\right),a\left(t\right)\right)$.

When the number of tuples is bigger than $I$ in $R_{b}$, system will sample $I$ tuples randomly from $R_{b}$ to form a mini-batch. Let $\left({{s}_{x}},{{a}_{x}},{{r}_{x}},s_{x}^{{}^{\prime}}\right),x\in\left[1,2,\ldots,I\right]$ be the $x$-th tuple in the mini-batch. The system will input the $s_{x}^{{}^{\prime}}$ to the target actor network, and get the output action $a_{x}^{{}^{\prime}}={{\mu}_{{{\delta}_{1}}}}\left(s_{x}^{{}^{\prime}}|{{\delta}_{1}}\right)$. Then input $s_{x}^{{}^{\prime}}$ and $a_{x}^{{}^{\prime}}$ to the target critic network, and get the action-value function ${{Q}_{{{\xi}_{1}}}}\left(s_{x}^{{}^{\prime}},a_{x}^{{}^{\prime}}\right)$. The target value can be calculated as:

According to ${{s}_{x}}$ and ${{a}_{x}}$, critic network will output the ${{Q}_{\xi}}\left({{s}_{x}},{{a}_{x}}\right)$, then the loss of tuple $x$ is given by:

When all the tuples act as the input to the critic network and target networks, we can get the loss function:

In this case, the critic network updates $\xi$ by employing the gradient descent of ${{\nabla}_{\xi}}L\left(\xi\right)$ to the loss function $L\left(\xi\right)$.

Similarly, actor network updates $\delta$ by employing the gradient ascent, i.e., ${{\nabla}_{\delta}}J\left({{\mu}_{\delta}}\right)$, to minimize $J\left({{\mu}_{\delta}}\right)$[48], where ${{\nabla}_{\delta}}J\left({{\mu}_{\delta}}\right)$ is calculated by action-value function approximated by critic network as follows:

Here the input of ${Q}_{\xi}$ is $a_{x}^{\mu}={{\mu}_{\delta}}\left({{s}_{x}}|\delta\right)$.

In the end of a time slot $t$, we update the parameters of target networks as follows:

where $\tau$ is a constant and $\tau\ll 1$.

Then input $s^{\prime}$ to actor network and start the same procedure for the next time slot. When the time slot $t$ reaches $N$, this episode is completed. In this case, system will initialize the state $s\left(1\right)=\left(Tr\left(1\right),\mu\left(1\right),{{d}_{x}}\left(1\right),a\left(0\right)\right)$ and execute the next episode. When the number of episodes reaches $E_{max}$, the training is finished. We get the optimized ${\delta}^{*}$, ${\xi}^{*}$, $\delta_{1}^{*}$ and $\xi_{1}^{*}$. The overall DDPG flow diagram is listed in Fig. 2.

In this section, we will introduce the process of AFL in detail, which is used in the step 10 in Algorithm 1.

Let ${{V}_{k}},k\in\left[1,{{K}_{1}}\right]$ be the selected vehicles, where $K_{l}$ is the total number of the selected vehicles. In the AFL, each vehicle will go through three stages: global model downloading, local training, uploading and updating. Specifically, vehicle $V_{k}$ will first download the global model from the RSU, then it will train a local model using local data for some iterations. Then it will upload the local model to the RSU. Once the RSU receives a local model, it updates the global model immediately. To be clearly, we use the AFL training at time slot $t$ of vehicle $V_{k}$ as an example.

The testing stage employs the achieved critic network, target actor network and target critic network in the training stage. In the testing stage, the system will select the policy with optimized parameter ${{\delta}^{*}}$. The process of testing stage is shown in Algorithm 3.

In this section, the simulation tool is python 3.9. Our actor network and critic network are all selected as the DNN with two hidden layers. The two hidden layers have 400 and 300 neurons, respectively. The exploration noise obeys the Ornstein-Uhlenbeck (OU) process with variation 0.02 and decay-rate 0.15. We use MNIST dataset to allocate data to vehicles and the computing resources of vehicles obey a truncated Gaussian distribution. Here the unit of the computing resources is CPU-cycles/s. We configured a vehicle as the bad vehicle, that is, it has small amount of data and computing resources, and the local model is disturbed by random noise. The rest simulation parameters are shown in Table 2.

Fig. 3 shows the system reward with respect to different epochs in the training stage. One can see that when the number of epoches is small, the reward has a large variation. This is due to that the system is learning and optimizing the network in the initial phase, so some explorations (i.e., action) incur poor performance. As the number of epoches increases, the reward gradually becomes stable and smoother. It means that at this time, the system has gradually learned the optimal policy, and the training of the neural network has been close to be completed.

Fig. 4 depicts the two components of the reward in the testing stage: the loss calculated in the AFL and the sum of the local training delay and the transmission delay. It can be seen that the loss is decreasing as the number of steps increases. This is attributed to the fact that vehicles are constantly uploading local model to update global model at the RSU, so the global model becomes more accurate. The sum of delay shows a certain fluctuation, because of the dynamic available computing resources of the vehicle and its time-varying location.

Fig. 5 shows the accuracy of our scheme, traditional AFL and traditional FL in the presence of bad node. From the figure one can see that the accuracy of our scheme remains at a good level and gradually increases and finally reaches stability. This indicates that our scheme can effectively remove the bad node in the model training. Since traditional AFL and FL do not have the function of selecting the vehicles, their accuracy are seriously affected by the bad node, resulting in large fluctuations.

Fig. 6 shows the accuracy of our scheme and traditional AFL and FL after the vehicle selection. From the figure one can see that the accuracy of all schemes are increasing when the number of step increases and finally become stable. However, our scheme has the highest accuracy among them. This is because our scheme considers the impact of local training delay and transmission delay of vehicles in global model updating.

Fig. 7 depicts the training delay of our scheme compared to the FL as the global round (i.e., step) increases. It shows the delay of FL remains to be high while our scheme keeps a small delay. This is because FL only starts updating the global model when all local models of selected vehicles are received, whereas in our scheme, RSU is updated every time when it receives a local model uploaded from a vehicle. At the same time, we can observe that the delay of our scheme appears to rise at first and then fall and rise again. This is because the proposed scheme selects four vehicles to update the global model one by one. At the same time, owning to the large local computing delay of the vehicles compared to the transmission delay, the local computing delay of the vehicle dominates. In this case, since the vehicle that finishes the earliest local training will update the global model first, the training delay gradually increases. When the four vehicles all update the global model, the vehicles will repeat the above global model updating until reaches the maximum number of step.

Fig. 8 depicts the accuracy of our scheme and traditional AFL with selected vehicles under different value of $\beta$. It shows when $\beta$ is small, the accuracy of the model keeps relatively high. In contrast, when $\beta$ gradually increases, the accuracy of global model gradually decreases. This is because when $\beta$ is relatively large, the weight of the local model is much small, thus the update of the global model mainly depends on the previous hyperparameter values of global model, which decreases the influence and contribution of new local model of all vehicles, so it has a significant impact on the accuracy of global model. At the same time, the accuracy of our scheme is better than that of AFL. This is because our scheme has considered the influence of local computing delay and transmission delay of vehicles.