Optimal Beam Association for High Mobility mmWave Vehicular Networks: Lightweight Parallel Reinforcement Learning Approach

Various solutions in the literature have been proposed to address the above inherent limitations of mmWave communications, e.g., [6]-[11]. In [6], the authors aimed to jointly optimize the transmit power, interference coordination, and beamforming to maximize the signal to interference plus noise ratio and the sum-rate. In [7], the authors aimed to enhance the users’ quality-of-experience by scheduling communication tasks on both microwave and mmWave bands. Similarly, a new protocol that enables simultaneous connections to conventional 4G cells and 5G mmWave cells is introduced in [8]. Differently, a new adaptive beamforming strategy is proposed in [10] to improve the system communication rate by taking the noise into account and selecting the beamforming vectors based on the posterior of the angle-of-arrival.

The aforementioned solutions and most others in the literature did not account for the high mobility of vehicles and dynamics of the environment. In [12], the authors proposed an adaptive channel estimation mechanism for beamforming with the aid of location information. The position information is then demonstrated as an important factor to greatly improve the initial association of vehicles to the infrastructure. Similarly, in [13], the authors introduced a beam switching mechanism in mmWave vehicle-to-infrastructure communications. In [14], a deep learning approach is proposed to predict mmWave beams and blockages with sub-6 GHz channels. Specifically, the authors studied that there are deterministic mapping from uplink sub-6 GHz channels to downlink mmWave channels. Hence, with the uplink channel knowledge, it is possible to predict/infer mmWave beams and blockages with low training overhead because sub-6 GHz channels can be estimated by using a small number of pilots. Nevertheless, it is very challenging to derive and analyze these mapping functions. To tackle this, the authors developed a deep learning model with a sufficiently large deep neural network to efficiently predict the optimal beams and blockages status.

To deal with the inherent dynamic and uncertainty of mmWave links’ quality, the authors in [15] proposed an online learning algorithm to obtain the optimal beam selection policy based on the prior environment information. This problem is first modeled as a contextual multi-armed bandit problem. Then, the learning algorithm is developed to guide the mmWave base station to select an optimal subset of beams for vehicles. In [16], the authors aimed to maximize the number of bits delivered from a base station to a mobile user by considering the trade-off between directive data transmission (DT) and directional beam training (BT) in mmWave vehicular networks. Specifically, the system is formulated as a partially observable Markov decision process with the system state defined as the position of the mobile user within the road link. Note that in both [15] and [16], only one mmWave base station is considered. In practice where multiple base stations are often in place, they can cooperate in the beam association and handoff process to achieve a globally optimal beam selection solution. An online learning approach is proposed in [17] to jointly optimize beam training, data transmission, and handover processes. Nevertheless, the effect of high mobility on the system performance has not been studied. Moreover, similar to [16], only one mobile user is considered in this paper. In [18], the authors considered the vehicle-cell association problem for mmWave vehicular networks to maximize the average rate of vehicles. Specifically, the authors first formulated the association problem as a discrete non-convex optimization problem. Then, a learning algorithm is developed to estimate the solution for the non-convex optimization problem. It is worth noting that the above works and others in the literature have not accounted for the effects of the high mobility and blockage on beam association/handover. In addition, these learning approaches do not leverage the fact that there are multiple vehicles running on the road at the same time to improve the learning efficiency.

Given the above, this work aims to develop an optimal beam association framework for mmWave vehicular communications under the high mobility of vehicles and the uncertainty of blockages. In particular, to capture the dynamics of blockages, channel quality, and mobility, we first model the problem as a semi-Markov decision process (SMDP). The Q-learning (QL) algorithm is then often adopted to solve the optimization problem the underlying SMDP. Nevertheless, the QL algorithm is well known for its slow convergence rate, especially in dynamic and complicated environments. Instead of adopting deep reinforcement learning structures e.g., double deep QL, deep dueling, in this article, we leverage the fact that there are usually multiple vehicles on the road to speed up the convergence to the optimal solution. To that end, we develop a lightweight yet very effective parallel QL algorithm to quickly obtain the optimal policy by simultaneously learning from various vehicles. Specifically, vehicles on the road can act as active learners to help the system simultaneously collect data. Based on the collected data, the proposed parallel QL algorithm can quickly learn the environment information, e.g., RSSI profile, beam’s location, and blockage, to derive the optimal beam association strategy. The proposed parallel QL algorithm does not only require lower complexity but also converge faster than the latest deep learning-based approaches (e.g., double QL, deep dueling). Moreover, unlike deep reinforcement learning methods (e.g., [18]), our proposed parallel QL framework is proven to always converge to the optimal policy. We show that the high mobility and parallelism of vehicles now become helpful in speeding up the learning process of our underlying algorithm.

Moreover, unlike existing works, e.g., [15], [18], in which learning algorithms are deployed at the vehicles with limited resources, in our design, the eNodeB executes the parallel QL algorithm and sends optimal beam association/handover actions to vehicles. As such, the computing complexity is moved to the eNodeB. The simulation results then show that our proposed solution can increase the data rate for each vehicle by up to 47% and reduce the disconnection probability by 29% compared to existing approaches.

The major contributions of this paper are as follows.

• •

  Develop an optimal beam association solution for high-mobility mmWave vehicular communication networks using the semi-Markov decision process framework that can effectively deal with the inherent dynamics of the mmWave connection quality and the uncertainty of the environment, e.g., beam’s location, RSSI profile, the velocity of the vehicle, and blockages, in a real-time manner.

• •

  Develop a lightweight yet very effective parallel QL algorithm to quickly obtain the optimal policy by simultaneously learning from various vehicles to update the global Q-table at the eNodeB. The proposed parallel QL algorithm does not only require lower complexity but also converge faster than advanced deep reinforcement learning approaches (e.g., deep double and deep dueling). Specifically, the algorithm deploys multiple learning processes at the eNodeB, and each learning process is assigned to learn from a vehicle on the considered road to update the global policy.

• •

  Prove that the proposed parallel QL framework converges with probability one to the optimal policy. Note that deep QL based approaches (e.g., deep double and deep dueling) with advanced deep neural networks are not always guaranteed to converge. We also provide a comprehensive analysis of the convergence time/rate, complexity, and overhead of the proposed framework.

• •

  Perform extensive simulations to demonstrate the effectiveness of the proposed parallel QL algorithm. By learning from multiple vehicles and exploiting the high mobility of vehicles, our proposed algorithm can achieve the performance close to that of the hypothetical scheme which requires complete environment information in advance.

The rest of paper is organized as follows. The system model is described in Section II. Section III presents the problem formulation based on the SMDP. Then, Section IV describes the conventional QL algorithm and our proposed parallel QL algorithm. After that, we provide the evaluation results in Section V. Finally, the conclusion is highlighted in Section VI.

The state space of each vehicle is represented as the discretized space of RSSI levels, the connected beam, the speed, and the direction of the vehicle. Formally, the state space $\mathcal{S}$ is defined as follows:

where $l$ is the RSSI level of the current vehicle and $b_{n,k}$ is the current connected beam of the current vehicle. $v$ is the vehicle’s speed, $v_{\mathrm{max}}$ is the maximum speed of the vehicle, and $d$ is the direction of the vehicle (0 and 1 represent two directions of the vehicle). $b_{0,0}$ is a virtual beam used to capture the case that there is no available beam at a given location of the current vehicle. In this case, the RSSI level of the current vehicle is always $0$. Note that due to the inherent dynamics of mmWave link’s quality, the mmWave instantaneous RSSI estimate can be less accurate, providing less useful information to decide the next action. This is one of the key motivations of our machine learning-based framework that can learn the RSSI profiles of beams by interacting with the environment and observing the actual reward (transmission rate) other than just relying on the instantaneous RSSI estimate.

The road is modeled as a one-dimension area $\mathcal{W}\in\mathbb{R}$ that is discretized to $\left\lfloor\frac{W}{z}\right\rfloor$ zones, where $W$ is the length of the considered road, $z$ is the length of each zone, and $\lfloor.\rfloor:\mathbb{R}\rightarrow\mathbb{N}$ is the floor function. When the vehicle at location $w\in\mathcal{W}$, the vehicle is at zone $\left\lfloor\frac{w}{z}\right\rfloor$-th. At the current state $s\in\mathcal{S}$, an event $e_{s}$ is triggered if a vehicle reaches a new zone. Note that as the vehicle’s speed is not fixed, the time interval between two consecutive epochs varies. To capture that, the SMDP is used in this work, instead of the conventional Markov decision process with identical time slots. Specifically, the eNodeB only takes an action when event $e_{s}$ occurs. Note that unlike existing works that have the location information as part of the state space, our framework captures the location information via events $e_{s}$ under the SMDP formulation. This allows us to efficiently deal with dynamics and uncertainties during the learning process, which is very beneficial in our considered system with the dynamics of blockages, channel quality, and mobility.

When the vehicle reaches a new zone on the road, i.e., event $e_{s}$ is triggered (given its current state $s\in\mathcal{S}$), the eNodeB decides if the vehicle needs to associate to a new beam or stay on the current beam. The action space $\mathcal{A}_{s}$ is defined as:

where $a$ is the action made at state $s$. $a=b_{n,k}$ if the eNodeB guides the vehicle to connect to beam $b_{n,k}$. This includes the case staying with the current beam. $a=b_{0,0}$ if there is no available beam at the current location.

In this paper, we aim to maximize the long-term average data rate of the system. As mentioned, at decision epoch $t$, if an action is taken so that the vehicle connects to beam $b_{n,k}$, it can communicate with a rate of $0\leq r_{n,k}\leq R_{max}$ corresponding to the current RSSI level. The resulting data that the vehicle receives from the mmBS is calculated as $j_{n,k}^{t}r_{n,k}$, where $j_{n,k}^{t}$ is the connection time between two consecutive decision epochs (during which the vehicle can communicate with the mmBS through beam $b_{n,k}$). As the algorithm observes the reward at the end of each decision epoch, $j_{n,k}^{t}$ is the duration from the time that the vehicle enters the current zone until it leaves to enter the next zone. $j_{n,k}^{t}$ hence depends on the speed of the vehicle $v_{t}$ at the epoch $t$. Practically, the speed $v_{t}$ can change from one to another epoch or even during the time $j_{n,k}^{t}$. However, without loss of generality, we assume that the time $j_{n,k}^{t}$ between two consecutive epochs is small enough (e.g., by setting the length per zone $z$ as small as necessary) so that the vehicle’s RSSI level and speed remain unchanged. Thus, connection time $j_{n,k}^{t}$ can be calculated as $\frac{z}{v_{t}}$.

In addition, the service may be interrupted during the handover/beam-switching, denoted as $h$, i.e., the time it takes for the vehicle to switch to the new mmBS. We assume that the handover time is the same for all the mmBSs. Taking the handover time into account, at state $s\in\mathcal{S}$, the immediate data rate after performing action $a$ is obtained in (5).

where $s_{t}$ and $a_{t}$ are the system state and the action taken at decision epoch $t$, respectively and $r_{n,k}$ is the communication rate when the vehicle connects to beam $b_{n,k}$ as defined in (2). Note that $j_{n,k}^{t}$, $r_{n,k}$, and $h$ depend on the channel and environment conditions such as blockages, beam’s location, and RSSI profiles. Under our design, they are implicitly learned through interacting with the environment and observing the immediate reward in an online manner.

The decision policy $\pi$ of the proposed SMDP can be defined as a mapping from the state space to the action space: $\mathcal{S}\rightarrow\mathcal{A}_{s}$ [21], [36]. Thus, with initial state $s$, the long-term average data rate is formulated as follows:

where $\xi_{t}$ is the time interval between the $t$-th and $(t+1)$-th decision epochs, $\pi(s)$ is the action at state $s$ based on policy $\pi$, and $r$ is the immediate reward after performing an action. In Theorem 1, we will prove that the limit in (6) exits [22].

Next, in Theorem 2, we prove that the underlying Markov chain is irreducible, and thus the long-term average date rate $\mathcal{R}(\pi)$ does not depend on the initial state $s_{0}$.

Then, the long-term average data rate optimization problem can be formulated as follows:

Our aim in this work is finding the optimal beam association policy to maximize the long-term average data rate, i.e.,

This section presents the QL algorithm [26], which enable the eNodeB to obtain the optimal beam association strategy for vehicles without prior environment parameters, e.g., RSSI profiles and blockages. The key idea of the QL algorithm is updating the Q-value function for all state-action pairs stored in a Q-table. At a given system state, the QL algorithm performs an action and observes the immediate reward as well as the next state of the system. Based on these observations, the algorithm can update the Q-value for the current state-action pair based on the Q-value function [26]. As such, the learning process is able to learn from the previous experiences, i.e., current state, action, next state, and immediate reward, to derive the optimal solution [26]. In the following, we present the fundamentals of the Q-value function.

We first define the beam association policy as $\pi:\mathcal{S}\rightarrow\mathcal{A}$. In particular, $\pi$ is a mapping from a given state to its corresponding action. Our aim in this paper is finding the optimal beam association policy $\pi^{*}$ to optimize the system performance in terms of the average data rate, disconnection probability, and number of handovers. Then, we define $\mathcal{V}^{\pi}(s):\mathcal{S}\rightarrow\mathbb{R}$ as the expected value function of state $s\in\mathcal{S}$ given policy $\pi$. $\mathcal{V}^{\pi}(s)$ can be formulated as follows:

where $0\leq\gamma<1$ denotes the discount factor. In particular, $\gamma$ represents the effect of the future rewards. The higher the discount factor is, the more important future rewards are. At each state $s$, we aim to find the optimal action to derive the optimal beam association policy $\pi^{*}$, which is a map from a given state to the optimal action. To do that, the optimal value function for each state has to be obtained as in (13).

We then denote the optimal Q-function state-action pair $(s,a),\forall s\in\mathcal{S},\forall a\in\mathcal{A}$ as follows:

Hence, the optimal value function is written as follows:

To solve (15), we can update the Q-function to determine the optimal Q-values of all state-action pairs by using (16) [26]:

where $\tau_{t}$ denotes the learning rate, determining the impact of new experiences to the current Q-value [26]. By updating the Q-value functions of all state-action pairs by using (16), the algorithm can derive the optimal beam association policy.

Note that the conventional QL algorithm can converge to the optimal beam association policy quickly when the system is simple. However, with the dynamics and uncertainties of the system considered in this work, the QL algorithm may take a very long time to obtain the optimal strategy. This is due to the fact that the QL algorithm require a huge number of training episodes to collect enough data for learning. To speed up the QL algorithm, the deep reinforcement learning algorithm is usually adopted in the literature. Nevertheless, this algorithm requires high performance computing resources and does not ensure to converge to the optimal policy due to the overestimation of the optimizer [22], [23]. The convergence rate of the deep reinforcement learning algorithm can be improved by implementing several techniques. For example, by using the deep dueling neural network architecture [24], one can significantly improve the convergence rate and stability of the QL algorithm. However, in our work, we leverage the fact that there are simultaneously multiple vehicles running on the road to design the parallel QL algorithm. The proposed algorithm does not only require lower complexity but also converge faster than the latest deep learning-based approaches. In particular, vehicles running on the road act as active learners which can help the system simultaneously collect data and significantly speed up the learning process as shown in Fig. 2.

To that end, the parallel QL algorithm employs multiple learning processes. Each learning process is assigned for a vehicle running on the road (in the coverage of the eNodeB). Specifically, each learning process $i$ updates the Q-value function at the global Q-table as follows:

where $0\leq\gamma<1$ is the discount factor that presents the effect of future rewards [26]. In particular, when $\gamma$ is low, e.g., close to 0, the learning process prefers the current reward. Differently, when $\gamma$ is high, e.g., close to 1, the long-term reward will be considered. In this work, we set $\gamma$ the same for all vehicles, i.e., learners. $r_{t}^{i}(s_{t}^{i},a_{t}^{i})$ is the immediate reward when vehicle $i$ performs action $a_{t}^{i}$ at state $s_{t}^{i}$ (computed using equation 5 above). $\tau_{t}$ is the learning rate at decision epoch $t$ [26]. Note that the learning rate can be fixed at a constant value or it can be adjusted when running the algorithm. In this paper, the learning rate is fixed during the training process and is the same for all learning processes. At each decision epoch and given a current state, i.e., RSSI level and current connected beam, the current vehicle chooses to connect to a beam (following the current beam association policy sent from the eNodeB) and observes the data rate of the connected beam as well as the next state. Then, these observations are sent to the eNodeB for learning by the corresponding learning process to update the global Q-table (equation 17). Algorithm 1 describes the proposed parallel QL algorithm.

In particular, vehicle $i$ first observes the current state $s_{t}^{i}\in\mathcal{S}$ and performs action $a_{t}^{i}$ based on the $\epsilon$-greedy policy [32]. Then, the eNodeB selects an action that maximizes the Q-value function with probability $1-\epsilon$ and a random action with probability $\epsilon$. Then the eNodeB sends this action to vehicle $i$ to perform. In this work, we gradually reduce the value of $\epsilon$. In other words, the algorithm first chooses random actions and gradually change to the deterministic strategy, i.e., choose an action with the highest Q-value at a given state. To that end, $\epsilon$ is set at a high value (e.g., 1) when the algorithm starts running. Then, at later iterations, the value of $\epsilon$ is slowly reduced to a small value (e.g., 0.01). After performing action $a_{t}^{i}$, vehicle $i$ observes immediate data rate $r_{t}^{i}(s_{t}^{i},a_{t}^{i})$ and next state $s_{t+1}^{i}$. These observations are then sent to the eNodeB for learning. Note that the learning process of each vehicle is independent from others, and all the learning processes share the same global Q-table. By doing that, the Q-table is updated with more experiences from multiple vehicles running on the road. As such, the convergence rate and convergence time of the parallel QL algorithm will be better than that of the conventional QL algorithm as demonstrated in Section V.

The convergence of our proposed parallel QL algorithm can be always guaranteed. In particular, as the learning processes update the global Q-table simultaneously, the selection of an action of a learning process may be based on a new Q-value that is updated by other learning processes. Specifically, at state $s$, a learning process chooses to perform action $a$ that maximizes the Q-value function. However, before that instance, another learning process may reach state $s$, then take action $a^{\prime}$, and update the Q-value of action $a^{\prime}$ to be the highest value. In this case, action $a$ may still be selected by the $\epsilon$-greedy policy mentioned above. In other words, an iteration for exploitation becomes an iteration for exploration. Moreover, multiple learning processes (corresponding to multiple vehicles running on the road), which simultaneously update different non-related states, can be considered as independent processes [33]. As a result, the convergence of the proposed parallel Q-learning algorithm is guaranteed and sped up. Given the above, the convergence of the proposed parallel QL algorithm to the optimal policy is formally stated in Theorem 3.

As mentioned, the proposed parallel QL is particularly useful in vehicular networks where the system can simultaneously learn from experiences of multiple vehicles. Moreover, the high mobility of vehicles is also exploited in collecting new experiences in our proposed framework. In particular, the duration to the next decision epoch can be calculated as follows:

where $v_{t}$ is the speed of the vehicle between the $t$-th and $(t+1)$-th decision epochs. After a period of $\xi_{t}$, the vehicle can collect a new sample of experience at decision epoch $(t+1)$-th, i.e., $(s^{i}_{t+1},a^{i}_{t+1},r^{i}_{t+1},s^{i}_{t+2})$. From (20), it is clear that with high speeds, the time to move to the next decision epoch of the vehicle will be short. As such, with a given time period, with higher speeds, the vehicle can collect more experiences. Thus, the parallel QL algorithm can converge to the optimal policy faster. In the simulations below, we can observe that the convergence time reduces from $6,000$s to $1,000$s when the speed increases from 2 m/s to 9 m/s.

The proposed parallel QL algorithm is efficient with low computational complexity and memory complexity. As mentioned, the state space of our system includes only the current RSSI level, the current connected beam, and the vehicle’s velocity. For that, in common mmWave vehicular networks setting with a few mmBSs, the number of states is small, and thus the size of the global Q-table is also small. Hence, the algorithm can obtain the optimal beam association strategy quickly as the lookup and update table processes are very fast.

Regarding the computational complexity, our proposed algorithm only performs basic calculations without any complex functions as in the other reinforcement learning algorithms, e.g., deep double QL, deep QL, and deep dueling [22], [23]. These algorithms implement deep neural networks to approximate the Q-value function to obtain the optimal policy with complicated mathematical operations, e.g., multiply matrices and gradient descent. As a result, they require longer time to process and higher computing resources compared to our proposed parallel QL algorithm. In addition, in this work, we deploy only one Q-table at the eNodeB to store the Q-values for all state-action pairs instead of implementing a separated Q-tables on each vehicle with limited resources. As a result, the computing complexity is moved to the eNodeB which has sufficient resources to obtain the optimal policy in a short time.

Finally, our proposed parallel QL algorithm incurs minimal communication overhead. In particular, the RSSI level can be inferred by the mmBS through the signals received from the vehicle. Moreover, the mmBS can always know which beam the vehicle connected to. These information are sent to the eNodeB through the backhaul link with high bandwidth. Furthermore, in intelligent transport systems [35], the location information of vehicles is frequently reported to the RSU, i.e., mmBS. Thus, to update event $e_{s}$ of the SMDP, the eNodeB can collect the information of each vehicle through the mmBSs. Therefore, our proposed solution does not add additional overheads to the current ITS standards.

We consider a road with a length of $1000$ meters in the coverage of an eNodeB. On the considered road, $10$ mmBSs are deployed. Each mmBS is equipped with $10$ orthogonal beams. Each beam is assumed to cover an area (on the considered road) ranging randomly from $20$ meters to $50$ meters. The blocking probability (including both temporary and permanent blockages) of each beam is generated randomly from $0$ to $1$. We define $10$ RSSI levels for each beam corresponding to $10$ data rates ranging from $0$ to $R_{\max}=9$ Gbps, i.e., $r_{n,k}\in\{0,1,\ldots,9\}$. Unless otherwise stated, at each decision epoch, a vehicle enters the road with probability $\lambda=0.5$. The handover time is set at $0.5$ seconds [9], [31]. $z$ is set at $5$ meters. The average vehicle speed is set at $7$ m/s (about 25 km/h), which is a typical vehicle urban speed [18]. During a decision epoch, the vehicle’s speed remains unchanged. The average speed is then varied from $1$ m/s to $9$ m/s (about 4 km/h to 32 km/h) in several scenarios to demonstrate the effectiveness of the proposed solution under high mobility. It is important to note that the proposed parallel QL algorithm can learn without requiring these parameters in advance. Instead, the algorithm will learn them by interacting with the environment. For the proposed parallel QL and QL algorithms, the learning rate and discount factor are set at $0.1$ and $0.9$, respectively. Moreover, for the $\epsilon$-greedy method, the initial value of $\epsilon$ is set at $1$ and gradually reduced to $0.01$.

We compare our proposed algorithm with three other methods: (i) MaxRate, (ii) Blockage Aware, and (iii) Upper Bound.

• •

  MaxRate: This scheme first explores all available beams at the current location. Then, the beam with the highest RSSI level will be selected to connect. Once the MaxRate scheme selects the best beam, it will keep connecting to this beam until the end of the current decision epoch. This scheme is used to show the performance of non-adaptive and greedy solutions. As in mmWave systems the temporal degradation of the channel quality frequently occurs, the best beam at the current time may become worse later. Thus, this scheme results in poor system performance in terms of data rate, the number of handovers, and disconnection probability.

• •

  Blockage Aware: This scheme is assumed to know the prior knowledge about the blocking probability of all available beams in the current location. Then, the beam with the lowest blocking probability will be chosen for the vehicle to connect. This scheme is used to show the effects of the high mobility. Specifically, when the vehicle speed is low, this scheme will achieve a good system performance in terms of data rate, the number of handovers, and disconnection probability. However, its performance becomes worse when the vehicle speed is high as the high mobility is not considered.

• •

  Upper Bound: Similar to the Blockage Aware scheme, this scheme is assumed to know the prior knowledge about the blocking probability of all available beams in the current location. However, the handover decision is only made if the data rate achieved is higher than that of staying in the current beam. Note that in practice, this data rate is not available in advance and depended on the channel quality. This scheme is adopted to show the optimistic upper bound of the system performance.

The evaluation metrics are the average data rate, the disconnection probability, and the number of handovers. The average data rate is defined as the data received (in bits) by a vehicle running on the road in a second. The number of handovers is defined as the total number of handovers that a vehicle needs to do when running on the road. The disconnection probability is defined as the average probability that the vehicle cannot communicate with mmBSs.