Technical Report for Trend Prediction Based Intelligent UAV Trajectory Planning for Large-scale Dynamic Scenarios

Consider an area of interest (AoI) where the explosive growth of access requirements have already far outstripped the capacity of existing terrestrial base stations. One UAV with velocity $v_{uav}$ is dispatched to provide the extra communication capacity for the GUs inside the AoI (denoted as a GU set $\Omega_{all}^{t}$) with fixed flight altitude $H$. In particular, the entire mission period of the UAV is discretized into multiple individual time slots and each with equal duration $\tau$. Similarly, referring to [9], the AoI has been divided into $K\times K$ equal grids, of which the centers are used as way-points of the UAV in each time slot. Thus, the location of the UAV (projected on the ground) and the $i$-th mobile GU in time slot $t$ can be formulated as $\mathbf{L_{u}^{t}}=[x_{u}^{t},y_{u}^{t}]$ and $\mathbf{L_{i}^{t}}=[x_{i}^{t},y_{i}^{t}]$, respectively. Following the model proposed in [9], the velocity and moving direction of the GU $i$ will be updated as

where $\overline{v}$ and $\widetilde{\theta}$ denote average velocity and steering angle, respectively, and $k_{2}$ follows $\epsilon$-greedy model^†††In each time slot, the GU will keep the same moving direction with probability $\epsilon$, otherwise the GU chooses one of the remaining directions randomly..

Since the terrestrial infrastructures cannot provide communication services, the GUs’ data will be temporarily stored in their on-board buffer and wait for the UAV to start the data upload process. Here, the GU-to-UAV channel is modeled as Rician models [1] that can capture the shadowing and small-scale fading effects due to multi-path propagation, in which the channel coefficient from the GU $i$ to the UAV in time slot $t$ can be expressed as

where $\beta_{i}^{t}=\frac{\alpha}{(H^{2}+||\mathbf{L_{u}^{t}}-\mathbf{L_{i}^{t}}||^{2})^{k_{ps}/2}}$ and $\hat{h_{i}^{t}}=\sqrt{\frac{k_{s}}{k_{s}+1}}\tilde{h_{i}^{t}}+\sqrt{\frac{k_{s}}{k_{s}+1}}\tilde{\tilde{h_{i}^{t}}}$, $|\tilde{h_{i}^{t}}|=1$, $|\tilde{\tilde{h_{i}^{t}}}|\sim CN(0,1)$, and $K_{ps}$ and $K_{s}$ denote the GU-to-UAV path loss component and Rician factor, respectively. $\alpha$ denotes the channel power gain at the reference distance $||\mathbf{L_{u}^{t}}-\mathbf{L_{i}^{t}}||=1$m.

By adopting OFDMA technology, the UAV can pre-divide communication resources into multiple equal and orthogonal resource blocks in advance, of which the bandwidth allocated to each communication GU is $W$. In this way, the UAV using OFDMA technology can support concurrent data transmissions with surrounding GUs. Thus, the communication rate with respect to each GU is given by $r_{i}^{t}=W\log_{2}(1+\frac{|h_{i}^{t}|^{2}p}{\sigma^{2}})$, where $p$, $W$, and $\sigma^{2}$ denote the transmission power, bandwidth and noise power, respectively. In this way, the data uploaded throughput of the $i$-th GU is obtained by $r_{i}^{t}\tau_{c}$, in which $\tau_{c}$ denotes the duration of the hovering time used for communications within one time slot, which is assumed as $\tau\gg\tau_{c}$. Here we denote the data queue length of the GU $i$ in time slot $t$ as $B_{i}^{t}$, which depends on both the newly generated data (i.e. $I_{i}^{t}$) and the data uploaded (i.e. $r_{i}^{t}\tau_{c}$) to the UAV, which is given by $B_{i}^{t}={B_{i}^{t-1}+I_{i}^{t}-r_{i}^{t}\tau_{c}}$.

In addition, the energy consumption of the UAV in time slot $t$ is associated with the flying distance of the UAV in the past time slot [9], which is formulated as

where $p_{f}$ denotes the UAV flying power.

Regarding the dynamic scenarios considered in this work, the location, number, and data queue of GUs that change in real time greatly increases the feature dimension of the UAV-Ground communication scenario, which poses a severe challenge to build the efficient and reliable data links between GUs and the UAV. Therefore, our objective is to find a policy that helps the UAV make the optimal decision on trajectory planning, so that the utility of the UAV-enabled communication system over all the time slots is maximized, subjected to the coverage range, energy consumption, and the communication QoS. Thus, the UAV trajectory planning issue^‡‡‡We summarize the notations used in this paper in section II of our technical report [Wang2021tech]. is formulated as

where the system utility function $\mathcal{U}^{t}$ depends on both the throughput and fairness during the data upload processes. Furthermore, C1 ensures that the communication link should meet the required QoS, namely the channel coefficient of each communication GU $i$ (see (3)) should not be smaller than the threshold $\underline{h}$. C2 limits the total energy consumption during the UAV flight process $\sum_{t}E_{f}^{t}$ (see (4)) to within the portable energy $\overline{E}$ under the premise that the communication energy consumption is negligible.

Furthermore, the utility function $\mathcal{U}^{t}$ is defined here, $\mathcal{U}^{t}=f_{G}^{t}\sum_{i\in\Omega_{all}^{t}}r_{i}^{t}\tau_{c}$, in which $f_{G}^{t}$ called as the Jain’s fairness index [3] is defined as

in which $c_{i}$ denotes the number of UAV-enabled communication services that GU $i$ has participated in. As a widely-used metric for fairness, the value of $f_{G}^{t}$ approaches $1$ when the total number of time slots that each GU is served are very close, which can be regarded as a measure of the fairness of UAV communication services in existing scenarios.

In general, conventional methods normally would narrow down P0 for an individual time slot, namely $\max U^{t},\forall t\in\mathcal{T}$, and solve it by classical convex optimization methods or heuristic algorithms, which may obtain the greedy-like performance due to lacking of a long-term target. To tackle above issue, we apply Markov Decision Process (MDP), defined as a tuple ($\mathcal{S}$, $\mathcal{A}$, $\mathcal{R}$, $\mathcal{P}$), to model the UAV trajectory planning problem in large-scale dynamic scenarios, which are detailed as follows.

1) The $state$ $\mathcal{S}$ contains the locations of the UAV and GUs as well as their real-time status, which is formulated as

2) The $action$ $\mathcal{A}$ contains available actions of the UAV in each time slot, which is formulated as

3) The $reward$ $\mathcal{V}_{\pi}(s_{t})$ represents the discounted accumulated reward from the state $s_{t}$ to the end of the task with the policy $\pi$, which is formulated as

where $r(s_{t},a_{t})$ denotes the immediate reward through executing action $a_{t}$ at the state $s_{t}$, and $\mathcal{J}$ denotes the end of the task. In particularly, the action $a_{t}$ adopted here is selected following the policy $\pi$, i.e., $\pi(s_{t})=a_{t}$. According to problem P0, $r(s_{t},a_{t})$ is defined as

4) The $transition$ $probability$ $\mathcal{P}\triangleq\{p(s_{t+1}|s_{t})\}$ represents the probability that the UAV reaches the next state $s_{t+1}$ while in the state $s_{t}$, which is formulated as

where $\eta$ corresponds to the greedy coefficient during the action selection process.

Based on the formulated MDP model ($\mathcal{S}$, $\mathcal{A}$, $\mathcal{R}$, $\mathcal{P}$), the UAV first observes the state $s_{t}$ in each time slot $t$. Then, it takes an action $a_{t}$ following the policy $\pi$, i.e., $a_{t}=\pi(s_{t})$, and thus obtains the corresponding immediate reward $r(s_{t},a_{t})$. Then, the UAV moves to the next way-point and the $state$ updates to $s_{t+1}$. Therefore, the problem P0 can be transformed into maximizing the discounted accumulated reward by optimizing the policy $\pi$.

A table that summarizes all notations in this paper is given in Table. 1.

It is well-known that the DRL algorithm is efficient and effective in solving MDP for the uncertain (i.e. $\mathcal{V}_{\pi}(s_{t})$) and complex (i.e. $\mathcal{S}$) system, in which the critical issue here is how to obtain the optimal policy $\pi^{*}$ for the action selection process.

To achieve that, we first rephrase $\mathcal{V}^{\pi}(s_{t})$ into the form of state-action pairs as $\mathcal{V}^{\pi}(s_{t})=Q^{\pi}(s_{t},a_{t})$ where $a_{t}=\pi(s_{t})$ means the action $a_{t}$ is selected at the state $s_{t}$ according to the policy $\pi$. Here we call $Q^{\pi}(s_{t},a_{t})$ as the Q-function. Referring to [9], it is easy to draw the conclusion that the optimal policy can be derived by $\pi^{*}(s_{t})=\mathop{\arg\max}\limits_{a\in\mathcal{A}}Q(s_{t},a)$. In other words, once the UAV selects the action that can maximize the corresponding Q-function at each $state$ $s_{t}\in\mathcal{S}$, the optimal policy can be realized. Then, the remaining issue is how to obtain $Q(s_{t},a_{t})$ with the given environment $s_{t}$ and action $a_{t}$.

Based on above conclusions and (8), we have

where $Q^{\pi^{*}}(s_{t+1},\pi^{*}(s_{t+1}))=\mathop{\max}\limits_{a\in\mathcal{A}}Q^{\pi^{*}}(s_{t+1},a)$, thus the issue about searching the Q-function $Q^{\pi^{*}}(s_{t},a_{t})$ can be formulated as a regression problem, that is, through iteratively optimizing the parameters of the Q-function so that the left-hand side of the (9) is infinitely close to the right-hand side. Here, considering the high dimension of $\mathcal{S}$ and $\mathcal{A}$ in the scenario we studied, it is common to apply the neural networks (NN) to fit the Q-function mentioned above. To be specific, the NN can address the sophisticate mapping between the $state$ $s_{t}$, $action$ $a_{t}$ and their corresponding Q-function value $Q(s_{t},a_{t})$ based on a large training data set. Accordingly, two individual deep neural networks are established here. One with parameters $\theta_{P}$ is utilized to construct the Evaluation network $Q_{\theta_{P}}(s_{t},a_{t})$ that models the function $Q^{\pi^{*}}(s_{t},a_{t})$, and another one with parameters $\theta_{T}$ is used to construct the Target network $Q_{\theta_{T}}(s_{t},a_{t})$ for obtaining the target value (i.e., $r(s_{t},a_{t})+\gamma Q^{\pi^{*}}(s_{t+1},\pi^{*}(s_{t+1}))$) in the training process. Finally, the parameters $\theta_{P}$ of $Q_{\theta_{P}}(s_{t},a_{t})$ are updated by minimizing the loss function $L$, which is formulated as

Finally, after the NN has been well trained, the UAV can make a decision at the state $s_{t}$ according to the obtained Q-function that is modeled as the NN, and then the UAV takes the action $a_{t}=\arg\max_{a\in\mathcal{A}}Q_{a}(s_{t},a)$

In order to extract spatial features of the scenario, here we adopt Convolutional Neural Networks (CNNs) to achieve the Evaluation network $Q_{\theta_{P}}(s_{t},a_{t})$, which sets a three-channel tensor with size $\mathbb{R}^{K\times K\times 3}$ as the input. Meanwhile, each channel of the tensor is modeled as a matrix with the size $\mathbb{R}^{K\times K}$, corresponding to the scenario model that is divided into $K\times K$ equal grids. (see the system model in Section. II-A). Here, two convolution layers are constructed to extract the features from the input, and two full connected layers are constructed to establish associations between them. The design of the three-channel tensor are given as follows.

In order to enable the UAV to update the DRL model at the same time during executing the communication task, a training framework that combines online and offline manner is proposed, in which the UAV performs the task following the policy obtained at the offline stage, and then updates the its policy online in current episode.

The offline learning process is designed to learn the most valuable experiences from the past. Therefore, a large ”Replay Memory” (denoted as $R_{l}$) is constructed to store the experiences including $state$ $s_{t}$, $action$ $a_{t}$, and reward $\mathcal{U}^{t}$ during the past episodes, of which seventy percent are with the largest reward and the remaining thirty percent are chosen randomly from the rest. In contrast, the online learning process aims to learn from the current experiences, resulting in the current best behavioral decisions. Therefore, it will generate a relative small ”Replay Memory” (denoted as $R_{s}$) to store the experiences sampled during the current communication task, of which eighty percent are with the largest reward from the current episode and the remaining twenty percent are picked randomly.

The combination of the above two training processes not only allows the UAV to take full advantage of past experiences, but also adjust its policy according to the current actual situation, and sum up all past experiences after the end.

The computational complexity of our proposed moving Trend Prediction (TP) based DRL framework is mainly involved in two parts, namely the building process of the channel $3$ and the CNN calculation process. The former can be obtained as follows: step $1$, $2$ and $3$ only involve simple calculations, step $4$ requires $O(4\times|\Omega_{all}^{t}|)$ and step $5$ requires $O(8\times N\times|\Omega_{all}^{t}|)$. Then, the computational complexity of the CNN calculation process is given by $O(N_{c}\times M_{c}^{2}\times K_{c}^{2}\times C_{in}\times C_{out})$ [14] in which $N_{c}$=$2$ $M_{c}$=$32$, $K_{c}$=3, $C_{in}$=2 and $C_{out}$=32 are the parameters of the model in our scheme.