Feeling of Presence Maximization: mmWave-Enabled Virtual Reality Meets Deep Reinforcement Learning

On the aspect of designing lightweight VR HMDs, considering heavy image processing tasks, which are usually insufficient in the graphics processing unit (GPU) of a local HMD, one might be persuaded to transfer the image processing from the local HMD to a cloud or network edge units (e.g., edge servers, base stations, and access points (APs)). For example, the work in [1] proposed to enable mobile VR with lightweight VR glasses by completing computation-intensive tasks (such as encoding and rendering) on a cloud/edge server and then delivering video streams to users. The framework of fog radio access networks, which could significantly relieve the computation burden by taking full advantages of the edge fog computing, was explored in [11] to facilitate the lightweight HMD design.

In terms of proposing VR solutions with improved bandwidth utilization, current studies can be classified into two groups: tiling and video coding [12] As for tiling, some VR solutions propose to spatially divide VR video frames into small parts called tiles, and only tiles within users’ FoV are delivered to users [13, 14, 15]. The FoV of a user is defined as the extent of the observable environment at any given time. By sending HD tiles in users’ FoV, the bandwidth utilization is improved. On the aspect of video coding, the VR video is encoded into multiple versions of different quality levels. Viewers receive appropriate versions based on their viewing directions [16].

Summarily, to improve bandwidth utilization, the aforementioned works [13, 14, 15, 16] either transmit relatively narrow user FoV or deliver HD video frames. Nevertheless, wider FoV is significantly important for a user to have immersive and presence experiences. Meanwhile, transmitting UHD video frames can enhance users’ visual experiences. To this aim, advanced wireless communication techniques (particularly, millimeter wave (mmWave)), which can significantly improve data rates and reduce propagation latency via providing wide bandwidth transmission, are explored in VR video transmission [4, 17, 18]. For example, the work in [4] utilized a mmWave-enabled communication architecture to support the panoramic and UHD VR video transmission. Aiming to improve users’ immersive VR experiences in a wireless multi-user VR network, a mmWave multicast transmission framework was developed in [17]. Besides, the mmWave communication for ultra-reliable and low latency wireless VR was investigated in [18].

Although mmWave techniques can alleviate the current bottleneck for UHD video delivery, mmWave links are prone to outage as they require line-of-sight (LoS) propagation. Various physical obstacles in the environment (including users’ bodies) may completely break mmWave links [19]. As a result, VR requirements for a perceptible image-quality degradation-free uniform experience cannot be accommodated. However, the mmWave VR-related works in [4, 17, 18] did not effectively investigate the crucial issue of guaranteeing the transmission reliability of VR video frames. To significantly improve the transmission reliability of VR video frames under low-latency constraints, the coordinated multipoint (CoMP) transmission technique, which can improve the reliability via spatial diversity, can be explored [20]. Besides, it is extensively considered that proactive computing (e.g., image processing or frame rendering) enabled by adopting machine learning methods is a crucial ability for a wireless VR network to mandate the stringent low-latency requirement of UHD VR video transmission [1, 21, 19, 22]. Therefore, this paper investigates the issue of maximizing users’ feeling of presence (FoP) in their virtual world in a mmWave-enabled VR network incorporating CoMP transmission and machine learning. The main contributions of this paper are summarized as follows:

• •

  Owing to the user movement and the time-varying wireless channel conditions, we formulate the issue of maximizing users’ FoP in virtual environments as a mixed-integer and sequential decision problem, subject to power consumption constraints on APs and users’ HMDs. This problem is difficult to be directly solved by exploring conventional numerical optimization methods due to the lack of accurate users’ tracking information (including users’ locations and orientation angles) and mixed-integer and sequence-dependent characteristics.

• •

  As users’ historical tracking information is separately collected by diverse APs, a parallel echo state network (ESN) learning method is designed to predict users’ tracking information while accelerating the learning process.

• •

  With the predicted results, we develop a deep reinforcement learning (DRL) based optimization algorithm to tackle the mixed-integer and sequential decision problem. Particularly, to avoid generating infeasible solutions by simultaneously optimizing all variables while alleviating the curse of dimensionality issue, the DRL-based optimization algorithm decomposes the formulated mixed-integer optimization problem into an integer association optimization problem and a continuous power control problem. Next, deep neural networks (DNNs) with continuous action output spaces followed by an action quantization scheme are implemented to solve the integer association problem. Given the association results, the power control problem is solved to criticize them and optimize the transmit power.

• •

  Finally, the performance of the proposed DRL-based optimization algorithm is compared with various benchmark algorithms, and the impact of different design parameters is also discussed. Simulation results demonstrate the effectiveness of the proposed algorithm.

In VR applications, FoP represents an event that does not drag users back from engaging and immersive fictitious environments [24]. For wireless VR, the degrading FoP can be caused by the collection of inaccurate users’ tracking information via APs and the reception of low-quality VR video frames. Therefore, we consider the uplink user tracking information transmission and downlink VR video delivery when modelling the FoP experienced by users. Mathematically, over a period of time slots, we model the FoP experienced by users as the following

where ${{B}}_{t}^{{\rm{ul}}}\left({\bm{a}_{t}^{{\rm{ul}}}}\right)=\frac{1}{{{N}}}\sum\limits_{i\in{\mathcal{U}}}{\sum\limits_{j\in{\mathcal{J}}}{a_{ijt}^{{\rm{ul}}}}}$ with $\bm{a}_{t}^{\rm ul}=[a_{11t}^{\rm ul},\ldots,a_{ijt}^{\rm ul},\ldots,a_{NJt}^{\rm ul}]^{\rm T}$, $B_{t}^{{\rm{dl}}}\left({\bm{a}_{t}^{{\rm{dl}}}}\right)=\frac{1}{N}{\sum\limits_{i\in{{\mathcal{U}}}}{a_{it}^{{\rm{dl}}}}}$ with $\bm{a}_{t}^{\rm dl}=[a_{1t}^{\rm dl},\ldots,a_{it}^{\rm dl},\ldots,a_{Nt}^{\rm dl}]^{\rm T}$.

HMDs are generally battery-driven and constrained by the maximum instantaneous power. For any user $i$’s HMD, define $p_{it}^{\rm tot}$ as its instantaneous power consumption including the transmit power and circuit power consumption (e.g., power consumption of mixers, frequency synthesizers, and digital-to-analog converters) at time slot $t$, we then have

where $p_{it}^{\rm tot}={p_{it}}+p_{i}^{c}$, $p_{i}^{c}$ denotes the HMD’s circuit power consumption during a time slot, and $\tilde{p}_{i}$ is a constant. Without loss of generality, we assume that all users’ HMDs are homogenous.

The instantaneous power consumption of each AP is also constrained. As CoMP transmission technique is explored, for any AP $j$, we can model its instantaneous power consumption at time slot $t$ as the following

where $E_{j}^{c}$ is a constant representing the circuit power consumption, $\tilde{E}_{j}$ is the maximum instantaneous power of AP $j$.

To guarantee immersive and interactive VR experiences of users over a period of time slots, uplink user data packets should be successfully decoded, and downlink data rate requirements of users should be satisfied at each time slot; that is, users’ FoP should be maximized. According to (1) and (11), one might believe that increasing the transmit power of users’ HMDs would be an appropriate way of enhancing users’ FoP. However, as users’ HMDs are usually powered by batteries, they are encouraged to work in an energy-efficient mode to prolong their working duration. Further, reducing HMDs’ power consumption indicates less heat generation, which can enhance users’ VR experiences. Therefore, our goal is to maximize users’ FoP while minimizing the power consumption of HMDs over a period of time slots. Combining with the above analysis, we can formulate the problem of enhancing users’ immersive experiences as below

where $\bm{p}_{t}=[p_{1t},p_{2t},\ldots,p_{Nt}]^{\rm T}$.

However, the solution to (14) is highly challenging due to the unknown users’ tracking information at each time slot. Given users’ tracking information, the solution to (14) is still NP-hard or even non-detectable. It can be confirmed that (14) is a mixed-integer non-linear programming (MINLP) problem as it simultaneously contains zero-one variables, continuous variables, and non-linear constraints. Further, we can know that (9) and (13) are non-convex with respect to (w.r.t) $a_{it}^{\rm dl}$ and ${\bm{g}}_{ijt}$, $\forall i$, $j$, by evaluating the Hessian matrix. To tackle the tricky problem, we develop a novel solution framework as depicted in Fig. 4. In this framework, we first propose to predict users’ tracking information using a machine learning method. With the predicted results, we then develop a DRL-based optimization algorithm to solve the MINLP problem. The procedure of solving (14) is elaborated in the following sections.

In this section, the principle of echo state network (ESN) is exploited to train users’ location prediction model as the ESN method can efficiently analyze the correlation of users’ location information and quickly converge to obtain users’ predicted locations [25]. It is noteworthy that there are some differences between the traditional ESN method and the developed parallel ESN learning method. The traditional ESN method is a centralized learning method with the requirement of the aggregation of all users’ locations scattered in all APs, which is not required for the parallel ESN learning method. What’s more, the traditional ESN method can only be used to conduct data prediction in a time slot while the parallel ESN learning method can predict users’ locations in $M>1$ time slots. An ESN is a recurrent neural network that can be partitioned into three components: input, ESN model, and output, as shown in Fig. 4. For any user $i\in{\mathcal{U}}$, the $N_{i}$-dimensional input vector $\bm{x}_{it}\in{\mathbb{R}}^{N_{i}}$ is fed to an $N_{r}$-dimensional reservoir whose internal state $\bm{s}_{i(t-1)}\in{\mathbb{R}}^{N_{r}}$ is updated according to the state equation

where $\bm{W}_{\rm in}^{r}\in{\mathbb{R}}^{N_{r}\times N_{i}}$ and $\bm{W}_{\rm r}^{r}\in{\mathbb{R}}^{N_{r}\times N_{r}}$ are randomly generated matrices with each matrix element locating in the interval $(0,1)$.

The evaluated output of the ESN at time slot $t$ is given by

where $\bm{W}_{\rm in}^{o}\in{\mathbb{R}}^{N_{o}\times N_{i}}$, $\bm{W}_{\rm r}^{o}\in{\mathbb{R}}^{N_{o}\times N_{r}}$ are trained based on collected training data samples.

To train the ESN model, suppose we are provided with a sequence of $Q$ desired input-outputs pairs $\{(\bm{x}_{i1},{\bm{y}}_{i1}),\ldots,(\bm{x}_{iQ},{\bm{y}}_{iQ})\}$ of user $i$, where $\bm{y}_{it}\in{\mathbb{R}}^{N_{o}}$ is the target location of user $i$ at time slot $t$. Define the hidden matrix $\bm{X}_{it}$ as

The optimal output weight matrix is then achieved by solving the following regularized least-square problem

where $\bm{W}_{t}=[\bm{W}_{\rm in}^{o}\bm{W}_{\rm r}^{o}]^{\rm T}$, $\xi\in{\mathbb{R}}_{+}$ is a positive scalar known as regularization factor, the loss function $l(\bm{X}_{it}^{\rm T}\bm{W}_{t})=\frac{1}{2}||{\bm{X}_{it}^{\rm T}\bm{W}_{t}-\bm{Y}_{it}}||_{F}^{2}$, the regulator $r(\bm{W}_{t})=||\bm{W}_{t}||_{F}^{2}$, and the target location matrix $\bm{Y}_{it}=[\bm{y}_{i1}^{\rm T};\ldots;\bm{y}_{iQ}^{\rm T}]\in{\mathbb{R}}^{Q\times N_{o}}$.

Based on the principle of the ESN method, we next elaborate on the procedure of the parallel ESN learning method for users’ location prediction. To facilitate the analysis, we make the following assumptions on the regulator and the loss function.

According to Fenchel-Rockafeller duality, we can formulate the local dual optimization problem of (18) in the following way.

Denote the objective function of (21) as $D(\bm{A})$, and define $\bm{V}(\bm{A}):=\frac{1}{\xi Q}{(\bm{X}\bm{A})^{\rm T}}\in{\mathbb{R}}^{N_{o}\times(N_{i}+N_{r})}$, we can then rewrite $D(\bm{A})$ as

where $R_{j}(\bm{A}_{[j]})=\frac{1}{Q}\sum\limits_{m\in{\mathcal{Q}}_{j}}{\sum\limits_{{n}=1}^{N_{o}}{l^{\star}(-{a_{mn}})}}$, $\bm{A}_{[j]}=\hat{\bm{Z}}_{j}\bm{A}$, and $\hat{\bm{Z}}_{j}\in{\mathbb{R}}^{Q\times Q}$ is a square matrix with $J\times J$ blocks. In $\hat{\bm{Z}}_{j}$, the block in the $j$-th row and $j$-th column is a $q_{j}\times q_{j}$ identity matrix with $q_{j}$ being the cardinality of a set ${\mathcal{Q}}_{j}$ and all other blocks are zero matrices, ${\mathcal{Q}}_{j}$ is an index set including the indices of $Q$ data samples fed to slave VM $j$.

Then, for a given matrix ${\bm{A}}^{t}$, varying its value by $\Delta{\bm{A}}^{t}$ will change (24) as below

where $\Delta{\bm{A}}_{[j]}^{t}=\hat{\bm{Z}}_{j}\Delta{\bm{A}}^{t}$.

Note that the second term of the right-hand side (RHS) of (25) includes the local changes of each VM $j$, while the first term involves the global variations.

As $r(\cdot)$ is $\zeta$-strongly convex, $r^{\star}(\cdot)$ is then $\frac{1}{\zeta}$-smooth [26]. Thus, we can calculate the upper bound of $r^{\star}(V({\bm{A}}^{t}+\Delta{\bm{A}}^{t}))$ as follows

where $\bm{X}_{[j]}=\bm{X}\hat{\bm{Z}}_{j}$, $\kappa>\frac{1}{\zeta}$ is a data dependent constant measuring the difficulty of the partition to the whole samples.

By substituting (26) into (25), we obtain

From (27), we observe that the problem of maximizing $D({\bm{A}}^{t}+\Delta{\bm{A}}^{t})$ can be decomposed into $J$ subproblems, and $J$ slave VMs can then be exploited to optimize these subproblems separately. If slave VM $j$ can optimize $\Delta{\bm{A}}^{t}$ using its collected data samples by maximizing the RHS of (27), the resultant improvements can be aggregated to drive $\bm{D}({\bm{A}}^{t})$ toward the optimum. The detailed procedure is described below.

As shown in Fig. 4, during any communication round $t$, a master VM produces $\bm{V}(\bm{A}^{t})$ using updates received at the last round and shares it with all slave VMs. The task at any slave VM $j$ is to obtain $\Delta\bm{A}_{[j]}^{t}$ by maximizing the following problem

Calculate the derivative of $\Delta{D_{j}}\left({\Delta{\bm{A}_{[j]}^{t}};\bm{V}({\bm{A}^{t}}),{\bm{A}_{[j]}^{t}}}\right)$ over $\Delta{\bm{A}_{[j]}^{t}}$, and force the derivative result to be zero, we have

where $\bm{Y}_{[j]}=\hat{\bm{Z}}_{j}\bm{Y}$.

Next, slave VM $j$, $\forall j$, sends $\Delta\bm{V}_{[j]}^{t}=\frac{1}{{\xi Q}}{{{({\bm{X}_{[j]}\Delta{\bm{A}_{[j]}^{t\star}}})}^{\rm T}}}$ to the master VM. The master VM updates the global model as $\bm{V}(\bm{A}^{t}+\Delta\bm{A}^{t})=\bm{V}({\bm{A}^{t}})+\sum\nolimits_{j=1}^{J}{\Delta\bm{V}_{[j]}^{t}}$. Finally, alteratively update $\bm{V}(\bm{A}^{t})$ and $\{\Delta\bm{A}_{[j]}^{t\star}\}_{j=1}^{J}$ on the global and local sides, respectively. It is expected that the solution to the dual problem can be enhanced at every step and will converge after several iterations.

At time slot $t$, based on the above derivation, the parallel ESN learning method for predicting locations of user $i$, $\forall i$, in $M$ time slots can be summarized in Algorithm 1.

Define a vector $\bm{g}_{it}=[\bm{g}_{i1t};\ldots;\bm{g}_{ijt};\ldots;\bm{g}_{iJt}]\in{\mathbb{C}}^{JK}$ and a vector $\bm{h}_{it}=[f_{i1t}\bm{h}_{i1t};\ldots;f_{ijt}\bm{h}_{ijt};$ $\ldots;f_{iJt}\bm{h}_{iJt}]\in{\mathbb{C}}^{JK}$, $\forall i$, $t$. Let matrix $\bm{G}_{it}=\bm{g}_{it}\bm{g}_{it}^{\rm T}$ and matrix $\bm{H}_{it}=\bm{h}_{it}\bm{h}_{it}^{\rm T}$. As ${\rm tr}({\bm{AB}})={\rm tr}({\bm{BA}})$ for matrices $\bm{A}$ and $\bm{B}$ of compatible dimensions, the signal power received by user $i\in{\mathcal{U}}$ can be expressed as ${{{{|\sum\nolimits_{j\in{\mathcal{J}}}{{f_{it}}{\bm{h}_{it}^{\rm T}}{\bm{g}_{ijt}}}|}^{2}}}}={|{{\bm{h}}_{it}^{\rm T}{{\bm{g}}_{it}}}|^{2}}={\left({{\bm{h}}_{it}^{\rm T}{{\bm{g}}_{it}}}\right)^{\rm T}}{\bm{h}}_{it}^{\rm T}{{\bm{g}}_{it}}={\rm tr}({\bm{g}}_{it}^{\rm T}{{\bm{h}}_{it}}{\bm{h}}_{it}^{\rm T}{{\bm{g}}_{it}})={\rm tr}({{\bm{h}}_{it}}{\bm{h}}_{it}^{\rm T}{{\bm{g}}_{it}}{\bm{g}}_{it}^{\rm T})={\rm tr}({{\bm{H}}_{it}}{{\bm{G}}_{it}})$. Likewise, by introducing a square matrix $\bm{Z}_{j}\in{\mathbb{R}}^{JK\times JK}$ with $J\times J$ blocks, the transmit power for serving users can be written as ${{\bm{g}}_{ijt}^{\rm T}{{\bm{g}}_{ijt}}}={\rm tr}({\bm{Z}}_{j}{{\bm{G}}_{it}})$. Besides, each block in $\bm{Z}_{j}$ is a $K\times K$ matrix. In $\bm{Z}_{j}$, the block in the $j$-th row and $j$-th column is a $K\times K$ identity matrix, and all other blocks are zero matrices. Then, by applying ${\bm{G}_{it}}={\bm{g}_{it}}{\bm{g}}_{it}^{\rm T}$ $\Leftrightarrow{\bm{G}_{it}}\succeq 0$ and ${{\rm rank}({\bm{G}_{it}})\leq 1}$, we can convert (14) to the following problem