Virtual Reality in Metaverse over Wireless Networks with User-centered Deep Reinforcement Learning

We first introduce the computation offloading model based on the wireless cellular network. The server VSP will manage the downlink channels $\mathcal{M}$ of all VUs $\mathcal{N}$. And then, the server VSP Furthermore, we denote $D_{n}^{t}$ ($n\in\mathcal{N}$) as the size of the virtual scene frame at time step $t$ that needs to be transmitted to user $n$.

We adopt the Non-Orthogonal Multiple Access (NOMA) system as this work’s propagation model. In NOMA system, several users can be multiplexed on one channel by the successive interference cancellation (SIC) and superposition coding, and the received signals of VUs in channel $m$ are sorted in ascending order: $p_{1}|h_{1,m}^{t}|^{2}>p_{2}|h_{2,m}^{t}|^{2}>...>p_{N}|h_{N,m}^{t}|^{2}$ [11]. In this paper, we assume the decoders of the VUs can recover the signals from each channel through SIC. We denote $h_{i,m}^{t}$ as the channel gain between the VSP and the $n^{th}$ user allocated to channel $m$ at time step (iteration) $t$. The Downlink rate can be expressed as [11]:

$P_{d}=\{p_{1},p_{2},...,p_{N}\}$ denotes the transmission power of each VU’s device. Note that transmission power is not time-related in our scenario. $h_{n,m}^{t}=g_{n,m}^{t}l_{n}^{-\alpha}$ denotes the channel gain between VU $n$ and VSP in channel $m$, with $g_{n,m}^{t}$, $l_{n}$, $\alpha$ being the Rayleigh fading parameter, the distance between VU $n$ and VSP, and the path loss exponent, respectively. $W$ is the bandwidth of each channel, and $W\sigma^{2}$ denotes the background noise. Accordingly, the total delay $d_{n,o}$ of each frame in time step $t$ is divided into (1) Execution time and (2) downlink transmission time:

where $f_{v}$ is the computation capability of VSP, and $C_{n}^{t}$ is the required number of cycles per bit of this frame [14].

When VU is not allocated a channel, it needs to generate the virtual world frames locally at the expense of energy consumption. Let $f_{n}$ be the computation capability of VU $n$, and it varies across VUs. Adopting the model from [15], the energy per cycle can be expressed as $e_{n,cyc}=\eta f_{n}^{2}$. Therefore, the overhead of local computing in terms of execution delay and energy can be derived as:

where $\mu_{n}$ is the weighting parameter of energy for VU $n$. The battery state of each VU can be different, then, we assume that $\mu_{n}$ is closer to $0$ with a higher battery.

With the slotted structure, we set the VUs’ maximum tolerable delay to be $\iota$ for every frame to be the problem constraint. Different users have different purpose of use (video games, group chat, etc.). Thus, they also have varying expectations of satisfactory number of frames per second $\tau_{n,F}$. We set the tolerable frame transmission failure count of VU $n$ as $\tau_{n,f}$. Initially, the tolerable frame transmission failure count of VU $n$ is defined as $\tau_{n,f}^{0}=T-\tau_{n,F}$. For each successive frame, the delay in exceedance of tolerable threshold leads to a decrease in VU’s remaining tolerable count: $\tau_{n,f}^{t+1}=\tau_{n,f}^{t}-I_{n}^{t}$, where

Our goal is to find the near-optimal channel arrangement for the transmission of $T$ frames, to minimize the total frame transmission failure count and VU device energy consumption.

The $\omega_{1},\omega_{2}$ are the weighting parameters of delay and energy. Constraint $C1$ ensures that the frame transmission failure count of each user is within their tolerable limit. Constraint $C2$ is our integer optimization variable which denotes the computing method and channel assignment for each user at every time step.

This formulated problem is sequential, where the remaining tolerable frame transmission failure count $\tau_{n}^{f,t}$ of each user changes over time, and influences the following states. Thus, convex optimization methods are unsuitable for our proposed problem due to the huge space of integer variables and daunting computational complexity. Also, as the problem contains too many random variables, model-based RL approaches which require transition probabilities are infeasible techniques to tackle our proposed problem. We next introduce our deep RL environment settings according to the formulated problem.

We included the following attributes into the state: (1) Each VU’s virtual world frame size: $D_{n}^{t}$. (2) Each VU’s remaining tolerable frame transmission failure count: $\tau_{n,f}^{t}$. (3) The channel gain of each VU: $h_{n,m}^{t}$. (4) The remaining number of frames to be transmitted at each time step: $(T-t)$.

The discrete action channel assignment to each VU is:

In practice, we use a tuple in which there are $N$ elements corresponding to $N$ users and each element can take $M+1$ values, which corresponds to the number of channels; plus 1 for a user being assigned to perform local computing. However, we need to encode the discrete actions as discrete numbers to be evaluated by the neural network. The encoding method is shown in Fig. 2.

As the main objective is to minimize the frame transmission failure counts and energy consumption, the overall reward $R_{n}^{t}$ for each VU contains: (1) a penalty $R_{n,f}^{t}$ for every frame transmission failure and (2) a weighted reward $R_{n,e}^{t}$ for energy consumption corresponding to VU’s battery life. To implement the tolerance constraint $C1$, we give (3) a huge penalty $R_{n,end}^{t}$ corresponding to the number of frames left to be transmitted when any VU’s remaining tolerable frame transmission failure count is $0$. In the circumstance of (3), the episode ends immediately.

In contrast to decomposing the overall reward into separate sub-goal rewards as done in [16], we built a user-centered reward decomposition architecture as an extension to PPO, Hybrid Reward PPO (HRPPO), which takes in the rewards of different users and calculates the actions-values separately. In other words, we give the network a view of the state-value of each user, instead of merely evaluating the overall value of an action based on an overall state-value.

Function process: In each episode, when the current transmission is accomplished with the selected action $a_{t}$, the environment will issue rewards $R_{1}^{t},R_{2}^{t},...,R_{n}^{t}$ as feedback to different VUs. These rewards along with their corresponding states and next iteration state, will be sent to the Critic to generate the state-values $V_{1}^{t},V_{2}^{t},...,V_{n}^{t}$, representing the state-value of each VU. The state-value is then used to calculate the advantages and losses for each VU. The above-mentioned process is illustrated in Fig. 3.

Update function: In equation (13), we established the policy gradient for PPO Actor, and in HRPPO we have the gradient $\Delta\theta$ as:

where $A_{n}^{t}$ denotes the advantages of different VUs. The generalized advantage estimation (GAE) [18] is chosen as the advantage function:

$\bar{T}$ specifies the length of the given trajectory segment, $\gamma$ specifies the discount factor, and $\lambda$ denotes the GAE parameter. In terms of Critic loss, the equation (14) is formatted into:

Similar to the renowned centralized training decentralized execution (CTDE) framework [19], the $V_{\phi}^{n}$ also uses centralized training with equation (18). Therefore, the training time will not scale with the number of users.

We introduce a set of metrics (apart from RL rewards) to evaluate the effectiveness of our proposed methods.

• •

  Successful frames. The number of successful frames among total $T$ frames determines the Frame Rate per Second (FPS) of the virtual world scenes, and hence fluidity of the Metaverse VR experience.

• •

  Energy consumption. We illustrate the total energy consumption in each episode. Lower energy consumption signifies a more effective use of channel resources.

• •

  Average rate. The average downlink transmission rate of all VUs and frames in each episode is shown to evaluate the trained policy. A higher average rate indicates better allocation of channel resources.

Consider a $30\times 30$ $m^{2}$ indoor space where multiple VUs are distributed uniformly across the space. We set the number of channels to be $3$ in each experiment configuration, and the number of VUs to be from $5$ to $8$ across the different experiment configurations. The maximum resolution of one frame is 2k ($2048\times 1080$) and the minimum is 1080p ($1920\times 1080$). Each pixel is stored in 16 bits [20] and the factor of compression is 150 [13]. We randomize the data size of one frame to take values from a uniform distribution in which $D_{n}^{t}\in[\frac{1920\times 1080\times 16}{150},\frac{2048\times 1080\times 16}{150}]$. The flashed rate, $T$ frames in one second is taken to be 90, which is considered the best rate for VR [13] applications. The bandwidth of each channel is set to $10\times 180$ kHZ. The required successful frame transmission count $\tau_{n,F}$ is uniformly selected from $[75,80]$, which is higher than the acceptable of $60$ [13]. In terms of channel gain, the small-scale fading follows the Rayleigh distribution and $\alpha=2$ is the path loss exponent. For all experiments, we use $2\times 10^{5}$ steps for training, and the evaluation interval is set to be $50$ training steps. As there are several random variables in our environment, all experiments are conducted under global random seeds from 0-10, and the error bands are drawn to better illustrate the model performances.

We first illustrate the performances of the different models against different metrics in two experimental configurations (shown in Fig. 4): one with 6 VUs and the other with 8 VUs. We then show the overall results for each experimental configurations in Table  I. Results in Table  I are taken from the average of the final $200$ steps.

The training reward, successful frame transmission counts, and average downlink transmission rate show an overall upward trend as training progresses. When pitted against these metrics, HRPPO performed the best out of the tested baseline algorithms. In the experimental setting with $6$ VUs, although PPO and HRPPO are able to attain similar peak rewards towards later training stages, HRPPO converges in half the number of training steps taken for PPO to achieve convergence. In the experimental setting with $8$ VUs, HRPPO obtains a much higher final reward when compared to PPO. Both HRPPO and PPO achieved higher rewards than HRDQN, and performed better in each metric. HRPPO and PPOs’ performance superiority can be attributed to the PPO’s policy KL penalty and higher sample efficiency. However, HRDQN and PPO fail to find a good solution in more complicated scenarios. In the $6$ VU experimental setting, both HRPPO and PPO are able to allocate VUs to a VSP channel for computation offloading in each round, and this is reflected in zero energy spent on local device computation. However, in the experimental setting with $8$ VUs, there is insufficient channel resources and all of the three algorithms learn strategies to increase transmission rate and avoid frame rate decrement, by allocating some VUs to perform local computation, which increases energy consumption.

The complete results in Table. I shows that HRPPO obtains the best performance for almost every metric under every scenario. This demonstrates that decomposing the reward and using sum-losses which provides a user-centered view to the RL agent, is good approach to tackling a multi-user computation offloading problem.