ResLearn: Transformer-based Residual Learning for Metaverse Network Traffic Prediction

The Metaverse is a comprehensive ecosystem of interconnected virtual worlds that provide immersive experiences to users. The ecosystem enhances existing and generates new value from economic, environmental, social, and cultural perspectives [1]. Services in the Metaverse ecosystem are designed to be accessed using immersive extended reality (XR) environments. XR is an umbrella term that describes the technologies affecting the user’s immersive experience, such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) [2]. VR allows users to interact with virtually generated environments designed to simulate real-world experiences. AR overlays interactive, virtually generated information onto real-world objects or within real-world spaces. XR technologies lie on a spectrum between AR and VR. In cases where the distinction between the realities is ambiguous, the experience is considered MR. As the Metaverse’s growth continues and XR evolves, the popularity of its services increases. Driven by the rapid growth of the Metaverse, Internet traffic is expected to surpass current forecasts significantly  [3]. The entertainment and social media industries have seen the most substantial growth of Metaverse services, as evidenced by popular virtual performance events, one of which attracted an audience of 36 million users [4, 5]. Healthcare, training, and marketing for Metaverse services have also grown recently [6, 7]. Cloud rendering for Metaverse is crucial to offloading computing resources to make the services affordable, a popular technique for VR games  [8]. Consequently, the Ericsson 2022 report emphasizes the growing need for more intelligent interactions between XR services and the network to maintain high Quality of Service (QoS) [3]. Therefore, network management is crucial for Internet service providers (ISPs) to accommodate adequate resources and avoid cybersickness among users [9, 10, 11].

Metaverse traffic consists of video, audio, and control flows [8], among all downlink video frames are resource-demanding in the case of VR, and both uplink/downlink video frames for AR and MR traffic. Therefore, predicting the frame size is vital, and for latency-related issues, it is essential to predict frame inter-arrival time and frame duration [12]. Predicting frame size, inter-arrival time, and frame duration will help ISPs prepare and manage the Metaverse network for holistic and intelligent traffic management. However, there needs to be more real-world Metaverse data and research in prediction to make progress in the field. Essentially, frame-related information is time series data. The recent advent of different state-of-the-art artificial intelligence (AI) based time series models has shown tremendous progress in time series predictions [13]. The only VR frame size prediction work is available at [14]; therefore, we consider it state-of-the-art (SoA) work for benchmark comparison. The work studies different AI models to predict VR frame size. It establishes stacked LSTM to produce better results based on transfer learning methodologies. Therefore, the solution aims for online prediction, imperative for real-time network management. However, the work is evaluated on a small dataset captured in a controlled environment. Also, the performance can be improved with further reduction in the error. The frame identification methodology used in the work might need to be revised because the frame loss for 120 Mbps is more than 54 Mbps since the dataset is captured in a controlled environment. Our literature review identifies the following gaps: i) the need for holistic, comprehensive, and real-world Metaverse datasets and ii) accurate frame-related data predicting algorithms.

We chart out a state-of-the-art Metaverse testbed to capture a holistic, comprehensive, and real-world Metaverse dataset comprising VR games, VR videos, VR chat/VoIP, AR, and MR traffic. We also make it open for the research fraternity available at [15]. Our work treats the Metaverse traffic in segments to enhance the predictability of the data at higher speed. We propose an accurate view-frame (VF) algorithm that helps to identify the types of video frames using application-level information to comply with privacy-related policies. Our proposed VF algorithm can predict the number of frames in a segment, total frame size, and frame inter-arrival time. Finally, we propose a state-of-the-art residual learning (ResLearn) algorithm that uses transformer [16] to predict time-series data and fully connected neural networks (FCNN) to learn errors with a bias to identify the peaks of the frame-related data for accurate network management. Our solution outperforms the SoA work [14] by 99% in reducing the prediction errors. The implementation of the solution is made open for the research community at [17].

The system model, in Figure 1 illustrates the proposed framework for predicting Metaverse network traffic, focusing on frame-level features. The process begins with data preprocessing, where the VF algorithm is applied to extract relevant frame-related data. The key features extracted from the incoming traffic include frame count ($f^{c}$), frame size ($f^{s}$), and frame inter-arrival time (IAT) ($f^{iat}$). Our VF algorithm works on application-level features: time, packet length, packet direction, and packet inter-arrival time. Frame-related packets have a more considerable length with relatively more minor inter-arrival time. We use this property to determine the thresholds for packet length and inter-arrival time to identify frame-related packets as shown in Figure 2. In Figure 2, $len_{TH}$ is determined as 25% of the maximum length of the observed packet length. $dur_{th}$ is the frame duration threshold determined between the first two peaks. The first peak represents the start of the video frame packet with less inter-arrival time, and the second peak represents the end of the video frame when audio and control-related flows are transmitted with slightly higher inter-arrival time. The $len_{TH}$ and $dur_{th}$ are calculated from the first segment at a session’s start for the VF algorithm. All packets within these thresholds are collected to identify the frames. The method is verified on different Metaverse rendering platforms: Meta air link [18], and Virtual Desktop Streamer (VDS) [19].

Based on the requirement, one of the features is selected for prediction. These features undergo Exploratory Data Analysis (EDA) and Data Engineering to ensure data quality and integrity, followed by Time Series Data Preparation with segmentation for model training. The core of the system is frame feature prediction ($f^{x}[t]$), where the previous frame feature values ($f^{x}[t-1]$) are used as inputs to predict the current network traffic behaviour. With this information, we provide the mathematical problem statement to predict key features of the Metaverse network traffic features. Let $\bm{f}_{t}$ be a frame vector, identified by VF algorithm for a given segment, having frame-related information given as $\bm{f}_{t}=[f^{c}_{t},f^{s}_{t},f^{iat}_{t}],$ where $t$ indicates the time index of the given segment. Let $f^{x}[t-1]$ represents the Metaverse frame-related network traffic at the previous time step, where $x$ indicates one of the three frame-related features in $\bm{f}_{[t-1]}$. Therefore, frame count, size, and inter-arrival time are individually predicted using historical data. The prediction function for the current network traffic, $f^{x}[t]$, is

where $\psi(\cdot)$ is the prediction model.

The residual Learning (ResLearn) algorithm is a two-step prediction approach involving a transformer deep neural network model [16] designed to enhance Metaverse network traffic forecasting by leveraging residual learning inspired by ResNet [20]. Transformer is known for learning short and long-term dependencies from time series data. However, the prediction error is inevitable due to the randomness introduced by network health and users in Metaverse infrastructure. However, we can learn the nature of error using a neural network. ResLearn is a novel approach that uses a Transformer in the first step, given as $\mathcal{F}_{1}(\cdot)$, the predictive Transformer model. The residual from the $\mathcal{F}_{1}(\cdot)$ is fed to a fully connected neural network (FCNN) to learn the nature of the error, given as $\mathcal{F}_{2}(\cdot)$. The final prediction model from Eq. (1) is given as $\psi(\cdot)=\mathcal{F}_{1}(\cdot)+\mathcal{F}_{2}(\cdot)$.

Figure 3 illustrates the workflow of the ResLearn algorithm. The input data is split into training ($y_{\text{train}}$) and validation ($y_{\text{val}}$) sets. The first model, including a transformer network, processes the training data and generates an initial prediction result ($y_{\text{train\_pred}}$). This output, called Output 1, is then compared to the actual training data to calculate the residuals, which capture the difference between the predicted and true values. These residuals are passed to the second model, an FCNN designed to learn and predict the patterns in the residuals, producing a corrected prediction ($res_{\text{pred}}$). Both predictions from the transformer and FCNN are combined to form the final output ($y_{\text{out}}$) of the ResLearn model. This combined output is then validated with the unseen validation data, improving the final prediction’s overall accuracy. The ResLearn training algorithm is shown in Algorithm 1. $S_{N}^{X}$ is a time-series data segment, where $N$ is the segment size and $X$ is the number of segments. For each segment, the data is split into training and validation sets. The transformer model is trained on the training set, and its predictions, $T_{PR}$, are computed. The residuals, representing the error between the actual and predicted values, are calculated and adjusted by adding a bias, $Res_{B}$, to highlight essential peaks. The dense model is then trained on these residuals, and the final ResLearn model, $M_{RL}$, is created by combining the outputs of both models. The combined model is then evaluated on the validation set using error metrics. The process is repeated for each segment, refining the prediction accuracy iteratively until the validation error is stabilized for $M_{RL}$. The algorithm’s output is the final ResLearn model $M_{RL}$, which integrates the strengths of both models to deliver improved predictive performance.

The time complexity of the ResLearn algorithm, which involves training a transformer model and a FCNN, can be approximated as follows: the training complexity of the transformer model is typically $O(T\cdot N^{2})$, where $T$ is the number of training epochs and $N$ is the sequence length. The training complexity of the FCNN can be approximated as $O(T^{\prime}\cdot N\cdot D)$, where $T^{\prime}$ is the number of epochs, $D$ is the number of neurons in the hidden layer, and $N$ represents the number of training samples [21]. Considering $X$ segments of data, the overall time complexity of the algorithm is given by $O(X\cdot(T\cdot N^{2}+T^{\prime}\cdot N\cdot D)).$

Our work significantly advances Metaverse network traffic prediction by introducing a comprehensive, real-world dataset and developing novel algorithms including the view-frame and ResLearn algorithms. These algorithms enable ISPs to manage network resources in an effective manner, satisfying the QoS and enhancing the user experience for Metaverse applications. Given that our solution substantially reduces prediction errors about 99% than the SoA [14], future work can focus on expanding the dataset to cover a broader range of Metaverse applications and environments, integrating advanced AI techniques to improve prediction accuracy further, and exploring real-time deployment in diverse network architectures. Additionally, adaptive algorithms for dynamic resource allocation in response to traffic fluctuations will be investigated for enhancing the robustness in provisioning Metaverse ecosystem.