6G-enabled Edge AI for Metaverse: Challenges, Methods, and Future Research Directions

For the past decade, the way users interact with virtual worlds has been limited by screens and mobile devices. However, with the continuous development of Metaverse technology, the fusion between the physical world and the virtual world has largely alleviated this limitation. Specifically, in the Metaverse, the “stimulus” received by the avatar can be fed back to the user in the physical world with high fidelity through XR technology and intelligent wearable devices such as brain-computer interfaces (BCI)^{Schalk et al. (2004)}. For example, users can wear virtual reality (VR) glasses and augmented reality (AR) headsets to enter the Metaverse for remote video conferences, realizing the transformation from the 2D application interface of the physical world to the 3D simulation space of the virtual world. It can improve the user’s immersive experience^{Segawa et al. (2020); Qiao et al. (2019)}.

The Metaverse contains a variety of advanced digital technologies and AI technologies, such as digital twin^{Rasheed et al. (2020)}, XR, blockchain^{Feng et al. (2020)}, computer vision, and so on. Specifically, digital twin technology enables large-scale, high-fidelity real-time mapping of physical entities, allowing users to simulate and test physical entities in the Metaverse^{El Saddik (2018)}, XR technology uses mixed reality (MR) and AR to ensure the immersion of the Metaverse, and blockchain technology builds the economic system in the Metaverse.

Metaverse integrates multiple application scenarios, and some functions between the scenarios are interoperable^{Wang et al. (2022a)}. For example, services such as virtual clothes and virtual feelings purchased by users can be used in various scenarios of the Metaverse, reflecting interoperability.

The Metaverse is a new stage in the development of human social formations. Humans can pursue some higher-level needs beyond the physical world level in the Metaverse, such as virtual office, virtual entertainment, and other social behaviors. Therefore, sociality is an essential characteristic of communication between users in the Metaverse.

As a digital world, the Metaverse is different from the physical world and has longevity characteristics. Specifically, the user’s avatar, behavior, assets, and other data can be imprinted and stored in the Metaverse for a long time. However, in the real world, when an organization fails, or an individual dies, many things related to it will disappear with it. In contrast, users’ personal information and digital assets are permanently stored in the Metaverse.

1) Users: Users can interact with the virtual world of the Metaverse through intelligent devices such as VR, head-mounted displays (HMDs), AR goggles, and human gesture sensors^{Xu et al. (2022)}. In the virtual world, users can immensely experience some applications, including custom identity, social interaction, entertainment, and learning^{Ondrejka (2004)}. Specifically, in the Metaverse, users can redefine their digital image (i.e., 3D avatar), not limited by gender, appearance, and national boundaries. Meanwhile, users can immensely experience various virtualized scenarios in the Metaverse (e.g., virtual performances, virtual dating, 3D telecommuting) to meet their diverse needs. Therefore, users interact in the physical-virtual world with the characteristics of diversification of information, identities, and modes. These interactive behaviors ultimately give users an all-around immersive experience.

2) Smart Wearable Devices: Users upload recorded data and historical behaviors from the real world to the Metaverse platform by smart wearable devices (e.g., smartwatches, smart wearable clothes, etc.). Meanwhile, the users can directly get some feedback from the avatar’s behavior through the smart wearable devices, realizing the interaction between the physical world and the virtual world^{Chen et al. (2017)}. Thus, smart wearable devices are an important medium for users to realize inter-operation between the physical and virtual worlds.

3) Physical Service Providers: The physical service provider is the support and maintenance of the Metaverse by the technology of the physical world, including the communication at the edge of the network and the allocation of computing resources. When users wear VR smart devices and enter the Metaverse, the physical service network operator will respond to the user’s needs in the virtual world in real-time. For example, the reasonable allocation of network and communication resources is used to reduce the delay of interaction between the physical world and the virtual world^{Lim et al. (2022a)}. Moreover, the data generated by users in the Internet sensor network of the physical world will also be fed back to the virtual world for maintenance and utilization in real-time. Therefore, the normal operation of the Metaverse cannot be separated from the support of physical service providers.

As shown in Fig. 1, the virtual layer of the Metaverse includes virtual environments, virtual services, virtual currencies, and all behaviors of avatars in the virtual world. In the virtual layer, technologies such as high-definition rendering and simulation are used to develop and maintain the virtual spaces of the Metaverse (such as social networks, economic systems, and business activities) so that users can have an immersive, high-quality experience^{Guo et al. (2020)}. Similar to shopping in a real-world mall, users can purchase virtual goods such as paid videos and clothing from good providers in the Metaverse for an interactive experience^{Liu et al. (2019a)}.

In this section, we introduce some techniques applied to the Metaverse. A summary has been given in Tab. 3. Next, as shown in Fig. 1, we describe the support each technology provides to the Metaverse.

• •

  Digital Twin: The digital twin is a technique that can convert physical entities to virtual and digital models through information technology. In the process of modeling, digital twin technology can map the composition, characteristics, functions, and performance of physical entities into the Metaverse in real-time^{Wu et al. (2021)}. Thus, digital twins and physical entities in the Metaverse are nearly equivalent^{Liu et al. (2019b)}. This largely facilitates the fusion of the virtual and physical worlds.

• •

  Blockchain: All objects in the Metaverse, such as digital twins, avatars, and complex maps, exist as digits. However, this inevitably requires servers’ storage capacity because of massive data. Meanwhile, the traditional centralized architecture cannot upload all the data to the cloud servers due to limited network resources. On the other hand, the Metaverse system based on centralized architecture may lead to some risks such as monopoly and illegal control^{Xu et al. (2018)}. To be more specific, a company in a monopoly position holds a large amount of personal data about its users. Once the company’s servers are attacked, many users’ privacy will likely be leaked. Blockchain, where data are stored in blocks in chronological order and the blocks are sequentially linked to form a chained data structure, is a distributed shared database^{Kang et al. (2019a, b)}. More importantly, blockchain also has its characteristics of being tamper-proof and non-modifiable^{Mozumder et al. (2022)}. Thus, the distributed database stored in the blockchain can effectively alleviate the problems of limited storage resources and privacy leakage, which the centralized architecture faces in the Metaverse.

• •

  Network Communication: In the Metaverse, users’ immersive experience, virtual office, and perceptual communication rely on high bandwidth, low latency, and high-reliability network communication technologies. Specifically, VR devices require a transmission rate beyond 250 $Mbit/s$ and data error rate between $10^{-1}$ and $10^{-3}$ to prevent communication interruptions^{Lincoln et al. (2016)}. Haptic communication requires a lower data transfer rate at least $1$ $Mbit/s$ and a latency of no more than 1 $ms$^{Ivkovic et al. (2015)}. These requirements can be achieved with enhanced mobile broadband and ultra-reliable and low-latency communication technologies^{Rappaport et al. (2019)}.

• •

  Decentralization: In the Metaverse, due to security and privacy issues, it is not allowed to store all users’ data on a single server. It easily causes the whole Metaverse system to go down, and the user’s private data leaked because of the server’s breakdown. In contrast, in the decentralized storage system, every involved organization can become a storage node^{Frey et al. (2008)}. Therefore, the absolute power “king” of the Metaverse does not exist in the decentralized storage system, and no one can have absolute power to control all users’ data. Even Meta, a giant technology company, can only act as a builder in the Metaverse. In particular, many companies are all representatives of decentralized storage systems in the Metaverse, such as Filecoin, Storj, Arweave, and MEM. Specifically, they all dispersedly store slices of data on multiple independent network nodes and then establish an economic incentive mechanism to enable users to store data for a long time^{Xu et al. (2021)}.

• •

  Computer Vision: Computer vision is a technology that interacts between the digital world and the real world, laying the foundation for image processing for the realization of XR^{Huynh-The et al. (2022)}. Some applications in the Metaverse, such as visual localization and mapping, human pose and eye tracking, overall scene recognition and understanding, and picture recovery and enhancement, mainly apply computer vision. For instance, in Fig. 1, visual localization and mapping are like human beings in the physical world perceive everything with their eyes and then map them to the brain to reconstruct a three-dimensional world. Similarly, in the Metaverse, the avatar also needs to obtain the three-dimensional structure of the unknown environment and perceive the exact movement position of the object.

Metaverse builds immersive education scenarios through digital twin technology. Specifically, students can promote their understanding of learning content through immersive learning in virtual scenarios. Immersive education already has many application platforms in the Metaverse. For example, Hoodoo Labs^{Hong et al. (2017)} is a learning English platform that has transposed more than 300 characters and above 4,300 scenarios that English conversations can apply to virtual reality scenarios. And then, users can improve their English skills through immersion, even though they travel to any continent or village. Moreover, Virbela, as a virtual world platform created to address collaboration in distance learning courses, was used by Davenport University to create a custom virtual campus named “Davenport Global” during the COVID-19 pandemic. Specifically, “Davenport Global” preserves the physical campus’s classroom culture and practical experiences, such as interactive auditoriums, presentation screens, private tutorial rooms, and free space for socializing and learning, enabling students to gain a sense of belonging just like a real campus.

Under the influence of COVID-19, many applications, such as telecommuting and videoconferencing, have gained universal popularity in the latest two years. However, business-to-business (B2B) or customer-to-customer (C2C) videoconferencing brings some problems such as monotony and lack of spatiality. The emergence of the Metaverse has enriched the telecommuting and videoconferencing scenarios. Specifically, in Fig. 2, the offices, conference rooms, and tables have been built in the Metaverse to solve the lack of spatial sense. The monotony problem has also been solved by using spatial audio technology to simulate footsteps, live sounds, etc.

The Metaverse applies digital twin technology to mapping the real-world cities into the virtual world in real-time^{Ruohomäki et al. (2018)}. In detail, the digital twin technology can digitize a series of physical objects such as traffic roads, city buildings, and vehicles in real-world cities to construct virtual cities. Conveniently, researchers can conduct simulation experiments in the virtual world to achieve facilities and optimize the allocation of resources in real cities. For example, the intelligent parking solution based on the Metaverse platform built by Intel and Siemens in Berlin has brought great convenience to local traffic management^{Yan et al. (2011)}.

Before a product can be put on the market, it usually needs to be fully and complexly tested in a simulation environment. As a parallel, virtualized, and digital open world, the Metaverse provides natural advantages for product testing. In detail, the product can be tested simultaneously in physical space and virtual space through the combination of virtual and real, so as to reflect more intuitively the changes of the product to improve the efficiency of product testing and certification^{Lim et al. (2022b)}. For example, the industrial Metaverse provides a virtual test space for automotive-grade chips, which allows engineers to enter the Metaverse to simulate and test the chips (e.g., simulating and testing the safety of self-driving cars equipped with AI chips). In this way, it not only saves costs but also improves the testing efficiency of automotive-grade AI chips.

Metaverse can optimize the industrial structure of factories to improve production efficiency^{Han et al. (2022)}. Specifically, in the early stage of smart factory construction, digital twin technology can map the factory building structure, production line layout, and production process to the Metaverse in real-time. Afterward, workers can adjust the factory’s capacity, equipment, and staffing for maximum production efficiency in the Metaverse. For example, BMW’s introduction of Nvidia’s Metaverse platform Omniverse to coordinate the production of cars at 31 factories is expected to improve production planning efficiency by nearly 30%^{Alkazzi et al. (2020)}.

In the Metaverse, the traditional server-centric network architecture needs to transmit large amounts of data between cloud and terminal devices. However, due to the large-scale multiplexing nature of the network, the available bandwidth used to transfer data between the cloud server and terminal devices can change dramatically over time. This leads to many problems such as video frame drops and high latency, which degrade the quality of user experience^{Zhang et al. (2017)}. As a distributed, cloud-centric, and edge-cooperative integrated network, the cloud-edge network integrates terminal devices, edge computing nodes, and cloud servers with high real-time performance, efficient communication, and stable security. Therefore, in reference Younis et al. (2020), the cloud-edge network architecture was introduced into the Metaverse to solve the problems of insufficient bandwidth and high latency.

Although the edge cloud-Metaverse architecture has advantages of high real-time performance, efficient communication, and stable security, it also has limitations. For example, users in different scenarios of the Metaverse may generate data with different features and AI tasks with diverse needs, resulting in an imbalance in the distribution of computing power of edge nodes. To this end, we propose a self-balancing federated learning-based Metaverse framework to address the statistical heterogeneity faced by edge-cloud architectures. Federated learning (FL) consists of a server and multiple clients, in which the server delivers the global model and aggregates the client’s local model, and the client trains the received global model^{Liu et al. (2020d); Guo et al. (2021); Liu et al. (2020e); Kang et al. (2020, 2019c)}. In the proposed architecture, edge nodes also play the role of intermediaries in addition to the tasks of computational offloading and storage cache. As shown in Fig. 4, when these users (i.e., clients) collaborate together to train a high-quality global model, the edge node will plan the edge nodes that the user needs to access to achieve local balance according to the data distribution or AI task type of each user^{Duan et al. (2021b)}. Next, the model trained by each user using the local dataset is uploaded to the FL server through edge nodes for average aggregation. The entire FL training process is repeated iteratively until the global model converges.

Mobile edge computing (MEC), as an emerging paradigm for edge computing, has differed from traditional cloud computing in that cloud services can run as close to users as possible to reduce network latency^{Qiao et al. (2018)}. MEC is applied in some highly mobile devices, such as the Internet of Vehicles, mobile terminals, wearable devices, etc., to reduce network delay has become a new hot direction^{Corneo et al. (2021)}. For instance, Zhang et al.^{Zhang et al. (2017)} introduced the MEC into the Metaverse to improve the quality of users’ experience. Specifically, in the Metaverse of the MEC architecture, dynamic edge nodes can be as close as possible to users to combine multiple edge nodes to assist in completing the same user’s instructions, effectively solving the delay problem caused by user mobility.

However, when multiple edge nodes assist in cooperation, it is necessary to transmit the user’s data, which may lead to leakage of user privacy and serious consequences such as identity crisis in the Metaverse. Therefore, in this paper, we propose an FL-based Metaverse of MEC architecture for protecting users’ privacy. Specifically, model rather than user data is transmitted between edge nodes in this architecture. In addition, in Fig. 5, the edge nodes located at the FL layer can also construct a directed acyclic graph (DAG) to improve the computational efficiency of the system^{Tuli et al. (2021)}.

The Metaverse allows users to access virtual worlds through mobile smart devices, enabling real-time gameplay anywhere. However, the Metaverse development brings scalability issues to the basic physical facilities (e.g., cloud servers, edge computing nodes). In other words, as the number of online users of the Metaverse increases, the infrastructure needs to take on more computing tasks. To maintain a good user experience quality, the server must quickly process and respond to requests from more terminals in a short period. However, this is generally impractical for the Metaverse systems under traditional centralized architectures. Therefore, references Frey et al. (2008); Liu et al. (2019a); Xu et al. (2021); Nguyen et al. (2021) proposed some blockchain-based Metaverse systems to solve the above problems. Due to the independence and security of the blockchain, reference Nguyen et al. (2021) describes how to use blockchain to solve the scalability problem in the Metaverse. Reference Xu et al. (2021) designed an incentive mechanism to improve the transaction efficiency in the Metaverse by combining the Dutch double auction mechanism and reinforcement learning.

However, the above methods suffer from the insufficient performance of the trained models. Specifically, in a blockchain-based decentralized Metaverse system, users can only interact with each other to train models without effectively utilizing as much user data as possible due to strict privacy restrictions. We propose a decentralized Metaverse architecture based on federated learning combined with blockchain for this problem. As shown in Fig. 6, a user can request a computing task through the blockchain, and then users with similar data distribution will receive the task request broadcast by the blockchain. Next, these users upload their local models to the blockchain, and the blockchain aggregates the received models on average and returns them to the task requester. Finally, the entire transaction process will be recorded on the blockchain for traceability. Therefore, under this architecture, the local model is optimized through the cooperation of multiple user nodes, which effectively solves the problem of insufficient model performance in the blockchain-based decentralized Metaverse.

With the continuous development of various communication and virtual technologies, the Metaverse provides users with excellent experiences in some virtual scenarios such as entertainment games and smart cities. However, the Metaverse is a virtual world that is interconnected with the real world and exists in parallel, and it has the same digital information security issues as the real world. In reference Leenes (2007), the authors proposed that there is a privacy problem in the Metaverse. Metaverse can potentially monitor users’ physiological responses and body movements, thereby leaking sensitive personal information such as user habits and their physiological characteristics to third parties. In addition, in the virtual world, evil users can also imitate other users to obtain other people’s personal information. In large-scale online games, some matters such as user harassment and theft of other people’s finances threaten users’ privacy and security all the time^{Falchuk et al. (2018)}.

Considering the risk of private data leakage of users, the authors in reference Falchuk et al. (2018) proposed two privacy protection mechanisms “clone cloud” and “private copy”. The “clone cloud” creates multiple users’ substitutes in the system and makes the substitutes move continuously in the Metaverse environment to confuse the users’ location information, activities, goals, and other information. This approach can make the Metaverse environment into chaos so that intruders cannot distinguish the real users. The “private copy” allows users to apply for virtual private storage space in the Metaverse, and the entire experience process takes place in the private space rather than in the complete virtual world that is widely accessed, effectively protecting the user’s privacy. Besides, the authors in reference Lu et al. (2021) proposed a blockchain-based digital twin wireless network (DTWN) edge computing federated learning framework to solve the problem of user privacy data security. The blockchain was introduced to improve the security of the untrusted user’s digital twin because it can ensure the security of the digital twin data, and the user’s part model can be stored in it.

One of the characteristics of the Metaverse is an immersive virtual experience for users. However, a high-speed and low-latency network connection is an important foundation for improving the user experience. In the virtual world created by the basic digital technology, the user may experience visual jitter or delay and other undesirable phenomena, which seriously affect the user’s immersive experience due to insufficient network bandwidth.

To reduce network latency, an incentive mechanism framework for VR services was proposed in reference Xu et al. (2021), which uses perceived quality as a criterion for measuring immersive experience and effectively evaluates the immersive experience in the Metaverse. Meanwhile, the authors designed a double dutch auction mechanism based on deep reinforcement learning (DRL), which not only improves the communication efficiency and reduces the auction cost but also realizes the dynamic pricing of users’ immersive services in the virtual world, ensuring the rationality of the incentive mechanism and the authenticity. Jiang et al.^{Jiang et al. (2021)} found that coded distributed computing (CDC) can improve the latency problem in the Metaverse and proposed a CDC and dual blockchain distributed collaborative computing framework. The CDC is responsible for aggregating the idle resources of devices connected to the Metaverse to process the more intensive computing tasks in the Metaverse. The dual-blockchain measures CDC’s worker reliability by estimating future rewards through the workers’ historical records. Furthermore, the authors design an incentive mechanism (i.e., optimal computing rewards can be obtained for devices assigned to large-scale computing tasks) to motivate CDC workers to actively participate in the computing tasks of the Metaverse to improve the computing speed.

The key to the connection between the real world and the virtual world is collecting data in the physical world and updating it in real-time in the virtual world so that users can have a real and lasting interactive experience. However, the computing, communication, and storage shortage will seriously affect the user’s immersive experience. At present, in the Metaverse, reasonably allocating resources and effectively coordinating work among computing nodes are still essential challenges.

For the resource allocation problem, a new blockchain-based framework called Metachain was proposed in reference Nguyen et al. (2021). Metachain introduces a new sharding technology, which can create shards according to users’ actual computing resource requirements, which was used so that service providers can dynamically allocate computing resources to shards. Moreover, to meet the demand for computing resources in the Metaverse, the author uses Stackelberg game theory^{Stackelberg (1934)} analysis to propose an incentive mechanism, i.e., users obtain corresponding rewards by providing resources to blockchain shards. Based on the intelligent 6G edge network, a machine learning framework was proposed in reference Huang et al. (2021) for decentralized learning and coordination of edge nodes to improve resource allocation strategies. Specifically, during the training process, the nodes select an appropriate model training scheme by recording the different resource requirements of different users. One of the solutions is the computational offloading and resource allocation algorithm of multi-agent DRL, i.e., the local model is cooperatively trained using local information sharing. Another solution is the resource allocation algorithm of the federated framework based on DRL, i.e., the edge node uploads the model parameters to the server to update the global model after completing a round of training.