The Internet of Senses:
Building on Semantic Communications and Edge Intelligence

The IoS is defined as the notion of digitally streaming the signals conveying information for all senses and therefore, blurring the difference of physical and virtual reality as well as enabling telepresence style communications. Thus, as inspired by the Tactile Internet, the IoS architecture is presented as a 1) Primary domain (head-mounted VR glasses, haptic devices, holograms and BCI), 2) Network domain including edge computing, 3) Replica domain (audio, visual, olfactory, gustatory and tactile sensors and streams, haptic feedback, robots) For instance, a consumer with his head-mounted VR glasses and haptic devices (primary domain and send the request) can sense the stuff with all senses in the grocery store (replica domain) which is equipped with sensors and cameras through the wireless network (network domain).

EXtended Reality (XR) includes AR, VR and merged reality, all of which aim for creating realistic, interactive digital environments using sound and vision [11]. XR is extensively used in entertainment, education, medical applications and in the manufacturing industry. The next step of creating realistic environments involves using haptics, taste and smell to provide a truly immersive experience. Hence the retail, entertainment, education, health and food industries would be the first adopters of this alluring technology.

There are ongoing efforts to replicate scents and tastes digitally. Digital scent technology, also known as olfactory technology, aims to sense, transmit and receive scent-enabled digital media. The sensing part of this technology relies on using olfactometers and electronic noses. After sensing, the digital flavor interface digitally stimulates taste (using electrical and thermal stimulation methodologies applied to the human tongue) and smell senses (using a controlled scent emitting mechanism) simultaneously [12]. The transmission of end-to-end olfactory and gustatory information using a wireless medium imposes tight throughput and latency specifications. Additionally, the high-fidelity audio and video signals replicating the ambiance and augmenting human senses through XR interfaces further tightens the throughput and latency specifications.

The tactile internet supports real-time access, monitoring and control of remote objects or processes using haptic feedback[1]. A typical example of the tactile internet use case is remote surgery, where both the haptic sensory signals and audio-visual signals are transmitted from a remote site (or remote patient) to an operator (or a surgeon). Depending on whether the operator is directly interacting with the holograms or wearing head-mounted VR glasses, this audio-visual monitoring can be provided by immersive 3D HTC or XR video streaming techniques [2]. Based on the information obtained from different inputs (haptic and audio-visual), the operator then transmits the corresponding control signal to the remote site. The tactile internet has been a hot research topic for years and it is a compelling use case of IoS.

Holographic-Type Communication (HTC) is based on digitally delivering 3D images between remote locations [2]. It relies on a multitude of cameras recording the same scene from equi-angle positions spread across 360°, where remote users appear with their holographic presence. Holographic displays have to satisfy all aspects of human observation such as viewpoint tilts, angles and positions relative to the hologram in addition to the color, depth, resolution and frame rate of standard video specifications [1]. HTC will play an important role in IoS, as it creates realistic telepresence environments. Naturally, the multiplicity of angular views are highly correlated, hence they lend themselves to high compression holographic encoding, which is in its infancy and requiring further research.

HTC is naturally related to human vision, but when further augmented with data from other senses, it can facilitate the creation of a digital twin of an environment as highlighted in the following section.

A digital twin is the virtual representation of physical (real) entities, e.g., persons, objects, processes or systems [2]. By relying on a digital twin, users may become capable of real-time monitoring and controlling the physical reality in a virtual environment. In general, XR technologies or holograms are utilized to observe and explore the virtual world with the aid of 3D imaging. DTs are already employed in manufacturing and industrial applications. However, in the future IoS, they can be augmented with the aid of multi-sensory inputs for facilitating many compelling applications [13] such as online learning and education.

The communication requirements of 6G use cases have been identified in [1, 2, 11], some of which are applicable to the IoS. According to these reports, immersive 16k VR requires 0.9 Gbps throughput assuming a compression ratio of 400. For the tactile internet, the required throughput varies, depending on the motion-activity of the specific application. True holograms and computer-generated holograms require throughputs up to Terabits/sec (Tbps). For example, a hologram of 19.1 Gigapixel resolution requires 1 Tbps. Thus, at least a rate of 0.58 Tbps is required for conveying a hologram of 11.1 Gigapixel resolution over a mobile device. Thus, HTC needs ultra-high rates and bandwidth. However, holographic video-compression coupled with activity-dependent non-uniform sampling has the potential of mitigating these challenges. Finally, virtualized objects have to be represented by a huge amount of data. For instance, a digital twin of a $1\times 1$ $m^{2}$ area requires 0.8 Tbps throughput at a compression ratio of 300 for the first few frames, although subsequent updates of the digital twin may require lower data rates.

Latency is also a key performance indicator, and the critical applications of the IoS in remote surgery require ultra-low latency at a sub-ms level and ultra-high reliability [2]. Moreover, low latency is required for preventing cyber-sickness. For example, 360° VR systems have to react to head movements with a latency below 10 ms. The latency, reliability and security requirements are very stringent in critical real-time communications. Moreover, reliability is a salient factor in certain applications that demand deterministic performance. Synchronization among different sensory inputs or multiple transmission paths or data streams is also necessary in IoS use cases. Furthermore, power, storage and computation resources are required at the edge for HTC use cases and for frame synchronization, and for synthesizing, rendering and reconstructing 3D images.

The above-mentioned requirements of IoS use cases are shown in Fig. 1, while fledgling 6G technologies and IoS use cases are illustrated in Fig. 2. IoS use cases are empowered by exploiting a variety of 6G technologies to fulfill their requirements. The figure features powerful 6G technologies, such as RIS, massive MIMO, SAGIN [5], carrier aggregation and visible light communication [1]. The figure also shows a grocery store example that is connected to an intelligent edge server equipped with a semantic encoder and decoder, remote surgery robots, an HTC scenario and the digital twin of a car.

Technological advances in communication over the past decades have predominantly aimed for achieving reliable high-rate transmission. However, semantic communication additionally considers the meaning, significance and utility of the information conveyed [9]. Furthermore, by considering a priori information at the receiver, higher throughput can be achieved for IoS use cases. Specifically, in human-computer interaction applications, such as the digital twin of mobile robots and surveillance of an environment, the sampling, transmission and control of information may be triggered, whenever a change occurs at the source. Then, the receiver may apply semantic reception for decoding the information, again, according to its usefulness. Additionally, data analytics and ML may be utilised for making decisions concerning the usefulness of information based on past experiences.

Semantic information can be extracted based on different features such as the value of information, age of information, timeliness and relevance [9]. Semantically-aware filtering can be applied at the source to acquire the relevant samples of signals for transmission based on their freshness and significance. This may reduce the sampling rate below the Nyquist-rate, which reduces the channel occupancy. Furthermore, semantically-aware preprocessing could be applied to extract important features for transmission. For example, in the virtual workplace, some features of the environment, such as scent, can be inspected first and only the relevant information would be transmitted instead of sending all the acquired information. Then, the matched receiver applies semantic reception to reconstruct the samples in real-time based on its target quality. In summary, semantic communications motivate the joint design of sampling, transmission, reconstruction and the control of information, while considering the associated source variations, reconstruction efficiency, channel parameters, and QoS requirements, such as delay, throughput and reliability.

Furthermore, the interplay between semantic communications and ML may be exploited for improving both the communications and learning metrics [3, 9]. For instance, in IoS, imagine two distributed cell-edge nodes, where one of them senses the environment and the other selects a bespoke action according to this sensed information. However, the information has to be transmitted over imperfect channels to the second node. Here the objective of the communication protocol (e.g., modulation and coding parameters) is not only that of increasing the reliability, but also achieving the learning goal of the second node. This is a typical example of semantic communication. Thus, semantic communication is an emerging field of study and has the potential of supporting IoS applications in 6G.

Naturally, increasing both the spectral efficiency and the available bandwidth improves the network capacity, which is a key requirement of IoS use cases. A popular technology is constituted by RISs relying on passive reflectarrays and control elements for minimizing the BS’s power consumption [5]. Moreover, exploiting the large bandwidth of the mmWave and THz bands allows augmenting the throughput, but at the cost of high pathloss, which must be compensated by high-gain beamforming. They also necessitate sophisticated mobility management techniques. Nevertheless, the sub-6GHz frequency bands are still essential for improving the coverage. Additionally, the SAGIN concept is pivotal for providing ubiquitous coverage, since it is capable of filling the coverage holes. In addition to the above-mentioned advances, multi-connectivity and carrier aggregation continue to be important capacity-enhancing techniques. Although carrier aggregation has been used since LTE, there are still challenges that have to be addressed. For instance, the UE’s power consumption increases with carrier aggregation, because all active component carriers are monitored even though they are not used for data transmission. Furthermore, there are numerous challenges in carrier aggregation in dual connectivity scenarios and in the allocation of bandwidth to control channels. Carrier aggregation will also be an important technology for IoS, but erratic fluctuations in traffic, strict QoS requirements and several other factors will make it difficult to reach optimal decisions under uncertainty. Therefore, there is a need for learning carrier aggregation decisions in partially observable environments.

The growing number of edge servers provides abundant computing resources distributed at the network edge. Naturally, edge intelligence continues to be a key-enabler for the IoS, as it can be harnessed for the optimal and proactive exploitation of communication and computing resources that leads to improved throughput, reliability and latency which are beneficial for IoS use cases. In fact, AI will play a prominent role in the design, deployment and operational phases of 6G networks and IoS. Since the network will be flexible and programmable, AI facilitates automated network optimization and management, and as an additional benefit, a variety of tasks including radio resource management, carrier aggregation, mobility management, virtualized network function placement and slicing can be improved by exploiting AI. Additionally, designing the air interfaces by ML algorithms under AI-native air interfaces may enhance the wireless transmission [4]. Moreover, UE can be trained to learn both protocol parameters as well as control messages to optimize the network performance.

In ML-aided wireless systems, it is routinely assumed that intelligent agents are capable of fully observing the environment. However, in practice, there is substantial uncertainty in the state of the environment encountered, hence the agents may have to make decisions in the face of partial observability. The popular POMDP technique is eminently suitable for handling such circumstances. In particular, distributed learning in edge-intelligence typically relies on access to partial information at each edge node. Communicating in the face of limited information about the environment becomes even more challenging, when there is limited communications among the nodes. Therefore, distributed learning at the edge has to be modeled by a Multi-agent Markov Decision Process (MMDP) relying on partially observable state information, which is also known as Decentralised POMDP (DEC-POMDP) or Multi-Agent POMDP.

A distributed DEC-POMDP model relying on a pair of agents is portrayed in Fig. 3, where the environmental state $s$ is observed as $o_{1}$ and $o_{2}$ by agents 1 and 2, respectively. These observations depend both on the previous actions carried out and on the resultant state, as well. The pair of agents select the appropriate actions $a_{1}$ and $a_{2}$ according to the POMDP policy considered. The actions chosen will then alter the current state to a new state of the environment. Correspondingly, this updated state creates new observations for the agents. An intelligent POMDP policy acts as a function of the previous observations and actions, but additionally it also depends on the instantaneous reward achieved. A pivotal example of partial observability in the wireless domain emerges in the context of carrier aggregation, given the uncertainty in terms of the traffic arrival statistics. In the next section, we provide deeper insights into this problem, since efficient carrier aggregation is crucial for enhancing the throughput or equivalently transmission delay in IoS use cases.