Graph-Embedded Multi-Agent Learning for Smart Reconfigurable THz MIMO-NOMA Networks

We suppose that each RIS $j$ consists of $L$ reflecting elements, which is controlled by a software-defined RIS controller. Since the continuous phase control is hard to realize in practice, a finite-resolution passive phase shifter is utilized for each reflecting element. To save energy consumption as well as overcoming the non-ideal reflecting effect due to the discrete phase-shift control, we propose to leverage a dynamic RIS element selection structure during transmissions¹¹1Note that dynamic RIS element selection requires a complex design for the RIS array, which might be realized in the near future. In this work, we assume the reflecting elements in RISs can be dynamically turned ON/OFF by controlling the PIN diodes.. By flexibly controlling the ON/OFF states and the phase shifts of passive phase shifters, the dynamic selection structure can prevent unfavorable reflections and achieve higher energy efficiency.

Based on the dynamic selection structure, the phase-shift matrix $\bm{\Theta}^{j}$ on RIS $j$ can be given by

where $\omega_{l}^{j}$ denotes the ON/OFF state of the $l$-th RIS element, given by

The discrete phase shift of a selected element $l$ at RIS $j$ is determined by an integer $\beta_{l}^{j}\in\{0,1,...,2^{B^{\mathrm{R}}-1}\}$ and the RIS resolution bit $B^{\mathrm{R}}$, which can be written as

The equivalent channel vector from AP $i\in\mathcal{M}$ to user $U_{nk}^{m}$ via multiple RISs is defined as $\mathbf{h}_{nk}^{im}=[\mathbf{h}_{nk}^{im1},...,\mathbf{h}_{nk}^{imN_{\mathrm{R}}}]\in\mathbb{C}^{1\times N_{\mathrm{A}}}$, where $\mathbf{h}_{nk}^{ims}\in\mathbb{C}^{1\times N_{\mathrm{sub}}}$ is the equivalent channel from subarray $s$ at AP $i$ to user $U_{nk}^{m}$. Here, $\mathbf{h}_{nk}^{im}$ is given by

where $\widetilde{\mathbf{h}}_{nk}^{im}\in\mathbb{C}^{1\times N_{\mathrm{A}}}$, $\mathbf{f}_{nk}^{jm}\in\mathbb{C}^{1\times L}$, and $\mathbf{G}^{ij}\in\mathbb{C}^{L\times N_{\mathrm{A}}}$ denote the direct spatial channels of AP $i$-user $U_{nk}^{m}$, RIS $j$-user $U_{nk}^{m}$, and AP $i$-RIS $j$ links, respectively.

We assume LoS paths always exist between the APs and its neighboring RISs. However, the LoS paths for the AP-user and RIS-user links may be blocked. Therefore, channel $\mathbf{\widetilde{h}}_{nk}^{im}$ can be modeled by

where $\varUpsilon$ is the antenna gain, $L_{\mathrm{path}}(f,d_{nk}^{im,\mathrm{AU}})$ is the path loss determined by the frequency $f$ and the distance $d_{nk}^{im,\mathrm{AU}}$ between AP $i$ and user $U_{nk}^{m}$. $\mathbbm{1}_{nk}^{im,\mathrm{LoS}}\in\{0,1\}$ indicates the existence of the LoS path, which is determined by the LoS probability based on the indoor THz blockage model [7]. Moreover, $n_{\mathrm{NL}}$ signifies the number of NLoS paths, and $\lambda_{nk}^{iml}$ is the reflection coefficient for NLoS path $l$ [19]. $\varphi_{nk}^{iml}$ denotes the angle of departure (AoD) for the downlink channel between AP $i$ and user $U_{nk}^{m}$. Given AoD $\varphi$, the array response vector can be denoted by $\bm{\alpha}(\varphi)=\frac{1}{\sqrt{N_{\mathrm{A}}}}\left[1,...,e^{j\pi[m\sin\varphi]},...,e^{j\pi[(N_{\mathrm{A}}-1)\sin\varphi]}\right]^{T}$. According to [20], the path loss $L_{\mathrm{path}}(f,d)$ includes both spreading loss and absorption loss, i.e.,

where $k(f)$ is the frequency-dependent medium absorption coefficient, $d$ is the distance, and $c$ is the light speed.

In traditional CSI-based massive MIMO-NOMA networks, the user clustering usually contains a cluster-head selection (CHS) procedure that chooses cluster head based on channel conditions, followed by which the remaining users are grouped with the highest channel-correlated cluster heads [21]. However, the CSI-based user clustering is inflexible to guarantee the ultra-high data rate and mitigate interference for SE users. To achieve customizable and intelligent communication experiences, we extend the traditional CSI-based user clustering to a QoS-based user clustering. Relying on the smartly reconfigured beams, users’ spatial channels can be flexibly aligned to achieve adaptive user clustering. Specifically, we group multiple IoT users with a single SE user. In this way, we can ensure multiplexing gain for SE users while enhancing IoT user connections. In light of this, we assume the number of RF chains activated at each AP is equal to its serving SE users, i.e., $N_{\mathrm{R}}=K_{\mathrm{S}}$. Let $U_{n1}^{m}$ represent the SE user, and $U_{nk}^{m},\forall k>1$, be the IoT user. The low-complexity user clustering scheme based on the reconfigured channels can be stated as follows.

The SE users are firstly selected as the cluster heads of different clusters. Then, we define the channel spatial correlation $\mathcal{C}\left(\mathbf{h}_{k}^{mm},\mathbf{h}_{n1}^{mm}\right)$ between an ungrouped IoT user $k$ and the cluster head $U_{n1}^{m}$ as

Thereafter, multiple IoT users can be non-orthogonally grouped into the same cluster, where the cluster head achieves the strongest reconfigured channel correlations with them. The computational complexity of the QoS-based clustering scheme is $\mathcal{O}\left(K_{\mathrm{U}}N_{\mathrm{R}}\right)$. In comparison, the conventional CSI-based clustering scheme has the computational complexity of $\mathcal{O}\left(N_{\mathrm{R}}\left(K_{\mathrm{U}}+K_{\mathrm{S}}\right)^{2}\right)$ [21].

Relying on the adaptively aligned channels through RISs, the NOMA user decoding order in each cluster can be flexibly customized based on QoS requirements without degrading the system performance. Exploiting the NOMA successive interference cancellation (SIC), SE users can subtract signals of other users within the same cluster, and completely eliminate intra-cluster interference based on power-domain multiplexing to realize ultra-high-speed and reliability-guaranteed communications. To realize this goal, each SE user with the highest QoS requirements in its cluster would be decoded at last, while the data signals of $K_{n}^{m}-1$ IoT users in each cluster $n$ would be decoded first.

To ensure the SIC at each IoT users in the same cluster can be successfully carried out, the NOMA decoding order of IoT users are rearranged according to equivalent channel gains as $|\mathbf{h}_{n2}^{m}\mathbf{V}^{m}\mathbf{w}_{n}^{m}|^{2}\geq|\mathbf{h}_{n3}^{m}\mathbf{V}^{m}\mathbf{w}_{n}^{m}|^{2}\geq...\geq|\mathbf{h}_{nK_{n}^{m}}^{m}\mathbf{V}^{m}\mathbf{w}_{n}^{m}|^{2}$. Thereafter, the SE user would sequentially decode the signals of the former $K_{n}^{m}-1$ IoT users to completely cancel intra-cluster interference. To successfully cancel interference from IoT user $U_{nk}^{m},\forall k>1$, imposed to SE user $U_{n1}^{m}$, we have the following SIC constraint

where $\gamma_{n,k\to n,k}^{m}$ is the signal-to-interference-and-noise ratio (SINR) of IoT user $U_{nk}^{m}$ for decoding its signal, and $\gamma_{n,1\to n,k}^{m}$ is the SINR to decode the signal of IoT user $U_{nk}^{m}$ at SE user $U_{n1}^{m}$. We denote $\mathbbm{1}_{nk}^{m,\mathrm{fail}}=0$ if the SIC constraint (8) can be satisfied, i.e., the SE user $U_{n1}^{m}$ can cancel the intra-cluster interference from IoT user $U_{nk}^{m}$. Otherwise $\mathbbm{1}_{nk}^{m,\mathrm{fail}}=1$, and the intra-cluster interference from IoT user $U_{nk}^{m}$ would be taken as noise at the SE user. In this way, $\gamma_{n,1\to n,k}^{m}$ can be expressed as

where $\sigma^{2}$ is the power spectral density of additional white Gaussian noise (AWGN), $p_{nk}^{m}$ is the power allocation coefficient for user $U_{nk}^{m}$, and $P_{n}^{m}=\sum_{k=0}^{K_{n}^{m}}p_{nk}^{m}$ is the power consumption of RF chain $n$ at AP $m$.

We design the analog beamforming at each AP to increase effective data gain of each cluster, and design the digital beamforming for suppressing residual inter-cluster interference that is not mitigated by RISs based on zero-forcing (ZF).

The analog beamforming matrix with the sub-connected structure can be formulated as

where $\mathbf{v}_{mni}^{\mathrm{sub}}=\mathbf{0}$ for $n\neq i$, and $\mathbf{v}_{mnn}^{\mathrm{sub}}\in\mathbb{C}^{N_{\mathrm{sub}}\times 1}$ has an amplitude $\frac{1}{\sqrt{N_{\mathrm{sub}}}}$. The sub-connected analog beamforming is designed to improve the effective gains toward cluster heads that are highly spatial-correlated to the cluster members. Considering $B^{\mathrm{A}}$-bits quantized phase shifters in each antenna subarray, the $i$-th element of the active analog beamforming vector $\mathbf{v}_{mnn}^{\mathrm{sub}}$ is given by [6, 17]

where $\varpi_{mn}^{*}$ is the quantized phase calculated by

To further suppress the effective inter-cluster interference that hasn’t been canceled by RISs, we design the ZF digital beamforming as follows. Let $\mathbf{w}_{n}^{m}\in\mathbb{C}^{N_{\mathrm{R}}\times 1}$ denote the digital beamforming vector for cluster $n$ at AP $m$. The cluster center is given as $\widehat{\mathbf{h}}_{n}^{mm}=\frac{1}{K_{n}^{m}}\sum_{k\in\mathcal{K}_{n}^{m}}\left(\mathbf{h}_{nk}^{mm}\mathbf{V}^{m}\right)^{T}$. Denote $\widehat{\mathbf{H}}^{mm}=\left[\widehat{\mathbf{h}}_{1}^{mm},\widehat{\mathbf{h}}_{2}^{mm},...,\widehat{\mathbf{h}}_{N_{\mathrm{R}}}^{mm}\right]$. Therefore, the ZF digital beamforming is calculated as

By introducing column power normalizing, the baseband precoding matrix can be expressed as

Define $\mathbb{E}\{{s}_{nk}^{m}\left({s}_{nk}^{m}\right)^{H}\}=1$ as the transmitted symbols with normalized power, and $x_{n}^{m}=\sum_{k=0}^{K_{n}^{m}}\sqrt{p_{nk}^{m}}s_{nk}^{m}$ as the transmitted signal in cluster $n$. Define $z_{nk}^{m}$ as the AWGN. The baseband signal received by each SE user $U_{n1}^{m}$ after SIC can be formulated as

Therefore, the SINR of each SE user $U_{n1}^{m}$ can be written as

Moreover, the SINR of IoT users $U_{nk}^{m},\forall k>1$, can be given by

Hence, the data rate of each user $U_{nk}^{m}$ at each time slot can be expressed as

The total power consumption for the proposed networks at each time slot includes both transmit and circuit power, which can be formulated as

where $\xi$ denotes the inefficiency of the phase shifter in the THz network, $P_{\mathrm{D}}$ is the circuit power consumption at each user, and $P_{\mathrm{RIS}}\left(B^{\mathrm{R}}\right)$ denotes hardware energy dissipated at each RIS element depending on the RIS resolution bit $B^{\mathrm{R}}$[23]. Moreover, $P_{\mathrm{AP}}$ is the circuit power consumption at each AP, which is given by

where $P_{\mathrm{BB}}$ is the baseband power consumption, and $P_{\mathrm{RF}},P_{\mathrm{PS}}$, and $P_{\mathrm{A}}$ denote power consumption of per RF chain, per phase shifters and per power amplifies, respectively.

Therefore, the network energy efficiency $\eta$ at each time slot can be formulated as

Suppose the THz MIMO-NOMA networks maintain a traffic buffer queue $q_{nk}^{m}(t)$ for each user, $\forall t\in\mathcal{T}$, which aims to transmit specific data volume in a predefined time. At the beginning of each time slot $t$, the queue length $q_{nk}^{m}(t)$ of user $U_{nk}^{m}$ can be updated by

where $R_{nk}^{m}(t)$ is the transmission data rate of (18), $A_{nk}^{m}(t)$ denotes the traffic arrival column at time slot $t$ following the poisson distribution, and $\left[x\right]^{+}=\max\{x,0\}$.

To ensure reliable transmission, we should guarantee the queue stability and keep the outages below a predefined threshold [22]. The queue stability is ensured by

Moreover, the outage probability, i.e., the probability that the traffic queue length $q_{nk}^{m}(t)$ of device $U_{nk}^{m}$ exceeds the threshold $q_{nk}^{m,\max}$, is limited by a certain threshold $\epsilon_{nk}^{m}$. Therefore, the reliability conditions can be formulated as

We aim to dynamically find a stationary online policy $\pi$, which can jointly optimize the dynamic RIS element selection, coordinated discrete phase-shift control, and transmit power allocation by observing the environment state at each time slot $t$. Under the given user clustering, NOMA decoding, and hybrid beamforming schemes, the joint policy dynamically maximizes the expected system energy efficiency for the smart reconfigurable THz MIMO-NOMA network, while satisfying various QoS requirements in terms of data rate and transmission reliability of users. Therefore, the online objective function can be formulated as follows

where $\mathbf{a}(t)=\left[\bm{\omega}(t),\bm{\alpha}(t),\bm{\beta}(t)\right]$. Constraint (25b) ensures the minimum data rate of each user, (25c) denotes the total transmit power should be less than the maximum power $P_{\mathrm{AP}}^{\max}$, (25d) signifies the binary ON/OFF state of RIS elements, and constraint (25e) represents the discrete phase-shift control. Note that the SIC constraint (8) is ignored since it has been implied in the SINR expression (16) via the indicator $\mathbbm{1}_{nk}^{m,\mathrm{fail}}$.

This online policy can be modeled as a constrained Markov Decision Process (MDP). However, solving $\mathcal{P}_{0}$ in a centralized way is computationally inefficient due to the large-scale joint state-action space and the heavy overhead of high-dimensional information exchange from multiple APs and RISs to the centralized controller. To tackle $\mathcal{P}_{0}$ in an efficient and low-complexity manner, we model the long-term energy efficiency optimization problem as a constrained Dec-POMDP. In detail, the POMDP provides a generalized framework to describe an MDP with incomplete information, while Dec-POMDP extends POMDP to the decentralized settings as $\mathcal{P}=\left(\mathcal{I},\mathcal{S},\Omega,\mathcal{A},P_{\mathbf{s}\to\mathbf{s}^{\prime}}^{\mathbf{a}},\pi,r,\Gamma\right)$. Here, $\mathcal{I}$ denotes a set of agents. $\mathcal{S}$, $\Omega$, and $\mathcal{A}$ denote the state, observation, and action spaces. At each slot $t$, given the true state $\mathbf{s}(t)\in\mathcal{S}$, each agent $i$ partially observes a local observation, based on which it can determine a local action using decentralized policy $\pi$. The joint observation and action vectors of all agents are denoted by $\mathbf{\Omega}(t)\in\Omega$ and $\mathbf{a}(t)\in\mathcal{A}$, and the decentralized policy $\pi(\mathbf{a}(t)|\mathbf{s}(t))$ means the probability of taking joint action $\mathbf{a}(t)$ at the joint state $\mathbf{s}(t)$. Furthermore, $P_{\mathbf{s}\to\mathbf{s}^{\prime}}^{\mathbf{a}}=\mathbb{P}(\mathbf{s}^{\prime}|\mathbf{s},\mathbf{a})$ is the probability of transferring into a new state $\mathbf{s}^{\prime}$ by taking action $\mathbf{a}$ at state $\mathbf{s}$. Given the global reward function $r\left(\mathbf{s}(t),\mathbf{a}(t)\right)$ and the discount factor $\Gamma$, the agents can cooperate and coordinate to maximize the discounted return. In this work, we model APs and RISs as two types of agents, whose set is denoted as $\mathcal{I}=\mathcal{I}^{0}\cup\mathcal{I}^{1}$. Let $\mathcal{I}^{u}$ denote the set of the $u$-th type agents, where $u=\{0,1\}$ represent APs and RISs respectively. We specify the formulated Dec-PoMDP as follows.

Based on the Markov’s inequality [27], we can obtain the upper bound of the outage probability $\mathrm{Pr}\left(q_{nk}^{m}(t)\geq q_{nk}^{m,\max}\right)\leq\mathbb{E}\left[q_{nk}^{m}(t)\right]/q_{nk}^{m,\max}$. Therefore, the reliability constraint (24) can be strictly guaranteed by ensuring

To tackle the long-term average constraints (23) and (29), we apply the virtual queue technique and Lyapunov optimization theory [26] to recast the intractable constrained Dec-POMDP as a normal Dec-POMDP, which can be solved by general MADRL algorithms. To meet the reliability constraints (29), we construct a virtual queue $Y_{nk}^{m}(t)$ to model the instantaneous deviation of queue length constraints for each user $U_{nk}^{m}$. Here, $Y_{nk}^{m}$ evolves as follows

where we initialize $Y_{nk}^{m}(t)=0$ at $t=1$, $\forall k\in\mathcal{K}$.

Let $\bm{\Xi}_{nk}^{m}(t)=\left[q_{nk}^{m}(t),Y_{nk}^{m}(t)\right]$, $\forall k\in\mathcal{K}$. Define $\bm{\Xi}(t)=\left[\bm{\Xi}_{11}^{1}(t),\bm{\Xi}_{12}^{1}(t),...,\bm{\Xi}_{N_{\mathrm{R}}K^{M}}^{M}(t)\right]$ as the combined queue vector. Then, we introduce the Lyapunov function $\mathcal{L}(\bm{\Xi}(t))=\frac{1}{2}\bm{\Xi}(t)\bm{\Xi}^{T}(t)$ to measure the satisfaction status of the reliability constraint, which should be maintained under a low value. Moreover, the one-step Lyapunov drift is defined as $\Delta\mathcal{L}(t)=\mathcal{L}(\bm{\Xi}(t+1))-\mathcal{L}(\bm{\Xi}(t))$, which can be upper bounded by the following Lemma.

Based on the Lyapunov optimization, the original constrained Dec-POMDP of maximizing the system energy efficiency while ensuring the long-term reliability constraints can be transformed into minimising the following drift-minus-bonus function