Joint Power Allocation and Rate Control for Rate Splitting Multiple Access Networks with Covert Communications

Although possessing the advantages, the RSMA scheme is facing two main challenges. Unlike conventional multiple access schemes, e.g., Spatial Division Multiple Access (SDMA) which treats interference as noise or Non-Orthogonal Multiple Access (NOMA) which successively removes multi-user interference during the decoding process, RSMA partially decodes the messages and partially treat multi-user interference as noise. The RSMA transmitter hence needs to jointly optimize the power allocation and rate control for different messages to maximize the energy and spectral efficiency for the whole system. Aiming to address such problems, several research works have been proposed in the literature. In [9], the authors consider a rate splitting approach with heterogeneous Channel State Information at the Transmitter (CSIT). Two groups of CSIT qualities are considered and the transmitter is assumed to have either partial CSIT or no-CSIT. For the no-CSIT scenario, the users decode their messages by treating interference as noise without using a rate splitting strategy. In contrast, for the partial CSIT scenarion, a rate splitting strategy is applied and a fraction of total power is allocated equally among private symbols and common symbols. Simulation results show that with the proposed power allocation scheme, the sum-rate of users in the group with the rate splitting strategy can gain significant improvement compared to those in the group without using any rate splitting strategy. Similarly, in [7], the authors study a rate splitting strategy for a multi-group of users in a large-scale RSMA system. A more complex precoding design for power and rate allocated to users is proposed to find the maximum sum-rate of the system. The simulation results reveal that a precoding design with rate splitting can benefit sum-rate of the system. However, the authors in [7] only consider a perfect CSIT scenario in which the BS is assumed to know exact information of the channel and the channel is assumed to be fixed. In order to relax these assumptions, the authors in [6] and [10] consider a similar system, but under imperfect CSIT. In this case, a stochastic optimization formulation is proposed to deal with the uncertainty of the channel with imperfect CSIT. In [10], the authors show that with rate splitting under imperfect CSIT, the sum-rate of the system can achieve non-saturating rates at high SNR values compared to saturating rates of a conventional scheme, i.e., SDMA. In [6], the authors further investigate the impacts of different error models on the system performance. Numerical results reveal that in addition to the expected sum-rate gains, the benefits of rate splitting also include relaxed CSIT quality requirements and enhanced achievable rate regions compared with a conventional transmission scheme. A comprehensive analysis of RSMA performance is studied in [4]. In this work, through many simulations and performance analysis, they show that the rate splitting techniques are able to softly bridge the two extremes of fully treating interference as noise and fully decoding interference. Thus, in comparison with conventional multiple access approaches, e.g., SDMA and NOMA, RSMA can gain significant rate enhancement. Although the aforementioned works propose solutions to improve system performance for RSMA networks, the channel state distribution (or channel state matrix) is always assumed to be known by the transmitter. In addition, optimization methods in these works also introduce additional variables, e.g., equalizers and weights, that are highly correlated with the channel state. Thus, unavailability or drastic changes of the channel state information, e.g., due to mobility of users of link’s failures [12], can result in significant degradation of these algorithms. Therefore, a more flexible framework that not only deals with the dynamics of the environment but also efficiently manages power allocation and rate control for RSMA without requiring complete or partial information, e.g., posterior distributions, of the channel state, is in an urgent need.

The second challenge that RSMA is facing is security. Although a new data rate region can be achieved with RSMA, investigation on RSMA’s security is still in its early stage. Several works, such as [3, 14] and [13], are proposed to address the eavesdropping issues in RSMA networks. In particular, the authors in [3] propose a cooperative RSMA scheme to enhance the secrecy sum-rate of the system in which the common messages can be used not only as desired messages but also artificial noise. In this case, it is shown that the proposed cooperative secure rate splitting scheme outperforms conventional SDMA and NOMA schemes in terms of secrecy sum-rate. However, [3] only considers the perfect CSIT scenario in which the transmitter has complete information of the channel state. In order to address the challenges caused by imperfect CSIT, the authors in [14] investigate the impacts of imperfect CSIT on the secrecy rate of the system. Specifically, to deal with imperfect CSIT, a worst-case uncertainty channel model is taken into consideration with the goal to mitigate simultaneously inter-user interference and maximize the secrecy sum-rate. The simulation results show the robustness of the proposed solution against the imperfect CSIT of RSMA, and the secure transmission is also guaranteed. Furthermore, in comparison with NOMA, RSMA shows significant secrecy rate enhancement. In [13], a secure beamforming design is also proposed to maximize the weighted sum-rate under user’s secrecy rate requirement. Unlike [3] and [14], the authors in [13] consider the presence of an internal eavesdropper, i.e., an illegitimate user, that not only receives its messages but also wiretaps messages intended for other legitimate users. To deal with this internal eavesdropper, all user’s secrecy rate constraints are taken into consideration. The simulation results suggest that RSMA can outperform the baseline scheme in terms of weighted sum-rate.

All of the above works and others in literature only focus on dealing with the passive eavesdroppers, i.e., the eavesdroppers try to listen passively to the communication channel to derive the original message. To deal with such passive eavesdroppers, the transmitter can adaptively select different transmission rates [13] or utilize artificial noise [3] to confuse the eavesdropper, and thereby minimizing the information disclosure. However, in these cases, the eavesdropper can still detect and receive the signals from the transmitter, and thus it can still decode the information if it has a more powerful hardware computation, e.g., through employing cooperative processing with other eavesdroppers, or better antennas gains compared with those of the transmitter [16, 17]. The passive eavesdropper scenario cannot address the problem in which the eavesdropper is able to manipulate or control the environment [16]. For example, by manipulating the environment, the adversary can bias the resulting bits in the key establishment process [18]. In applications requiring a high security protection, e.g., military and IoT healthcare applications, leaking a small amount of data can result in a break of the whole system and/or cause effects to the users. Therefore, in this work, to further prevent potential information leakage, we focus on a more challenging adversary model in which we need to control the power and transmission rates allocated to users, so that the adversary is unable to detect transmissions on the channels. In this way, the possibility of leaking information can be minimized.

In this work, we aim to develop a novel framework that addresses the aforementioned problems. In particular, we consider a scenario in which an adversary, i.e., a warden, is present in the communication range of a Base Station (BS), i.e., the transmitter, and multiple mobile users, i.e., the receivers. The warden is assumed to be able to observe constantly the channel with a radiometer and interrupt the channel if it detects transmissions on the channel [15, 21]. Thus, it is challenging for the BS to allocate jointly power and control transmission rates for all the messages while hiding these messages from the warden. To minimize the probability of being detected by the warden, a possible policy for the BS is to decrease the power allocated to the messages, so that the warden can be confused the transmitted signals with noise. However, an inappropriate implementation can result in zero data rates at the receivers [15]. To maximize the transmission rate of the system and at the same time guarantee non-zero rate at each user, we formulate the problem of the BS as a max-min fairness problem. In particular, the BS aims to maximize the expected minimum rate (min-rate) of the system under the uncertainty of the environment. Furthermore, we consider a covert constraint that is derived from the theory of covert communications (i.e., low probability of detection communications) [20, 22, 23]. In this way, our proposed framework can help the BS secretly communicate with the legitimate mobile users with a small probability of being detected by the warden.

To find the optimal solution for the optimization formulation above, we develop a learning algorithm based on Proximal Policy Optimization (PPO)  [28]. By leveraging recent advances of deep reinforcement learning techniques [25, 26, 27], our proposed algorithm can effectively find the optimal policy for the BS without requiring complete information of the channel in advance. Specifically, our proposed algorithm takes the feedback from users via the uplink as inputs of the deep neural networks and then outputs the corresponding joint power allocation and rate control policy for the BS. This procedure is similar to the conventional optimization-based schemes in the view of implementation and resource. The differences in our proposed algorithm are twofold. First, our proposed algorithm is a model-free deep reinforcement learning algorithm which does not require a complete model of the environment, i.e., the channel state matrix or channel distribution, in advance. Second, the policy obtained by our proposed algorithm can be adaptively adjusted in cases the channel dynamics change over time, e.g., time-varying channel. Thus, our proposed algorithm is expected to show more flexibility and robustness against the uncertainty and dynamics of the wireless environment. With the obtained optimal policy, we then show that our proposed method can also achieve covert communications between the BS and mobile users by dynamically adjusting power and transmission rates allocated to the messages.

Here, we note that our previous work presented in [24] only focuses on power allocation problem without considering the security impact on the RSMA system. Furthermore, in this current work, we further investigate the rate and security performance of RSMA in the finite blocklength (FBL) regime where the achievable covert rate is limited and no longer follows the Shannon capacity (i.e., infinite blocklength regime). Thus, our framework can be applicable for a wide range of applications which include transmission between IoT devices where the transmitted data is expected to be sporadic and the number of channel uses is limited. In short, our main contributions are as follows:

• •

  We develop a novel stochastic optimization framework to achieve the max-min fairness of the considered RSMA network with the covert constraint. This framework enables the BS to make optimal decisions to maximize the expected min-rate under the covert requirement as well as the dynamics and uncertainty of surrounding environment. To the best of our knowledge, this is the first work that considers covert communications for RSMA. Therefore, our proposed framework is a promising solution for secure and reliable, high data transmission rate applications.

• •

  We propose a highly effective learning algorithm to make the best decisions for the BS. This learning algorithm enables the BS to quickly find the optimal policy through feedback from the users by leveraging advantages of both deep learning and reinforcement learning techniques. Furthermore, our proposed learning algorithm can effectively handle the continuous action and state spaces for the BS through using the PPO technique.

• •

  We conduct intensive simulations to evaluate the efficiency of the proposed framework and reveal insightful information. Specifically, the simulation results show that the positive covert rate is achievable with RSMA in the finite blocklength regime where the achievable covert rate is limited and no longer follows the Shannon capacity (i.e., infinite blocklength regime). More interestingly, with high values of transmission power (i.e., 20 dB to 40 dB), our achievable covert rate can be increased while the achievable covert rate of a baseline multiple access scheme, i.e., SDMA, is saturated. Thus, beyond conventional wireless networks, our framework can be applicable for IoT networks in which the data transmitted between devices is expected to be sporadic and with a relatively small quantity of information .

The rest of our paper is organized as follows. Our system model is described in Section II. Then, we formulate the stochastic optimization problem for the covert-aided RSMA networks in Section III. We provide details of our proposed learning algorithm to maximize the covert rate of the system in Section IV. After that, our simulation results are discussed in Section V, and Section VI concludes the paper.

Before introducing our mathematical formulation, we first describe the fundamentals of DRL. In conventional reinforcement learning (RL) settings, an agent aims to learn an optimal policy through interacting with an environment in discrete decision time steps. At each time step $t$, the agent first observes its current state $s_{t}$ in a state space $\mathcal{S}$ of the system. Based on the observed state $s_{t}$ and current policy $\Omega$, the agent takes an action $a_{t}$ in the action space $\mathcal{A}$. The policy $\Omega$ can be a mapping function from a state to an action (deterministic) or a probability distribution over actions (stochastic). After taking the action $a_{t}$, the agent transits to a new state $s_{t+1}$ and observes an immediate reward $r_{t}$. The goal of the agent is to find an optimal policy that can be obtained by maximizing a discounted cumulative reward.

In conventional RL settings, the agent usually deals with a policy search problem in which the convergence time of the RL algorithm depends on the search space $\mathcal{S}$ and $\mathcal{A}$. In environments with large discrete state-action space or continuous state-action space, the optimal policy is either nearly impossible or time-consuming to find. To address this problem, RL algorithms combined with deep neural networks, namely DRL, show significant performance improvements over conventional RL algorithms [33]. In DRL algorithms, the policy $\Omega$ is defined by a probability distribution over actions, i.e., $\Omega_{\theta}=Pr\{a_{t}|s_{t};\theta\}$, where $\theta$ is a parameter vector of the deep neural network. The parameter vector $\Omega_{\theta}$ can be trained by action-value methods, e.g., DQN [33], or policy gradient methods, e.g., PPO [28]. Action-value methods and policy gradient methods have their advantages and drawbacks which we will discuss later in Section III-C. In the following, we formulate our considered problem in the DRL setting by defining state space, action space, and immediate reward function in which the BS is empowered by an intelligent DRL agent.

We introduce the proposed DRL-based optimization framework for the joint power allocation and transmission rate control problem of the BS as follows. The state space of the BS is defined by:

where $\mathbf{h}_{k}$ is the channel state feedback of the user $u_{k}$ to the BS. $L_{k}$ is the length of the message $W_{k}$ intended for user $u_{k}$. Note that the channel state feedback from the users contains estimation errors, i.e., $\mathbf{h}_{k}=\mathbf{\hat{h}}_{k}+\mathbf{\tilde{h}}_{k}$, where $\mathbf{\hat{h}}_{k}$ is the actual channel state and $\mathbf{\tilde{h}}_{k}$ is the estimation error. The channel of user $u_{k}$ is realized as

where $g_{k}\in\mathbb{R}$ and $\phi\in\mathbb{R}$ are control variables [4]. The channel estimation error follows a complex Gaussian distribution, i.e., $\tilde{\mathbf{h}}_{k}\sim\mathcal{CN}(0,\sigma_{k}^{2})$, where $\sigma_{k}^{2}$ is inversely proportional to the transmission power at the BS, i.e., $\sigma_{k}^{2}=g_{k}P_{t}^{-\alpha_{k}}$where $\alpha_{k}$ is the degree of freedom (DoF) variable [4]. Note that the channel $\mathbf{h}_{w}$ of the warden is unknown to the BS and thus $\mathbf{h}_{w}$ is not included in the state space of the BS. We define the channel between the warden and the BS as follows:

At each time step $t$, the BS allocates the transmission power to the users, splits the messages to common and private messages, and controls transmission rate for the messages. Thus, the action space of the BS is defined as follows:

The reward function is designed to maximize the min-rate of the BS as in (11). To encourage the BS to optimize the min-rate while the covert and QoS constraints of users are taken into consideration, we penalize the BS for each violated constraint. For this, the immediate reward can be defined as follows:

where $p_{t}$ is the penalty received by the BS for action $a_{t}$ that does not satisfy the covert constraint and QoS constraint in (11). The penalty $p_{t}$ is defined as follows:

where $\beta_{0}$ and $\beta_{k}$ are control variables. $\mathbf{1}(a-b)$ is the indicator function in which $\mathbf{1}(a-b)=1$ if $a-b>0$, and otherwise $\mathbf{1}(a-b)=0$. The meaning of $p_{t}$ can be expressed as follows. The penalty is increased with each of covert or QoS constraints, i.e., (11e) and (11f), multiplying with corresponding weights $\beta_{0}$ and $\beta_{k}$ ($k=1,2,\ldots,K$). The penalty is then normalized so that $p_{t}\in[0,1]$ (i.e., the first line of (17)). The penalty can be rewritten as the sum of two components, i.e., covert penalty and QoS penalty as shown in the second line of (17). Note that the penalty of the BS can be calculated by using the feedback mechanism. Once the users receive the messages from the BS, they calculate the data rates of the messages and send the calculated data rates back to the BS along with their minimum rate requirements (QoS requirements) [34]. Based on the feedback from users, the BS can calculate the corresponding QoS penalty. Similarly, the BS can be notified by the users if the channel is interrupted by the warden and the covert penalty can be calculated accordingly. Our designed immediate reward aims to encourage the BS to minimize the penalty $p_{t}$ to 0. Thus, the max-min fairness is guaranteed while covert and QoS constraints are satisfied.

We consider a stochastic policy $\Omega_{\theta}$ of the BS (i.e., $\Omega_{\theta}:\mathcal{S}\times\mathcal{A}\rightarrow[0,1]$), as a probability that action $a_{t}$ is taken given the current state $s_{t}$, i.e., $\Omega_{\theta}=\text{Pr}\{a_{t}|s_{t};\theta\}$, where $\theta$ is the policy parameter vector of the deep neural network. Let $J(\Omega_{\theta})$ denote the expected discounted reward of the BS by following policy $\Omega_{\theta}$:

where $\mathcal{P}(s_{t+1}|s_{t},a_{t})$ is the state transition probability distribution which models the dynamics of the environment, i.e., the dynamics of channel state information. Here, $\mathcal{P}$ is unknown to the BS. Our goal is to find the optimal policy $\Omega_{\theta}^{*}$ for the BS that maximizes $J(\Omega_{\theta}$), i.e.,

Maximizing $J(\Omega_{\theta})$ is very challenging as we consider that the state and action spaces are continuous. It is noted that in (19), we do not require the complete information of the channel, i.e., channel matrix $\mathbf{H}$ in (11), as other works in literature [7, 6, 9, 10]. Instead, the patterns of the channel can be learned through feedback from the users with deep neural networks. For this, we develop a learning algorithm based on a policy gradient method, namely Proximal Policy Optimization (PPO)  [28], to approximate the optimal policy of the BS. PPO is a sample-efficient algorithm which can work under the large continuous state and action spaces and can deal with the uncertainty of the channel state.

As we discussed in Section III-A, action-value methods and policy gradient methods have their own advantages and drawbacks. In action-value methods, each action of the agent can be categorized by a real positive value, e.g., Q-value [33], and once the optimal policy is obtained, the optimal actions can be obtained by selecting the maximum action-value at each state. This family of algorithms is well studied for environments with discrete action space, e.g., a game requires a player to turn left, right, or jump. Thus, the action-value methods are suitable for discrete action space and the optimal policy can be effectively estimated if the number of actions are relatively small. However, in many cases, the action of an agent cannot be categorized by discrete action-values, e.g., a task requires controlling a robot arm by using continuous force. For this, policy gradient methods can be applied by directly estimating the policy of the agent instead of using a greedy selection over action-values. The policy of the agent can be a distribution, e.g., Gaussian, over actions. Therefore, instead of finding the action-values of the agent, policy gradient algorithms aim to find the “shape” of the action distribution, i.e., mean and variance of the distribution.

In our problem, all considered actions in (11) are continuous, and thus only gradient policy methods can be used. In the following, we describe an effective algorithm based on PPO [28] to maximize the min-rate of the BS. The operation of PPO in our proposed framework is illustrated in Fig. 2. In particular, input of the policy update procedure is the joint state of channel and packets’ lengths to be sent at the BS. Output is the BS’s policy, i.e., action distributions. We use one Gaussian distribution to illustrate the output of the policy update in Fig. 2 for the sake of presentation simplicity. In our actual implementation, the action of the BS has multiple dimensions and each dimension can be represented by a Gaussian distribution which differs in mean and variance values. The details of PPO algorithm are as follows.

The PPO uses two deep neural networks as a policy parameter vector and a value function vector, denoted by $\theta$ and $\Theta$, respectively, to efficiently update the policy. The policy parameter vector $\theta$ can be updated by using a gradient ascent method as follows:

where $\alpha$ is the step size, and $\hat{g}_{t}$ is a gradient estimator. The gradient estimator $\hat{g}_{t}$ can be calculated by differentiating a loss function as follows:

It can be observed from (20) and (21) that the choice of the loss function $L(\theta)$ has a significant impact on the policy update. $L(\theta)$ should have a small variance so that it does not cause bad gradient updates which result in significant decreases of $J(\Omega_{\theta})$. Since continuous action space is sensitive to the policy update, a minor negative change in updating $\theta$ can lead to destructively large policy updates [28]. To overcome this problem, PPO algorithm uses a loss function $L^{PPO}(\theta)$ to replace $L(\theta)$:

where $\hat{A}_{t}$ is the advantage function and $u(\varepsilon,\hat{A}_{t})$ is the clip function. $\hat{A}_{t}$ estimates whether the action taken is better than the policy’s default behavior and $u(\cdot)$ limits significant updates which may degrade $J(\Omega_{\theta})$. The advantage function at time step $t$ can be defined by:

where $Q_{t}(s_{t},a_{t};\theta)=\mathbb{E}_{a_{t}\sim\Omega_{\theta},s_{t}\sim\mathcal{P}}\Big{[}\sum_{l=0}^{\infty}\tau^{l}r_{t}(s_{t+l},a_{t+l})\Big{]}$ is the action value function and $V_{t}(s_{t};\Theta)=\mathbb{E}_{s_{t}\sim\mathcal{P}}\Big{[}\sum_{l=0}^{\infty}\tau^{l}r_{t}(s_{t+l},a_{t+l})\Big{]}$ is the state value function. The clip function is thus defined as follows [28]:

The idea of PPO is to prevent the new policy from being attracted to go far away from the old policy $\Omega_{old}$. The first term inside the $\min$ operator in (22), i.e., $\frac{\Omega_{\theta}}{\Omega_{\theta_{old}}}\hat{A}_{t}$, is the surrogate objective which takes into consideration the probability ratio between the new policy and old policy, i.e., $\frac{\Omega_{\theta}}{\Omega_{\theta_{old}}}$. The second term, i.e., $u(\varepsilon,\hat{A}_{t})$, removes the incentive for moving this probability ratio outside of the interval $[1-\varepsilon,1+\varepsilon]$. The pseudo-code of the PPO algorithm is described in Algorithm 1.

The main steps of the PPO algorithm can be described as follows. First, a policy parameter vector $\theta_{0}$ and a value function parameter vector $\Theta_{0}$ are randomly initialized (i.e., lines 2 and 3 in Algorithm 1). Second, at each policy update episode, numbered by $k$, the BS collects a set of trajectories $\mathcal{B}_{k}$, i.e., a batch of state, action, and reward values, by running current policy $\Omega_{\theta_{k}}$ over $T$ time steps (i.e., line 5). After that, the cumulative reward is calculated as in line 6. Next, the BS computes the advantage function as in (23) (i.e., line 6). With the obtained advantage function, the loss function can be calculated as (22) and the policy parameter vector can be updated as line 8 in Algorithm 1. Finally, the value function parameter vector can be updated as in line 9. The procedure is repeated until the cumulative reward values converge to saturating values.

We further analyze the computational complexity of the PPO algorithm used in our considered system. Since the PPO uses deep neural networks as an approximator function, the complexity mostly depends on updating these networks. As the two deep neural networks in PPO share the same architecture, the complexity of updating these networks can be analyzed as follows. Each network consists of an input layer $X_{0}$, two fully-connected layers $X_{1}$ and $X_{2}$, and an output layer $X_{3}$. Let $|X_{i}|$ be the size of the layer $X_{i}$, i.e., the number of neurons in layer $X_{i}$. The complexity of the two networks can be calculated by $2(|X_{0}||X_{1}|+|X_{1}||X_{2}|+|X_{2}||X_{3}|)$. At each episode update, a trajectory, i.e., a batch of state, action, and reward values, are sampled by running the current policy to calculate the advantage function and value function to update the network. Thus, the total complexity of the training process is $O\Big{(}2T|\mathcal{B}_{k}|(|X_{0}||X_{1}|+|X_{1}||X_{2}|+|X_{2}||X_{3})\Big{)}$, where $|\mathcal{B}_{k}|$ is the size of the trajectory sampled from environment. There are two main reasons that PPO is a sample-efficient algorithm. First, the size of a trajectory $\mathcal{B}_{k}$ is relatively small, i.e., from hundreds to thousands [28], compared with the size of a replay memory in conventional action-value methods, e.g., from 50,000 to 1,000,000 in DQN [33] . Here, in our simulations, $|\mathcal{B}_{k}|$ is set at 200. Second, the size of the output layer of PPO is equal to the number of action dimensions. As a result, the size of the output layer can be significantly smaller than those of action-value methods that discrete continuous action space into different chunks to compute the action values, e.g., Q-values. Clearly, the architecture of the deep neural networks are simple enough to be implemented in the base stations which are usually equipped with sufficient computing resources.

We consider our simulation parameters as follows. We use the same parameters for the RSMA and covert communications as those in [4, 30]. The total transmission power of the BS is set to be $P_{t}=20$ (dB). The control variables of channel in (13) are set at $(g_{1},g_{2},g_{3})=(1.0,0.8,0.2)$, and $(\phi_{1},\phi_{2},\phi_{3})=(0,\frac{\pi}{9},\frac{2\pi}{9})$. Furthermore, the channel estimation errors are set at $\sigma_{1}^{2}=P_{t}^{-0.6}$, $\sigma_{2}^{2}=0.8P_{t}^{-0.6}$, and $\sigma_{3}^{2}=0.2P_{t}^{-0.6}$ [4]. Those equivalent values for the warden in (14) are set as $g_{w}=0.4$ and $\phi_{w}=\frac{\pi}{6}$. The covert requirement parameter $\epsilon=0.1$ [30]. Unlike conventional transmission schemes, covert communications require a relatively low data rate to hide information from the warden/adversary. Therefore, we set the QoS requirements of the users to be $R_{1}^{0}=R_{2}^{0}=R_{3}^{0}=10^{-4}$ (bps/Hz). We assume that the length of message $W_{k}$ to be sent at the BS follows uniform distribution with the minimum and maximum values are 0 and 1 kilobits, respectively, i.e., $L_{k}\sim\mathcal{U}(0,1.0)$ (kilobits). The number of antennas at the BS and the number of users are set as $M=K=3$.

For the deep neural networks, our parameters are set as follows. The two deep neural networks representing the policy parameter vector and the value function vector, i.e., $\theta$ and $\Theta$, respectively, share the same architecture. Each deep neural network has two fully connected layer and each layer contains 64 neurons. The number of neurons in the output layer is equal to the number of dimensions of action, i.e., $3K+1$. The number of neurons in the input layer is equal to the number of dimensions of the joint state at the BS (as shown in Fig. 2), i.e., $2K$. The learning rate and clip values of PPO are adopted from [28].

In the following, we investigate the performance of our proposed PPO algorithm on RSMA and SDMA systems, denoted as P-RSMA and P-SDMA, respectively. To further understand the impacts of covert communications on both RSMA and SDMA systems, we run various simulations for scenarios in the FBL regime and infinite blocklength (IBL) regime. It is noted that in the IBL regime, the BS can achieve full data rate with Shannon capacity and the covert constraint, i.e., constraint (11e), is temporarily removed. In the case of covert communications under consideration with FBL, the optimization problem is fully considered as in (11). Furthermore, we introduce other baselines in which a Greedy algorithm is applied. This is to evaluate efficiency of the proposed learning algorithm. These baselines, denoted as G-RSMA and G-SDMA, aim to obtain the maximum immediate reward at each time step, compared with all the historical reward values stored in a buffer, without considering the long-term cumulative reward. In the following, we investigate the performance of all the aforementioned schemes. The considered metrics are (i) average (covert) min-rate (or average reward) and (ii) average (covert) sum-rate.

As defined in the hypothesis test of the warden in (7), we have the distribution of the i.i.d. Gaussian random variables with variance $\sigma_{w}^{2}$ is $\mathbb{P}_{0}=\mathcal{N}(0,\sigma_{w}^{2})$, which corresponds to the case when the BS is not transmitting. Note that the warden does not know the codebook. Therefore, the warden’s probability distribution of the transmitted symbols is of zero-mean i.i.d. Gaussian random variables with variance $P_{f}$. Since we have the signal $\mathbf{x}$ is transmitted with power $P_{t}$ at the transmitter and channel between the warden and the BS is defined in (14), we have $P_{f}=g_{w}P_{t}$. Therefore, the distribution of $\mathbb{P}_{1}$ is as follows:

Let $a=\sigma_{w}$ and $b=\sqrt{g_{w}P_{t}+\sigma_{w}^{2}}$, we have the respective probability distribution functions of $\mathbb{P}_{0}$ and $\mathbb{P}_{1}$ are as follows:

The relative entropy between $\mathbb{P}_{0}$ and $\mathbb{P}_{1}$ is then calculated by:

Now we need to solve $\mathcal{D}_{1}$. We expand $\mathcal{D}_{1}$ and apply linearity:

We first solve $\mathcal{D}_{2}$. For this, we integrate $\mathcal{D}_{1}$ by parts, i.e., $\int fg^{\prime}=fg-\int f^{\prime}g$. Let $f=x$ and $g^{\prime}=x\mathrm{e}^{-\frac{x^{2}}{2a^{2}}}$, we can calculate $f^{\prime}=1$ and $g=-a^{2}\mathrm{e}^{-\frac{x^{2}}{2a^{2}}}$. We now have:

$\mathcal{D}_{4}$ can be solved as follows. We substitute $u=\frac{x}{\sqrt{2}a}\rightarrow\mathrm{~{}d}x=\sqrt{2}a\mathrm{~{}d}u$. $\mathcal{D}_{4}$ becomes:

Note that we have a special integral in $\mathcal{D}_{4}$, i.e., $\int_{-\infty}^{+\infty}\frac{2\mathrm{e}^{-u^{2}}}{\sqrt{\pi}}\mathrm{d}u=\text{ erf}(u)$ is a Gauss error function. Let’s plug in $\mathcal{D}_{4}$:

Plug $\mathcal{D}_{4}$ in $\mathcal{D}_{2}$:

Once $\mathcal{D}_{2}$ is solved, $\mathcal{D}_{3}=\int_{-\infty}^{+\infty}\mathrm{e}^{-\frac{x^{2}}{2a^{2}}}\mathrm{~{}d}x$ can be calculated as follows. Let’s substitute $u=\frac{x}{\sqrt{2}a}\rightarrow\mathrm{~{}d}x=\sqrt{2}a\mathrm{~{}d}u$. $\mathcal{D}_{3}$ becomes:

Use the previous result of Gauss error function, we have:

Once $\mathcal{D}_{2}$ and $\mathcal{D}_{3}$ are solved, let’s plug (33) and (35) into (29):

Finally, we have:

Undo substitution for $a=\sigma_{w}$ and $b=\sqrt{g_{w}P_{t}+\sigma_{w}^{2}}$, we have:

The proof of Proposition 1 is now completed.