Distributed Computation Offloading for Energy Provision Minimization in WP-MEC Networks with Multiple HAPs

As shown in Fig. 1, we study a WP-MEC network, where $M$ one-antenna HAPs, and $N$ one-antenna WDs coexist. Let $\mathcal{M}=\{1,2,...,M\}$ denote the set of HAPs, and $\mathcal{N}=\{1,2,...,N\}$ denote the set of WDs. Equipped with one MEC server, each HAP broadcasts the RF signal to WDs through the downlink channel, and receives the computation data from WDs through the uplink channel. Equipped with one battery with capacity $E_{b}$, each WD harvests energy from RF signals for local computing or offloading the computation data to the chosen HAP. We consider that WDs adopt the harvest-then-offload protocol [15], i.e., WDs harvest energy before offloading data to the HAPs. In accordance with the real demand, we define the transmission zone as a circle centered at the WD with radius $R_{t}$. The WD could offload its computation data to the HAPs within the corresponding transmission zone.

To process the generated computation data in the WP-MEC network, we consider that WDs follow the binary offloading policy, i.e., there are two computing modes for WDs. The WD in local computing mode processes the computation data locally by utilizing the computing units, and the WD in edge computing mode offloads the computation data to the chosen HAP through the uplink channel. After receiving the computation data, the HAP processes the received computation data, and transmits computation results back to the WD through the downlink channel.

As illustrated in Fig. 2, the system time is divided into $\mathcal{T}$ time slots with equal duration $T$. Time slot $t$ is divided into the control information exchanges (CIE) phase with negligible duration due to small-sized control information [27], the WPT phase with duration $\alpha(t)$, and the computation data offloading (CDO) phase with duration $T-\alpha(t)$. During the CIE phase, the control information exchanges between HAPs and WDs complete. During the WPT phase, HAPs broadcast RF signals to WDs through the downlink channels, and WDs harvest energy from RF signals of HAPs simultaneously. During the CDO phase, the WDs in edge computing mode offload the computation data to the chosen HAPs through the uplink channels. While the WDs in local computing mode process the computation data during the time slot, since the energy harvesting circuit and the computing unit could work simultaneously [15, 21], as shown in the WD’s circuit structure of Fig. 1. At the beginning of time slot $t$, $N$ WDs generate computation data $\mathcal{D}(t)=[D_{1}(t),...,D_{N}(t)]$, where $D_{1}(t),...,D_{N}(t)$ respectively denote the amount of the computation data generated by WD₁, $...$, WD_N at time slot $t$, and WDs transmit the state information about the amount of the generated computation data and the amount of the energy in the batteries to the HAPs. Then the HAPs follow the proposed TMADO framework to output the feasible solutions of the offloading decisions, broadcast them to WDs, and conduct WPT with duration $\alpha(t)$. According to the received feasible solutions of the offloading decisions, each WD independently makes the optimal offloading decision, i.e., it processes the generated computation data locally during the current time slot, or offloads the computation data to the chosen HAP during the CDO phase. To avoid potential interference, the time division multiple access (TDMA) protocol is employed by WDs that offload the computation data to the same HAP at the same time slot, while the frequency division multiple access (FDMA) protocol is employed by WDs that offload the computation data to different HAPs. As the MEC server of the HAP has high CPU frequency [24], there is no competition among WDs for the edge resource. Similarly, as the MEC server of the HAP has high CPU frequency [24] and the computation results are small-sized [15], we neglect the time spent by HAPs on processing data and transmitting the computation results back to the WDs.

For the channels in the WP-MEC network, we adopt the free-space path loss model for the large-scale fading, and adopt the Rayleigh fading model [28] for the small-scale fading. Let ${h}_{n,m}(t)$ denote the channel gain between WD_n and HAP_m at time slot $t$ as

where ${\sigma}_{L,n,m}$ denotes the large-scale fading component, and ${\sigma}_{S,n,m}(t)$ denotes the small-scale fading component at time slot $t$. The uplink channel and downlink channel are considered to be symmetric [14], i.e., the channel gain of the uplink channel equals that of the corresponding downlink channel. We consider that channel gains remain unchanged within each time slot, and vary from one time slot to another.

During the WPT phase, WDs harvest energy from the RF signals broadcasted by HAPs. The amount of the energy harvested by WDs depends on the transmit power of the HAPs, the distance between WDs and HAPs, and the duration of the WPT phase. We adopt the linear energy harvesting (EH) model, and formulate the amount of the energy harvested by WD_n at time slot $t$ as

where $\mu\in(0,1)$ denotes the EH efficiency, ${P}_{{h},m}(t)\in[0,{P}_{\max}]$ denotes the transmit power of HAP_m at time slot $t$, and ${P}_{\max}$ denotes the maximum transmit power of HAPs.

The amount of the available energy in WD_n at time slot $t$, i.e., the sum of the amount of the initial energy in WD_n and the amount of the energy harvested by WD_n at time slot $t$, is formulated as

where $E_{i,n}(t)$ denotes the amount of the initial energy in WD_n at time slot $t$. At the beginning of the first time slot, the initial energy in WDs equals zero, i.e., $E_{i,n}(0)=0,\forall n\in\mathcal{N}$. As shown in (3), $\text{min}\{E_{i,n}(t)+E_{h,n}(t),E_{b}\}$ ensures that the amount of the available energy in WD_n does not exceed the battery capacity. Then the amount of initial energy in WD_n at time slot $t+1$ updates as

where $E_{{w},n}(t)$ denotes the amount of the energy consumed by WD_n at time slot $t$.

According to the computing modes, WDs are divided into nonoverlapping sets $\mathcal{N}_{m}(t)$, $\mathcal{N}_{0}(t)$, and $\mathcal{N}_{\text{-1}}(t)$. Let $\mathcal{N}_{m}(t),m\in\mathcal{M}$ denote the set of WDs that process the computation data by offloading it to HAP_m at time slot $t$, $\mathcal{N}_{0}(t)$ denote the set of WDs that process the computation data locally at time slot $t$, and $\mathcal{N}_{\text{-1}}(t)$ denote the set of WDs that fail to process the computation data at time slot $t$. If the available energy in WD is not enough for local computing in (9) or edge computing in (13), WD fails to process the computation data at time slot $t$. The WDs failing to process the computation data at time slot $t$ are not in the feasible solutions of the offloading decisions $(\mathcal{N}-\mathcal{N}_{\text{-1}}(t))$ from HAPs. Considering the quality of service requirement for the WDs in $\mathcal{N}_{\text{-1}}(t)$, the WDs in $\mathcal{N}_{\text{-1}}(t)$ directly drop the computation data without retransmitting, and accumulate the energy in the batteries so as to successfully process the computation data of the following time slots. $\mathcal{N}_{\text{-1}}(t)$ varies across time slots since WDs with sufficient harvested energy process the generated computation data locally, or offload it to the chosen HAPs. Then $\mathcal{N}_{m}(t)$, $\mathcal{N}_{0}(t)$, and $\mathcal{N}_{\text{-1}}(t)$ satisfy

where $\mathcal{N}_{x}(t)\cap\mathcal{N}_{y}(t)=\phi$, $x,y\in\{-1,...,M\}$, $x\neq y$, and $\phi$ denotes the empty set.

According to the received feasible solutions of the offloading decisions from HAPs, when WD_n chooses local computing mode at time slot $t$, i.e., $n\in\mathcal{N}_{0}(t)$, it processes the generated computation data of $D_{n}(t)$ bits locally during the whole time slot. Let $f_{n}(t)$ with $f_{n}(t)\in(0,f_{\max}]$ denote the CPU frequency of WD_n by using dynamic voltage and frequency scaling technique [29] at time slot $t$, where $f_{\max}$ denotes the maximum CPU frequency of WDs. Let $C_{n}>0$ denote the number of CPU cycles required by WD_n to process 1-bit data. The computing delay constraint, i.e., the delay of WD_n to process the generated computation data $D_{n}(t)$ at time slot $t$, denoted by $\tau_{l,n}(t)$, does not exceed the duration of the time slot $T$, is formulated as

Since energy is the product of power and time, the amount of the energy consumed by WD_n for local computing at time slot $t$ is the product of the CPU power of WD_n, denoted by $P_{\text{cpu},n}(t)$, and the delay of WD_n to process the generated computation data, i.e., $\tau_{l,n}(t)$. According to [30] and [31], the CPU power of WD_n at time slot $t$ is expressed as

where $C_{L,n}$ denotes the switched load capacitance of WD_n, and $V_{n}(t)$ denotes the CPU voltage of WD_n at time slot $t$. As pointed out by [31], when the CPU operates at the low voltage limit, which is in accordance with the real-world WDs, the CPU frequency is approximately linear with the CPU voltage. Consequently, $P_{\text{cpu},n}(t)$ in (7) can be reformulated as $k_{n}f_{n}^{3}(t)$, where $k_{n}$ satisfying $C_{L,n}V_{n}^{2}(t)=k_{n}f_{n}^{2}(t)$ is called the effective switched load capacitance of WD_n [32]. Then the amount of the energy consumed by WD_n for local computing at time slot $t$ is expressed as

which has been extensively used in the works related to MEC networks [14, 15, 20, 21, 32].

Since WDs in local computing mode are powered by the energy harvested during the WPT phase and that stored in the batteries, the energy constraint for WD_n in local computing mode, i.e., the amount of the energy consumed by WD_n for local computing at time slot $t$ is no larger than the available energy in WD_n at time slot $t$, is formulated as [24]

According to the received feasible solutions of the offloading decisions from HAPs, when WD_n chooses edge computing mode at time slot $t$, it offloads the computation data to HAP_m, i.e., $n\in\mathcal{N}_{m}(t)$, $m\in\mathcal{M}$. Let $\tau_{o,n,m}(t)$ denote the duration that WD_n offloads the computation data of $D_{n}(t)$ bits to HAP_m at time slot $t$. Then $\tau_{o,n,m}(t)$ and $\alpha(t)$ satisfy

Based on Shannon’s formula [33], to ensure that WD_n in edge computing mode successfully offloads the computation data to HAP_m during the CDO phase, $\tau_{o,n,m}(t)$ satisfies

where $B$ denotes the uplink bandwidth of each HAP, and ${v}\geq 1$ represents the communication overhead including the encryption and data header costs [14]. $P_{n}$ denotes the transmit power of WD_n, and ${N}_{0}$ denotes the power of the additive white Gaussian noise (AWGN). Besides, there is no interference between multiple WDs in (11) due to the TDMA protocol. Let $E_{o,n}(t)$ denote the amount of the energy consumed by WD_n for transmitting the computation data of $D_{n}$ bits to HAP_m at time slot $t$ as

where $P_{c,n}$ denotes the circuit power of WD_n. During the CDO phase, the energy constraint for WD_n in edge computing mode, i.e., the amount of the energy consumed by WD_n for edge computing is no larger than that of the available energy in WD_n at time slot $t$, is formulated as

Then the amount of the energy consumed by WD_n at time slot $t$ is defined as

If the WD fails to process the computation data under the TMADO framework at time slot $t$, its computation data would not be scheduled [23], and $E_{{w},n}(t)$ equals $0$.

The amount of the energy consumed by HAP_m at time slot $t$, denoted by $E_{{h},m}(t)$, is formulated as

(15) indicates that the energy provision by HAPs consists of the amount of the energy consumed by HAPs for broadcasting RF signals and the amount of the energy consumed by HAPs for processing the received computation data from WDs. During the WPT phase of time slot $t$, the amount of the energy consumed by HAP_m for broadcasting the RF signal, denoted by $E_{1,m}(t)$, is formulated as

During the CDO phase of time slot $t$, the amount of the energy consumed by HAP_m for processing the received computation data of $\sum_{n\in{\mathcal{N}_{m}(t)}}D_{n}(t)$ bits from the WDs in $\mathcal{N}_{m}(t)$, denoted by $E_{2,m}(t)$, is formulated as [34]

where $e_{m}$ represents the energy consumption of HAP_m for processing per offloaded bit.

We formulate the WCDO subproblem as a Markov decision process (MDP), represented by a tuple of the high-level agent $(\mathcal{S}^{h},\mathcal{A}^{h},\mathcal{R}^{h},\gamma^{h})$, where $\mathcal{S}^{h}$, $\mathcal{A}^{h}$, $\mathcal{R}^{h}$, and $\gamma^{h}$ represent state, action, reward, and the discount factor, respectively. The high-level agent receives state $\bm{s}^{h}(t)\in\mathcal{S}^{h}$, selects action $\bm{a}^{h}(t)\in\mathcal{A}^{h}$ at time slot $t$, and receives reward $r^{h}(t)\in\mathcal{R}^{h}$ and state $\bm{s}^{h}(t+1)$.

• •

  State space: The global state of the WCDO subproblem at time slot $t$ is defined as

  $${}\begin{split}\bm{s}^{h}(t)=\{\bm{s}_{p}^{h}(t),\bm{s}_{w}^{h}(t),\bm{s}_{c}^{h}(t)\},\end{split}$$

  (22)

  where $\!\bm{s}_{p}^{h}(t)\!\!\!\!=\!\!\!\!\{E^{\text{tot}}_{{h},1}(t),...,E^{\text{tot}}_{{h},M}(t)\}$ represents the state information of HAPs, $\bm{s}_{w}^{h}(t)=\{D_{1}(t),...,D_{N}(t),E_{i,1}(t),...,E_{i,N}(t)\}$ represents that of WDs, and $\bm{s}_{c}^{h}(t)=\{{h}_{n,m}(t)|n\in\mathcal{N},m\in\mathcal{M}\}$ represents the channel gains at time slot $t$. $E^{\text{tot}}_{{h},m}(t)$ is the state information of HAP_m that represents the total amount of the energy consumed by HAP_m during time slots $[0,t-1)$. $E_{i,n}(t)$ represents the amount of the initial energy in WD_n at time slot $t$.

• •

  Action space: At time slot $t$, the action space of HAPs is defined as

  $${}\begin{split}\!\mathcal{A}^{h}(t)\!\!=\!\!\{\alpha(t),{P}_{{h},1}(t),...,&{P}_{{h},M}(t),\\ &a_{{c},1}(t),...,a_{{c},N}(t)\}\!\!\in\!\mathcal{A}^{h},\end{split}$$

  (23)

  where $a_{\text{c},n}(t)$ represents the energy provision cost of HAPs for the energy supply of WD_n estimated by the high-level agent, i.e., the amount of the energy consumed by HAPs for WD_n to harvest per joule energy. To be specific, if the energy provision cost of HAPs for the energy supply of WD_n is small, the probability that HAPs determine that WD_n in local computing mode or edge computing mode processes the computation data is high. Otherwise, the probability that HAPs determine that WD_n is not in the feasible solutions of the offloading decisions is high. As $a_{\text{c},n}(t)$ is a continuous action, and the feasible solutions of the offloading decisions are discrete actions, we convert continuous action $a_{\text{c},n}(t)$ into discrete action, which represents the set of WDs to process the computation data, and obtain $\mathcal{N}_{\text{-1}}(t)$. To optimize the feasible solutions of the offloading decisions, the sub-action space $\mathcal{A}_{c}^{h}(t)$ is established in the ascending order of action $a_{\text{c},n}(t)$ as

  $${}\mathcal{A}_{c}^{h}(t)=\text{a-sorted}({a_{c,n}(t)}),\forall n\in\mathcal{N}.$$

  (24)

  The WDs in the feasible solutions of the offloading decisions are powered by HAPs with low energy provision cost, and satisfy (19b). Based on (24), we obtain the feasible solutions of the offloading decisions.

• •

  Reward function: The reward function measures the amount of the energy provision by HAPs at time slot $t$ as

  $$r^{h}(t)=-\sum_{m=1}^{M}{E_{{h},m}(t)}-\omega_{d},$$

  (25)

  where $\omega_{d}$ denotes the penalty of dissatisfying (19b). The high-level agent maximizes a series of rewards $r^{h}$ as

  $$r^{h}=\sum_{t=0}^{\mathcal{T}-1}\gamma^{h}~r^{h}(t).$$

  (26)

We formulate the ODO subproblem as a decentralized partially observable Markov decision process (Dec-POMDP), represented by a tuple of low-level agents $(\mathcal{S}^{l},\mathcal{O}^{l},\mathcal{A}^{l},\mathcal{R}^{l},$ Pr$,\gamma^{l})$, where $\mathcal{S}^{l}$, $\mathcal{O}^{l}$, $\mathcal{A}^{l}$, $\mathcal{R}^{l}$, Pr, and $\gamma^{l}$ represent states, local observations, actions, rewards, the transition probability, and the discount factor, respectively. As low-level agents, WDs adapt actions to maximize the rewards based on local observations. At the beginning of time slot $0$, the global state of the ODO subproblem is initialized as $\bm{s}^{l}(0)$. At time slot $t$, WD_n has observation $\bm{o}^{l}_{n}(t)$ from state $\bm{s}^{l}(t)$, and then outputs action $\bm{a}^{l}_{n}(t)$. The environment receives the set of $N$ WDs’ actions $\bm{a}^{l}(t)=\{{\bm{a}^{l}_{n}(t)}\}^{N}_{n=1}$, and calculates reward $r^{l}_{n}(t)=\mathcal{R}^{l}_{n}(\bm{s}^{l}(t),\bm{a}^{l}(t)),n\in\mathcal{N}$ for $N$ WDs. Then the environment transites to state $\bm{s}^{l}(t+1)$ according to transition probability Pr$(\bm{s}^{l}(t+1)|\bm{s}^{l}(t),\bm{a}^{l}(t))$.

• •

  State space: The global state of the ODO subproblem at time slot $t$ is defined as

  $$\begin{split}\bm{s}^{l}(t)\!=\!\!\{\bm{s}_{p,1}^{l}(t),...,\bm{s}_{p,M}^{l}(t),&\bm{s}_{w,1}^{l}(t),...,\bm{s}_{w,N}^{l}(t),\\ &\bm{s}_{c,1}^{l}(t),...,\bm{s}_{c,M}^{l}(t)\},\end{split}$$

  (33)

  where $\bm{s}_{p,m}^{l}(t)=\{E^{\text{tot}}_{{h},m}(t),T-\alpha(t)\}$ represents the state information of HAP_m, $\bm{s}_{w,n}^{l}=\{D_{n}(t),E_{n}(t),a_{c,n}(t)\}$ represents that of WD_n, and $\bm{s}_{c,m}^{l}(t)=\{{h}_{n,m}(t)|n\in\mathcal{N}\}$ represents the channel gains between $N$ WDs and HAP_m.

• •

  Observation space: $\bm{o}^{l}_{n}(t)$ denotes the observable state by WD_n from the global state of ODO subproblem $\bm{s}^{l}(t)$ as

  $${}\begin{split}\bm{o}^{l}_{n}(t)\!=\!\!\{\bm{o}_{p,1}^{l}(t),...,\bm{o}_{p,M}^{l}(t),&\bm{o}_{w,1}^{l}(t),...,\bm{o}_{w,N}^{l}(t),\\ &\bm{o}_{c,1}^{l}(t),...,\bm{o}_{c,M}^{l}(t)\},\end{split}$$

  (34)

  where $\bm{o}_{p,m}^{l}(t)=\{E^{\text{tot}}_{{h},m}(t),T-\alpha(t)\}$ represents the observation information of HAP_m, $\bm{o}_{w,n}^{l}=\{D_{n}(t),E_{n}(t),a_{c,n}(t)\}$ represents the observation information of WD_n, and $\bm{o}_{c,m}^{l}(t)=\{{h}_{n,m}(t)|n\in\mathcal{N}\}$ represents the channel gains between $N$ observable WDs and HAP_m. The size of $\bm{o}^{l}_{n}(t)$ for WD_n is the same as that of $\bm{s}^{l}(t)$. If HAP_m is located outside the transmission zone of WD_n, the observation information of HAP_m is considered as $\bm{o}_{p,m}^{l}(t)=\{0,0\}$, and the channel gain between WD_n and HAP_m is considered as $h_{n,m}(t)=0$ in $\bm{o}_{c,m}^{l}(t)$. As WD_n can not capture the observation information of the other WDs, the corresponding observation information of other WDs is considered as $\bm{o}_{w,k}^{l}(t)=\{0,0,0\}$ for $k\in\mathcal{N}\ \&\ k\neq n$ in (34).

• •

  Action space: $\mathcal{A}_{n}^{l}(t)=\{x_{n}(t)\}$ represents the action space of WD_n at time slot $t$. $x_{n}(t)=m$, $m\in\mathcal{M}$ represents that WD_n in edge computing mode offloads the computation data to HAP_m at time slot $t$, and $x_{n}(t)=0$ represents that WD_n in local computing mode processes the computation data locally at time slot $t$.

• •

  Reward function: In view of observation $\bm{o}^{l}_{n}(t)$, WD_n outputs offloading policy $\bm{a}^{l}_{n}(t)$ to interact with the environment, and receives reward $r^{l}_{n}(t)$ as

  $${r^{l}_{n}(t)}=\begin{cases}u-(a_{c,n}(t)E_{o,n}(t)+\!\!\!\!\!\!\!&e_{m}D_{n}(t)),\\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ {\text{if}}\ n\in\!\!\!\!&\mathcal{N}_{m}(t),m\in\mathcal{M};\\ u-a_{c,n}(t)E_{l,n}(t),&{\text{if}}\ n\in\mathcal{N}_{0}(t);\\ 0,&\text{otherwise},\end{cases}$$

  (35)

  where $u$ denotes a constant for the non-negative reward. The value of $u$ needs to satisfy two requirements: 1) $u>\max\{a_{c,n}(t)E_{o,n}(t)+e_{m}D_{n}(t),a_{c,n}(t)E_{l,n}(t)\}$ for $t\in[0,\mathcal{T}-1]$; 2) the value of $u$ and those of $a_{c,n}(t)E_{o,n}(t)+e_{m}D_{n}(t)$ and $a_{c,n}(t)E_{l,n}(t)$ are similar in the order of magnitude¹¹1We set the value of $u$ as the average upper bound of the energy provision by HAPs for the WP-MEC network with a single WD in edge computing mode over all time slots. Namely, when HAPs with the maximum transmit power $P_{\max}$ provide energy for the WD in edge computing mode during $\mathcal{T}$ time slots, the average value over all time slots, including the energy consumed by HAPs for broadcasting RF signals and that for processing the received computation data from the WD, is defined as $${}\begin{split}\overline{\Psi}=\frac{\mathcal{T}MP_{\max}T+\mathop{max}\limits_{m}\{e_{m}\}\sum_{t=1}^{\mathcal{T}}\mathbb{E}\left(D_{n}(t)\right)}{\mathcal{T}},\end{split}$$ (36) where $\mathbb{E}\left(D_{n}(t)\right)$ represents the expectation of offloaded bits from WD_n at time slot $t$.. The TMADO framework aims to maximize the cumulative sum of (35), i.e., the total reward of low-level agents, by minimizing the energy provision by HAPs in (19a). In the solution of $\mathbf{P0}$, HAPs only need to provide the required amount of energy for WDs in local computing mode or edge computing mode in (19a).

  Based on (22) and (24), HAPs determine the feasible solutions of the offloading decisions $(\mathcal{N}-\mathcal{N}_{\text{-1}}(t))$, and broadcast them at the beginning of time slot $t$. Then WDs receive the feasible solutions of the offloading decisions, and output actions $\bm{a}^{l}(t)$ based on local observations. For WD_n, $n\in\mathcal{N}$, the ODO subproblem aims to find the optimal policy $\pi^{l*}_{\bm{\theta}^{l}_{a,n}}$ that maximizes the long-term accumulated discounted reward as

  $$~~\underset{\pi^{l*}_{\bm{\theta}^{l}_{a,n}}}{\text{max}}~~~\mathbb{E}\left[\sum_{t=0}^{\mathcal{T}-1}\gamma^{l}_{t}~r^{l}_{n}(t)\right].$$

  (37)

  Let $\bm{\theta}^{l}_{a,n}$ denote the parameters of the actor network of low-level agent $n$,