Multi-Agent Meta-Reinforcement Learning for Self-Powered and Sustainable Edge Computing Systems

The problem of energy management for MEC-enabled wireless networks has been studied in [16, 17, 18, 19, 20, 21, 22] (summary in Table II). In [16], the authors proposed a joint mechanism for radio resource management and users task offloading with the goal of minimizing the long-term power consumption for both mobile devices and the MEC server. The authors in [17] proposed a heuristic to solve the joint problem of computational resource allocation, uplink transmission power, and user task offloading problem. The work in [18] studied the tradeoff between communication and computation for a MEC system and the authors proposed a MEC server CPU scaling mechanism for reducing the energy consumption. Further, the work in [19] proposed an energy-aware mobility management scheme for MEC in ultra-dense networks, and they addressed the problem using Lyapunov optimization and multi-armed bandits. Recently, the authors in [21] proposed a distributed power control scheme for a small cell network by using the concept of a multi-agent calibrate learning. Further, the authors in [22] studied the problem of energy storage and energy harvesting (EH) for a wireless network using deviation theory and Markov processes. However, all of these existing works assume that the consumed energy is available from the energy utility source to the wireless network system [16, 17, 18, 19, 20, 21, 22]. Since the assumed models are often focused on energy management and user task offloading on network resource allocations, the random demand for computational (e.g., CPU computation, memory, etc.) and communication requirements of the edge applications and services are not considered. In fact, even if enough energy supply is available, the energy cost related to network operation can be significant because of the usage of non-renewable (e.g., coal, petroleum, natural gas). Indeed, it is necessary to include renewable energy sources towards the next-generation wireless networking infrastructure.

Recently, some of the challenges of renewable energy powered wireless networks have been studied in [8, 9, 11, 10, 12, 13, 14, 23]. In [8], the authors proposed an online optimization framework to analyze the activation and deactivation of BSs in a self-powered network. In [9], proposed a hybrid power source infrastructure to support heterogeneous networks (HetNets), a model-free deep reinforcement learning (RL) mechanism was proposed for user scheduling and network resource management. In [10], the authors developed an RL scheme for edge resource management while incorporating renewable energy in the edge network. In particular, the goal of [10] is to minimize a long-term system cost by load balancing between the centralized cloud and edge server. The authors in [11] introduced a microgrid enabled edge computing system. A joint optimization problem is studied for MEC task assignment and energy demand-response (DR) management. The authors in [11] developed a model-based deep RL framework to tackle the joint problem. In [12], the authors proposed a risk-sensitive energy profiling for microgrid-powered MEC network to ensure a sustainable energy supply for green edge computing by capturing the conditional value at risk (CVaR) tail distribution of the energy shortfall. The authors in [12] proposed a multi-agent RL system to solve the energy scheduling problem. In [13], the authors proposed a self-sustainable mobile networks, using graph-based approach for intelligent energy management with a microgrid. The authors in [14] proposed a smart grid-enabled wireless network and minimized grid energy consumption by applying energy sharing among the BSs. Furthermore, in [23], the authors addressed challenges of non-coordinated energy shedding and mis-aligned incentives for mixed-use building (i.e., buildings and data centers) using auction theory to reduce energy usage. However, these works [9, 11, 10, 12, 13, 14, 23] do not investigate the problem of energy dispatch nor do they account for the energy cost of MEC-enabled, self-powered networks when the demand and generation of each self-powered BS are non-i.i.d.. Dealing with non-i.i.d. energy demand and generation among self-powered BSs is challenging due to the intrinsic energy requirements of each BS evolve the uncertainty. In order to overcome this unique energy dispatch challenge, we propose to develop a multi-agent meta-reinforcement learning framework that can adapt new uncertain environment without considering the entire past experience.

Some interesting problems related to meta-RL and multi-agent deep RL are studied in [24, 25, 26, 27, 28] (summary in Table II). In [24], the authors focused on studying the challenges of the tradeoff between effectiveness and available amounts of training data for a deep-RL based learning system. To this end, the authors in [24] tackled those challenges by exploring a deep meta-reinforcement learning architecture. This learning architecture comprises of two learning systems: 1) lower-level system that can learn each new task very quickly, 2) higher-level system is responsible to improve the performance of each lower-level system task. In particular, this learning mechanism is involved with one lower-level system that can learn relatively quickly as compared with a higher-level system. This lower-level system can adapt to a new task while a higher-level system performs fine-tuning so as to improve the performance of the lower-level system. In particular, in deep meta-reinforcement learning, a lower-level system quantifies a reward based on the desired action and feeds back that reward to a higher-level system to tune the weights of a recurrent network. However, the authors in [24] do not consider a stochastic environment nor do they extend their work for a multi-agent scenario. The authors in [25] proposed a stochastic gradient-based meta-parameter learning scheme for tuning reinforcement learning parameters to the physical environmental dynamics. Particularly, the experiment in [25] performed in both animal and robot environments, where an animal must recognize food before it starves and a robot must recharge before the battery is empty. Thus, the proposed scheme can effectively find meta-parameter values and controls the meta-parameter in both static and dynamic environments. In [26], the authors investigated a learning to learn (i.e., meta-learning) mechanism with the recurrent neural networks, where the meta-learning problem was designed as a generalized transfer learning scheme. In particular, the authors in [26] considered a parametrized optimizer that can transfer the neural network parameters update to an optimizee. Meanwhile, the optimizee can determine the gradients without relying on the optimizer parameters. Moreover, the optimizee sends the error to the optimizer, and updates its own parameters based on the transferred parameters. This mechanism allows an agent to learn new tasks for a similar structure. An asynchronous multi-agent RL framework was studied in [27], where the authors investigated how parallel actor learners of asynchronous advantage actor-critic (A3C) can achieve better stability during the neural network training comparted to asynchronous RL schemes. Such schemes include asynchronous one-step Q-learning, one-step Sarsa, and n-step Q-learning. The authors in [28] proposed a general-purpose multi-agent scheme by adopting the framework of centralized training with decentralized execution. In particular, the authors in [28] proposed an extension of the actor-critic policy gradient mechanism by modifying the role of the critic. This critic is augmented with an additional policy information from the other actors (agents). Sequentially, each local actor executes in a decentralized manner and sends its own policy to the centralized critic for further investigation. However, the environment (i.e., state information) of this model remains the same for all of the local actors while in our setting the environment of each BS agent is deferent from others based on its own energy demand and generation. Moreover, the works in [24, 25, 26, 27, 28], do not consider a multi-agent environment in which the policy of each agent relies on its own state information. In particular, such state information belongs to a non-i.i.d. learning environment when environmental dynamics become distinct among the agents.

The main contribution of this paper is a novel energy management framework for next-generation MEC in self-powered wireless network that is reliable against extreme uncertain energy demand and generation. We formulate a two-stage stochastic energy cost minimization problem that can balance renewable, non-renewable, and storage energy without knowing the actual demand. In fact, the formulated problem also investigates the realization of renewable energy generation after receiving the uncertain energy demand from the MEC applications and service requests. To solve this problem, we propose a multi-agent meta-reinforcement learning (MAMRL) framework that dynamically observes the non-i.i.d. behavior of time-varying features in both energy demand and generation at each BS and, then transfers those observations to obtain an energy dispatch decision and execute the energy dispatch policy to the self-powered BS. Fig. 1 illustrates how we propose to dispatch energy to ensure sustainable edge computing over a self-powered network using MAMRL framework. As we can see, each BS that includes small cell base stations (SBSs) and a macro base station (MBS) will act as a local agent and transfer their own decision (reward and action) to the meta-agent. Then, the meta-agent accumulates all of the non-i.i.d. observations from each local agent (i.e., SBSs and MBS) and optimizes the energy dispatch policy. The proposed MAMRL framework then provides feedback to each BS agent for exploring efficiently that acquire the right decision more quickly. Thus, the proposed MAMRL framework ensures autonomous decision making under an uncertain and unknown environment. Our key contributions include:

• •

  We formulate a self-powered energy dispatch problem for MEC-supported wireless network, in which the objective is to minimize the total energy consumption cost of network while considering the uncertainty of both energy consumption and generation. The formulated problem is, thus, a two-stage linear stochastic programming. In particular, the first stage makes a decision when energy demand is unknown, and the second stage discretizes the realization of renewable energy generation after knowing energy demand of the network.

• •

  To solve the formulated problem, we propose a new multi-agent meta-reinforcement learning framework by considering the skill transfer mechanism [24, 25] between each local agent (i.e., self-powered BS) and meta-agent. In this MAMRL scheme, each local agent explores its own energy dispatch decision using Markovian properties for capturing the time-varying features of both energy demand and generation. Meanwhile, the meta-agent evaluates (exploits) that decision for each local agent and optimizes the energy dispatch decision. In particular, we design a long short-term memory (LSTM) as a meta-agent (i.e., run at MBS) that is capable of avoiding the incompetent decision from each local agent and learns the right features more quickly by maintaining its own state information.

• •

  We develop the proposed MAMRL energy dispatch framework in a semi-distributed manner. Each local agent (i.e., self-powered BS) estimates its own energy dispatch decision using local energy data (i.e., demand and generation), and provides observations to the meta-agent individually. Consequently, the meta-agent optimizes the decision centrally and assists the local agent toward a globally optimized decision. Thus, this approach not only reduces the computational complexity and communication overhead but it also mitigates the curse of dimensionality under the uncertainty by utilizing non-i.i.d. energy demand and generation from each local agent.

• •

  Experimental results using real datasets establish a significant performance gain of the energy dispatch under the deterministic, asymmetric, and stochastic environments. Particularly, the results show that the proposed MAMRL model saves up to $22.44\%$ of energy consumption cost over a baseline approach while achieving an average accuracy of around $95.8\%$ in a stochastic environment. Our approach also decreases the usage of non-renewable energy up to $11\%$ of total consumed energy.

The rest of the paper is organized as follows. Section II presents the system model of self-powered edge computing. The problem formulation is described in Section III. Section IV provides MAMRL framework for solving energy dispatch problem. Experimental results are analyzed in Section V. Finally, conclusions are drawn in Section VI.

Consider a set $\mathcal{B}=\left\{{0,1,2,\dots,B}\right\}$ of $B+1$ ($0$ for MBS) BSs that encompass $B$ SBSs overlaid over a single MBS. Each BS $i\in\mathcal{B}$ includes a set $\mathcal{K}_{i}=\left\{{1,2,\dots,K_{i}}\right\}$ of $K_{i}$ MEC application servers. We consider a finite time horizon $\mathcal{T}={1,2,\dots,T}$ with each time slot being indexed by $t$ and having a duration of 15 minutes [29]. The observational period of each time slot $t$ ends at the $15$-th minute and is capable of capturing the changes of network dynamics [11, 12, 30]. A set $\mathcal{J}_{i}$ of $J_{i}$ heterogeneous MEC application task requests from users will arrive to BS $i$ with an average task arrival rate $\lambda_{i}(t)$ (bits/s) at time $t$. The task arrival rate $\lambda_{i}(t)$ at BS $i\in\mathcal{B}$ follows a Poisson process at time slot $t$. BS $i$ integrates $K_{i}$ heterogeneous active MEC application servers that has $u_{k_{i}}(t)$ (bits/s) processing capacity. Thus, $J_{i}$ computational task requests will be accumulated into the service pool with an average traffic size $S_{i}(t)$ (bits) at time slot $t$. The average traffic arrival rate is defined as $\lambda_{i}(t)=\frac{1}{S_{i}(t)}$. Therefore, an M/M/K queuing model is suitable to model these $J_{i}$ user tasks using $K_{i}$ MEC servers at BS $i$ and time $t$ [31, 32]. The task size of this queuing model is exponentially distributed since the average traffic size $S_{i}(t)$ is already known. Hence, the service rate of the BS $i$ is determined by $\mu_{i}(t)=\frac{1}{\mathbb{E}[\sum_{k_{i}\in\mathcal{K}_{i}}u_{k_{i}}(t)]}$. At any given time $t$, we assume that all of the tasks in $\mathcal{J}_{i}$ are uniformly distributed at each BS $i$. Thus, for a given MEC server task association indicator $\Upsilon_{jk_{i}}(t)=1$ if task $j$ is assigned to server $k$ at BS $i$, and $0$ otherwise, the average MEC server utilization is defined as follows [11]:

The energy supply of the self-powered wireless network with MEC capabilities relates to the network’s own renewable (e.g., solar, wind, biofuels, etc.) sources as well as the main grid’s non-renewable (e.g., diesel generator, coal power, and so on) energy sources [8, 9]. In this energy generation model, we consider a set $\mathcal{R}=\left\{{\mathcal{R}_{0},\mathcal{R}_{1},\dots,\mathcal{R}_{B}}\right\}$ of renewable energy sources of the network, with each element $\mathcal{R}_{i}$ representing the set of renewable energy sources of BS $i\in\mathcal{B}$. Each renewable energy source $q\in\mathcal{R}_{i}$ at BS $i\in\mathcal{B}$ can generate an amount $\xi^{\textrm{ren}}_{iq}(t)$ of renewable energy at time $t$. Therefore, the total amount of renewable energy generation $\xi^{\textrm{ren}}_{i}(t)$ at BS $i\in\mathcal{B}$ will be $\xi^{\textrm{ren}}_{i}(t)=\sum_{q\in\mathcal{R}_{i}}\xi^{\textrm{ren}}_{iq}(t)$ for time slot $t$. Thus, the total renewable energy generation for the considered network at time $t$ is defined as $\xi^{\textrm{ren}}_{t}=\sum_{i\in\mathcal{B}}\xi^{\textrm{ren}}_{i}(t)$. The maximum limit of this renewable energy $\xi^{\textrm{ren}}_{t}$ is less than or equal to the maximum capacity $\xi^{\textrm{ren}_{\textrm{max}}}_{t}$ of renewable energy generation at time period $t$. Thus, we consider a maximum storage limit that is equal to the maximum capacity $\xi^{\textrm{ren}_{\textrm{max}}}_{t}$ of the renewable energy generation [40, 41, 42]. Further, the self-powered wireless network is able to get an additional non-renewable energy amount $\xi^{\textrm{non}}_{t}$ from the main grid at time $t$. The per unit renewable and non-renewable energy cost are defined by ${c^{\textrm{ren}}_{t}}$ and ${c^{\textrm{non}}_{t}}$, respectively. In general, the renewable energy cost only depends on the maintenance cost of the renewable energy sources [40, 41, 42]. Therefore, the per unit non-renewable energy cost is greater than the renewable energy cost ${c^{\textrm{non}}_{t}}>{c^{\textrm{ren}}_{t}}$. Additionally, the surplus amount of the energy $\xi^{\textrm{sto}}_{t}$ at time $t$ can be stored in energy storage medium for the future usages [41, 42] and the energy storage cost of per unit energy store is denoted by ${c^{\textrm{sto}}_{t}}$.

In the MAMRL setting, each BS $i\in\mathcal{B}$ acts as a local agent and the number of local agents is equal to $B+1$ BSs (i.e., $1$ MBS and $B$ SBSs). We define a set $\mathcal{S}=\left\{{\mathcal{S}_{0},\mathcal{S}_{1},\dots,\mathcal{S}_{B}}\right\}$ of state spaces and a set $\mathcal{A}=\left\{{\mathcal{A}_{0},\mathcal{A}_{1},\dots,\mathcal{A}_{B}}\right\}$ of actions for the $B+1$ agents. The state space of a local agent $i$ is defined by $\boldsymbol{s}_{ti}\colon(\xi_{i}^{\textrm{d}},\xi_{i}^{\textrm{ren}}(t),C^{\textrm{sto}}_{ti},C^{\textrm{non}}_{ti})\in\mathcal{S}_{i}$, where $\xi_{i}^{\textrm{d}},\xi_{i}^{\textrm{ren}}(t),C^{\textrm{sto}}_{ti}$, and $C^{\textrm{non}}_{ti}$ represent the amount of energy demand, renewable generation, storage cost, and non-renewable energy cost, respectively, at time $t$. We execute Algorithm 1 to generate the state space for every BSs $i\in\mathcal{B}$, individually. In Algorithm 1, lines $3$ to $6$ calculate the individual energy consumption of the MEC computation and network operation using (2) and (4), respectively. Overall, the energy demand of the BS $i$ is computed in line $7$ and the self-powered energy generation is estimated by line $8$ in Algorithm 1. Non-renewable and storage energy costs are calculated in lines $9$ and $10$ for time slot $t$. Finally, line $11$ creates state space tuple (i.e., $\boldsymbol{s}_{ti}$) for time $t$ in Algorithm 1.

Consider each local BS agent $i\in\mathcal{B}$ that can take two types of actions $\xi^{\textrm{sto}}_{i}(t)$ and $\xi^{\textrm{non}}_{i}(t)$ which are the amount of storage energy $\xi^{\textrm{sto}}_{i}(t)$, and the amount of non-renewable energy $\xi^{\textrm{non}}_{i}(t)$ at time $t$. We consider a discrete set of actions that consists of two actions $\boldsymbol{a}_{ti}\colon(\xi^{\textrm{sto}}_{i}(t),\xi^{\textrm{non}}_{i}(t))\in\mathcal{A}_{i}$ for each BS unit $i\in\mathcal{B}$. Since the state $\boldsymbol{s}_{ti}$ and action $\boldsymbol{a}_{ti}$ both contain a time varying information of the agent $i\in\mathcal{B}$, we consider the dynamics of Markovian and represent problem $\eqref{Opt_1_8}$ as a discounted reward maximization problem for each agent $i$ (i.e., each BS). Thus, the objective function of the discounted reward maximization problem of agent $i$ is defined as follows [28]:

where $\gamma\in(0,1)$ is a discount factor and each reward $\Upsilon_{t}\boldsymbol{a}_{ti},\boldsymbol{s}_{ti})$ is considered as,

In (18), $\frac{\xi^{\textrm{ren}}_{ti}}{\xi^{\textrm{d}}_{ti}}$ determines a ratio between renewable energy generation and energy demand (supply-demand ratio) of the BS agent $i\in\mathcal{B}$ at time $t$. When renewable energy generation-demand ratio $\frac{\xi^{\textrm{ren}}_{ti}}{\xi^{\textrm{d}}_{ti}}$ is larger than $1$ then the BS agent $i$ achieves a reward of $1$ because the amount of renewable energy exceeds the demand that can be stored in the storage unit.

Each action $\boldsymbol{a}_{ti}$ of BS agent $i\in\mathcal{B}$ determines a stochastic policy $\pi_{\theta_{i}}$. $\theta_{i}$ is a parameter of $\pi_{\theta_{i}}$ and the energy dispatch policy is defined by $\pi_{\theta_{i}}\colon\mathcal{S}_{i}\times\mathcal{A}_{i}\mapsto[0,1]$. Policy $\pi_{\theta_{i}}$ decides a state transition function $\Gamma\colon\mathcal{S}_{i}\times\mathcal{A}_{B}\mapsto\mathcal{S}_{i}$ for the next state $\boldsymbol{s}_{t^{\prime}i}\in\mathcal{S}_{i}$. Thus, the state transition function $\Gamma$ of BS agent $i\in\mathcal{B}$ is determined by a reward function $\eqref{eq:each_reward_fn}$, where $\Upsilon_{t}(\boldsymbol{a}_{ti},\boldsymbol{s}_{ti})\colon\mathcal{S}_{i}\times\mathcal{A}_{i}\mapsto\mathbb{R}$. Further, each BS agent $i\in\mathcal{B}$ chooses an action $\boldsymbol{a}_{ti}$ from a parametrized energy dispatch policy $\pi_{\theta_{i}}(\boldsymbol{a}_{ti}|\boldsymbol{s}_{ti};\theta_{i})$. Therefore, for a given state $\boldsymbol{s}_{ti}$, the state value function with a cumulative discounted reward will be:

where $\gamma^{t^{\prime}-t}$ is a discount factor and ensures the convergence of state value function $V^{\pi_{\theta_{i}}}(\boldsymbol{s}_{ti})$ over the infinity time horizon. Thus, for a given state $\boldsymbol{s}_{ti}$, the optimal policy $\pi_{\theta_{i}}^{*}(\boldsymbol{a}_{ti}|\boldsymbol{s}_{ti})$ for the next state $\boldsymbol{s}_{t^{\prime}i}$ can be determined by an optimal state value function while a Markovian property is imposed. Therefore, the optimal value function is given as follows:

Here, the optimal value function (20) learns a parameterized policy $\pi_{\theta_{i}}(\boldsymbol{a}_{ti}|\boldsymbol{s}_{ti};\theta_{i})$ by using an LSTM-based Q-networks for the parameters $\theta_{i}$. Thus, each BS agent $i\in\mathcal{B}$ determines its parametrized energy dispatch policy $\pi_{\theta_{i}}(\boldsymbol{a}_{ti}|\boldsymbol{s}_{ti};\theta_{i})=P(\boldsymbol{a}_{ti}|\boldsymbol{s}_{ti};\theta_{i})$, where $P(\xi^{\textrm{sto}}_{i}(t))=P(\boldsymbol{a}_{ti}=\xi^{\textrm{sto}}_{i}(t)|\boldsymbol{s}_{ti};\theta_{i})$ and $P(\xi^{\textrm{non}}_{i}(t))=1-P(\xi^{\textrm{sto}}_{i}(t))$ for the parameters $\theta_{i}$. The decision of each BS agent $i\in\mathcal{B}$ relies on $\theta_{i}$. In particular, energy dispatch policy $\pi_{\theta_{i}}$ is the probability of taking action $\boldsymbol{a}_{ti}$ for a given state $\boldsymbol{s}_{ti}$ with parameters $\theta_{i}$. In this setting, each local agent $i\in\mathcal{B}$ is comprised of an actor and a critic [27, 51]. The policy of energy dispatch is determined by choosing an action in (20) that can be seen as an actor of BS agent $i$. Meanwhile, the value function (19) is estimated by a critic of each local BS agent $i$. The critic can criticize actions that are made by the actor of each BS agent $i$. Therefore, each BS agent $i\in\mathcal{B}$ can determine a temporal difference (TD) error [51] based on the current energy dispatch policy of the actor and value estimation by the critic. The TD error is considered as an advantage function and the advantage function of agent $i$ is defined as follows: