Joint Service Caching, Communication and Computing Resource Allocation in Collaborative MEC Systems: A DRL-based Two-timescale Approach

For the allocation of communication and computing resources in MEC networks, most researcher focus on reducing the task processing delay and/or energy consumption through aspects such as task offloading (partially), computing resource allocation, power control, and channel selection.

Tan et al. [31] proposed a two-level alternation method to minimize total energy consumption in multi-user collaborative MEC networks. The method uses a heuristic algorithm in the upper level for collaboration and offloading decisions and a DRL-based algorithm in the lower level for power, subcarrier, and computing resource allocation. Kai et al. [32] proposed a pipeline-based offloading scheme to minimize the overall latency for all users in a could-edge-end collaborative MEC network. They utilized the classic successive convex approximation (SCA) approach to optimize the offloading strategy, computation resource allocation, delivery rates, and power allocation. In [33], the authors considered the increasing tasks with stringent QoS requirements, as well as the limited communication and computational resources in multi-user multi-edge MEC networks. They formulated a long-term throughput maximization problem from the perspective of service providers under latency constraints. However, the above literature does not take into account the impact of task-related service caching. They all take it for granted that all the needed services are cached at all ESs. This is not realistic for the ESs with limited cache resources.

Works on service caching configuration and computing offloading mainly aim to reduce the task processing delay and energy consumption by jointly scheduling the services models, offloaded tasks, and computing resources within a small-timescale slot.

With the aim of maximizing the users’ QoS, Chu et al. [34] proposed a Lagrangian-dual based distributed algorithm to optimize the service selection and computing resource allocation for a single MEC network. They further designed an online service caching update strategy based on the cost of services switching. Similarly, Zhou et al. [35] extended Chu’s optimization problem to a multi-ES collaborative MEC system and proposed a DDPG-based algorithm to minimize the long-term energy consumption. Zhou et al. [36] investigated a use case of reverse computation offloading and service caching in a three-tier MEC system, in which the cloud server can purchase computation and communication resources from the ESs to assist with computing the offloaded tasks. Generally, the caching resources are adjusted at a larger timescale than that the computation resources are done at. However, this feature has not been captured in the literature mentioned above. For this, Zhang et al. [37] proposed a hierarchical DDPG-based algorithm to deal with the dual timescale resource allocation for IoT systems to minimize the delay and energy consumption. Lyapunov optimization approach and dependent rounding technique are adopted to solve the problem of the long-term service caching and the short-term UAV trajectory, user association, and offloading decisions optimization, respectively [38]. However, the aforementioned literature has not taken into account the communication resource needed in the caching and offloading process.

For the joint multi-dimensional resources optimization in MEC networks, most literature mainly studies service caching configuration, task offloading, and computing and communication resources allocation at the same timescale, with the goal of reducing the task processing delay, energy consumption, and resource purchase costs.

Bi et al. [39] have investigated a single MEC scenario where a user has a series of dependent tasks. They transformed the joint optimization problem of the service caching placement, computation offloading, and system resource allocation ( CPU processing frequency and transmit power of user) into an equivalent pure 0-1 integer linear programming for solution. In order to minimize the computation and delay costs, Zhang et al. [15] investigated the joint allocation of service caching, transmission, and computing resources in a general scenario of multiple users with multiple tasks. They also considered the scenario where each user has a computation cost budget and formulated a quadratically constrained quadratic program problem solved with a semi-definite relaxation-based algorithm. However, these studies make the ideal assumption that caching and other resources can be scheduled simultaneously, which leads to frequent cache switching and high system communication overheads.

Several researchers, including Wen et al. [40] and Sun et al. [41], have addressed the issue by incorporating dual timescale into account in doing multi-dimensional resources allocation. In [40], a consensus alternating direction method of multipliers-based joint policy for the caching, offloading, and time slot allocation was proposed to minimize the weighted sum energy consumption subject to the storage and delay constraints. However, the temporal relationship between the caching and the offloading is replaced by simple statistics, meaning that the caching strategy is done based solely on the static statistics. To address this, Sun et al. [41] proposed a joint multi-agent RL and game approach to manage multi-dimensional resources in Fog radio access networks. RL is used to learn the long-term service caching strategies from historical information of user requests and channel states in small timescales. However, due to the inherent limitations of RL, the temporal relationship between the caching and the offloading is still not fully utilized. In addition, these works generally focus on the resource allocation of a single ES, while ignoring the collaboration of multiple ESs. Therefore, how to design a intelligent two-timescale multi-dimensional resources allocation scheme with collaborative ESs remains a challenging problem. To address this problem, we establish a collaborative MEC-enabled IIoT system, and propose a DRL-based two-timescale multi-dimensional resources scheduling scheme, DGL-DDPG.

As shown in Fig. 1, the collaborative MEC network comprises of one cloud server (CS), $M$ ESs, and $N$ TDs. Each TD can connect with one and only one ES via a wireless link at a time, and the ES can communicate with other ES and CS through wired links. The ES $m$ can act as an access point (AP) with a built-in MEC server, which has limited available spectrum, computing, and caching resources, represented by $\{W_{m},C_{m},F_{m}\}$. We assume that all ESs can be regarded as collaborative servers and are willing to share their computing and caching resources with each other. There are $F$ types of computing tasks and corresponding $F$ types of service models. The service set is denoted as $\mathcal{F}=\{1,2,\cdots,f,\dotsb,F\}$. We adopts a discrete time model which is divided into two-timescale, i.e., large-timescale for service caching placement and small-timescale for task offloading and resource allocation as shown in Fig.2. The large-timescale index of caching is denoted by $i\in I=\{1,2,...,I\}$, each of which includes $T$ short timescale slots. The small-timescale index within the $i$th large-timescale slot is represented by $k\in K=\{1,2,...,K\}$. At each short time slot $k$, TD $n$ randomly generates a computation task. We use a three-tuple $\{s_{n,i,k}^{type},s_{n,i,k}^{input},s_{n,i,k}^{comp}\}$ to denote the task of the $n$th TD, where $s_{n,i,k}^{type}\in\mathcal{F}$ represents the type of the task, $s_{n,i,k}^{input}$ represents the amount of input data (in bits), and $s_{n,i,k}^{comp}$ represents the required computational capacity (in CPU cycles). AS illustrated in Fig. 1, the tasks of TDs can be processed locally (e.g., red task) or offloaded to the ESs for computing. When the task is offloaded to the associated ES, if the corresponding service is cached, the task will be executed at the ES (e.g., blue and green tasks). Alternatively, the task can be further offloaded to the other collaborative ES for computing (e.g., purple and orange tasks). If none of the collaborative ES have cached the service, the task will be executed at the CS (e.g., red tasks). At each large-timescale slot $i$, the ESs update service caching placement (e.g., red and orange service models).

We define $\mathbf{c}(i)=\{c_{m}^{f}(i)\in\{0,1\}|m\in\mathcal{M},f\in\mathcal{F}\}$ to denote the service caching decision of all ES at large-timescale $i$ and the cache configurations of all ES maintain unchanged for a large-timescale slot. If the ES $m$ cached the $f$th service at the $i$th large-timescale, $c_{m}^{f}(i)=1$, otherwise, $c_{m}^{f}(i)=0$. The total cached services should not exceed the cache capacity of the ESs, that is

where $S^{cache}_{f}$ is the storage size of the $f$th service model (in bits). Services in $F$ are all available at the CS and can be cached to the ESs via backhaul. The cache switching cost is defined as the communication delay cost for downloading the services,

where $\mathbb{I}[x]$ is an indicator function that equals 1 if condition $x$ is true and 0 otherwise, $(c_{m}^{f}(i)-c_{m}^{f}(i-1))=1$ indicates that the service should be obtained from the CS, $R^{back}$ is the fixed backhaul transmission rate. Obviously, the cost is 0 at the small-timescale slots.

We define $\mathbf{d}(k)=\{d_{n,i}^{m}(k)\in\{0,1\}|n\in\mathcal{N},m\in\mathcal{M}\cup\{0,C\}\}$ to be the decision variable for TDs’ collaborative offloading at the $k$th small-timescale slot in the $i$th large-timescale slot, $d_{n,i}^{m}(k)=1$ indicates that the $n$th TD offloads its task to the $m$th ES at time slot $k$, similarly, $d_{n,i}^{0}(k)=1$ indicates local computing and $d_{n,i}^{C}(k)=1$ indicates cloud computing. Besides, we define $\mathbf{f}(k)=\{f_{n,i}^{m}(k)\in[0,1]|n\in\mathcal{N},m\in\mathcal{M}\}$ as the computing resource allocation decision, $f_{n,i}^{m}(k)F_{m}$ represents the computing frequency allocated to TD $n$ by ES $m$. It should be noted that the task can only be processed at the ES if and only if the relevant service model is cached at the ES and the necessary computing resources are allocated to it too. Hence, we have

According to different collaborative offloading decisions, the computing mode of TD $n$ can be divided into four cases:

Case 1: Local computing only ie., $d_{n,i}^{0}(k)=1$. The local computing delay (in seconds ) is

where $F_{n}^{l}$ is the computing capability of TD $n$. The corresponding energy consumption is

where $\varepsilon$ is the energy coefficient depending on the chip architecture of the TD $n$.

Case 2: Edge computing at associated ES $m$. In this case, the task is offloaded to ES $m$ and the service model of task is cached, ie., $d_{n,i}^{m}(k)=1$ and $c_{m}^{f}(i)=1$. Then the computing latency can be given as

Case 3: Edge computing at cooperative ES $\acute{m}$. In this case, there is no ability to process the offloaded task at associated ES and the task is forwarded to the cooperative ES $\acute{m}$ for processing, where the service model of task is cached and the resource is sufficient at ES $\acute{m}$, ie., $d_{n,i}^{\acute{m}}(k)=1\ \mathbf{and}\ c_{\acute{m}}^{f}(i)=1$. Then the computing latency can be express as

where $\acute{m}\in\mathcal{M}/m$ is the cooperative ES of the ES $m$, $f_{n,i}^{\acute{m}}(k)F_{\acute{m}}$ is the computing frequency allocated to TD $n$ by ES $\acute{m}$.

Case 4: Could computing. In this case, the service model of task is not cached at any ES, so it can only be further offloaded to the CS for computing, ie., $d_{n,i}^{C}(k)=1$. The computing latency can be written as

where $F_{c}$ is the computing capability of the CS.

Considering that the data size of the task processing results is negligible compared to the data size of the tasks themselves, in this work we only focus on the uplink transmission. We consider the scenario that each ES is pre-assigned spectrum bandwidth to serve its associated TDs, and we define $\mathbf{b}(k)=\{b_{n,i}^{m}(k)\in[0,1]|n\in\mathcal{N},m\in\mathcal{M}\}$ as the bandwidth allocation decision, $b_{n,i}^{m}(k)$ represents the bandwidth allocated to the $n$th TD by the $m$th ES for task offloading at the $k$th small-timescale slot within the $i$th large-timescale slot. The uplink transmission rate from TD $n$ to ES $m$ can be written as

where $p_{n}$ is the uplink transmit power of TD $n$ , $h_{n,m,i}(k)=g_{n,m,i}(k)dis_{n,m}^{-\alpha}$ represents the corresponding channel gain, with $g_{n,m}(k)$ and $dis_{n,m}^{-\alpha}$ are the small-scale Rayleigh fading and the large-timescale path fading, respectively, $dis_{n,m}$ represents the distance between TD $n$ and ES $m$ and $\alpha$ is the path loss exponent of the TD-ES links, $\sigma^{2}$ denotes the variance of the additive white Gaussian noise. We assume that the wired links transmit data at a fixed rate $R^{co}$ and $R^{back}$ (in bits per second, bps). The transmission delay and corresponding energy consumption from TD $n$ to its associated ES $m$ for task offloading can be defined as

The transmission delay from ES $m$ to the cooperative ES $\acute{m}$ for task offloading is written as

The transmission delay from ES $m$ to the CS for task offloading is written as

The delay for processing the task of the $n$th TD at the $k$th small-timescale slot within the $i$th large-timescale slot for any case can be expressed as

The energy consumption for processing the task of the $n$th TD at the $k$th small-timescale slot within the $i$th large-timescale slot for any case can be expressed as

As stated in Section I, delay and energy consumption are critical performance metrics for TDs in IIoT systems. Therefore, we define the QoS for TDs, in terms of the satisfaction with latency and energy consumption relative to a set threshold. This can be expressed as

where $\alpha,\beta\in[0,1]$, $\alpha+\beta=1$ are the weighted parameters specifying the TD’s preferences for delay and energy consumption in the QoS, respectively, $T_{n}^{th},E_{n}^{th}$ are the threshold of the latency and energy, which guarantee TD $n$ can work.

Our objective is to maximize the long-term QoS of all TDs, while reducing the long-term cache switching costs of all ESs in a collaborative MEC-enabled IIoT system. To achieve this goal, we propose a two-timescale multi-dimensional resource joint optimization framework by properly designing the service caching, collaborative offloading, computation, and bandwidth resource allocation. The problem of multi-dimensional resource allocation can be formulated as

where $\delta$ is a factor balancing the weight of the cache switching cost and the TDs’QoS, the constraint (17a) indicates that the task cannot be processed if there is no relevant service model cached at the ES or no computing resources have been allocated to it, Constraint (17b) implies that the bandwidth of ESs is allocated only to the TDs offloading their tasks, Constraint (17c) indicates that the task generated by TD $n$ can only be processed by a single server, Constraints (17d, 17e, 17f) denote the limitations of the caching, computing, and bandwidth capacities of each ES, respectively, Constraints (17g, 17h) indicate the maximum delay and energy limitations of TDs. Constraints (17i, 17j, 17k, 17m) make the problem a mixed-integer nonlinear programming (MINLP) problem, with variables on different timescales, which typically results in an NP-hard problem. The challenge lies in the difficulty of modeling the temporal relationship between the service caching and the task offloading. The scale and complexity of the problem are prohibitively high for utilizing traditional optimization methods to solve it. To address this issue, we propose a two-timescale solution DGL-DDPG in the next section.

As illustrated in Fig. 2, the DGL-DDPG algorithm consists of two layers, one is the large-timescale LSTM-DDPG based service caching, the other is the small-timescale Improved-GA based Off-RA. Specifically, service caching decisions are made at the beginning of each large-timescale slot $i$ by an LSTM-DDPG agent, while each large-timescale caching slot is composed of $K$ small-timescale Off-RA slots. The Improved-GA is used to make the corresponding Off-RA decisions in each small-timescale slot $k$. In particular, Fig. 2 shows the case that a large-timescale slot includes three small-timescale slots. First of all, the LSTM-DDPG agent observes the states of the environment over the first three small-timescale slots and feeds them into the LSTM network to learn, and then the LSTM network outputs an encoded state, that is, the current state of the large-timescale slot. It is then fed into DDPG network to get the large-timescale service caching decision. Finally, the caching decision is executed in the environment, the small-timescale Improved-GA will continue to output the Off-RA decisions and update the states of the environment under the new caching decision, and the LSTM-DDPG agent will continue to collect states and update the caching decision at the next large-timescale slot. The core of the two-layer optimization is to use the LSTM network to predict the time-varying cache requirements on a small timescale and, similar to the iterative algorithm, the two layers of DGL-DDPG are trained together until convergence. Correspondingly, as shown in Fig. 3, the input of the small-timescale Improved-GA algorithm is the service caching decision, and the output is the Off-RA decisions.

The variant Off-RA decisions affect the large-timescale service caching decisions, and the current decisions will also affect the future decisions. For example, the ESs tend to store those services that are frequently used but not cached. So, this is a typical dynamic sequential decision problem. Based on this, the dynamic environment and original optimization problem can be transformed into a Markov decision process (MDP) across different timescales and then we use deep reinforcement learning algorithm to solve it. The reason for choosing LSTM-DDPG as the DRL agent mainly because of its capabilities of model-free temporal decision making and state prediction. Specifically, DDPG is a state-of-the-art DRL algorithm based on stable and efficient actor-critic framework. When the LSTM network is embedded in DDPG as the temporal feature extraction network future caching demands can be predicted. In the following, we define the state, action and reward function of service caching and then the design of LSTM-DDPG agent was presented.

1) Service caching MDP. State Since the service caching strategy relies heavily on the small-timescale Off-RA decisions and collaboration between ESs. The state space consists of the current service caching configuration of each ESs, the cumulative cache gain of each service, and the popularity of each service on each ES and is written as $\mathbf{s}_{k}=\{\mathbf{c}_{k},\mathbf{g}_{k},\mathbf{p}_{k}\}$, with

where $g_{f}(k)$ describes the cumulative cache gain of service $f$, $g_{f}(k)=\sum_{\tau\in I}\sum_{k\in K}\sum_{n\in N}\mathbb{I}[s_{n,\tau,k}^{type}=f]Q_{n,\tau}(k)-Q_{n,\tau}^{l}(k)$, and $Q_{n,\tau}(k)-Q_{n,\tau}^{l}(k)$ represents the QoS gain caused by edge service caching compared to local computing, $Q_{n,\tau}^{l}(k)=\alpha\times\frac{T_{n,\tau}^{th}(k)-T_{n,\tau}^{l}(k)}{T_{n,\tau}^{th}(k)}+\beta\times\frac{E_{n,\tau}^{th}(k)-E_{n,\tau}^{l}(k)}{E_{n,\tau}^{th}(k)}$ denotes the Qos of TD $n$ when computing locally.

where $p_{m,f}(k)=\sum_{\tau\in I}\sum_{k\in K}\sum_{n\in N}\mathbb{I}[s_{n,\tau,k}^{type}=f]d_{n,\tau}^{m}(k)$ represents the popularity of service $f$ in ES $m$.

To predict the time-varying service caching demands , the agent will observe the state once in each small-timescale slot $k$, and then use the LSTM network to transform all the small time slot states$\{s_{1},\cdots,s_{k},\cdots,s_{T}\}$ into a large-timescale state $S_{i}$, which will be feed into DDPG network to output the large-timescale service caching decision.

Action: The action is the service caching policy of each ES, $\mathbf{a}_{i}=\{\mathbf{c}_{1}(i),\cdots,\mathbf{c}_{M}(i)\}_{M\times F},\mathbf{c}_{m}(i)=\{c_{m}^{1}(i),\cdots,c_{m}^{f}(i),\cdots,c_{m}^{F}(i)\}$.

Reward: For better caching strategy, we take into account the long-term QoS of TDs, the cache switching cost and the penalty for exceeding the cache space. The reward function can be described as follows.

2) The design of LSTM-DDPG. As illustrated in Fig. 2, LSTM-DDPG comprises three main components: the actor network, the critic network, and an episode experience pool. Both the actor and critic networks have a target network and a current network that have exactly the same structure. Each neural network is comprised of an input layer, output layer, two hidden layers, and a LSTM layers.

The actor network is responsible for generating the cache action according to the observed state.

where $\eta^{i}$ is a random Gaussian variable that decreases with the training steps to allow the agent to explore potential better caching strategy.

The primary function of the critic network is to assess the actions performed by the actor network. This is accomplished by learning a state-action value function $\mathcal{Q}(\mathbf{s},\mathbf{a}|\bm{\theta}^{c})=E\{\sum_{i=0}^{\infty}\gamma\times r_{i}\}$ that can be computed recursively using the Bellman equation,

where $\gamma$ represents the discount factor utilized to indicate the extent of future impact exerted by the current actions.

The experience replay technique involves sampling a mini-batch of transitions from the episode experience pool. Using these sampled transitions, the actor applies the policy gradient to update the current network parameters $\bm{\theta}^{a}$

and the critic adjusts parameter $\bm{\theta}^{c}$ by minimizing the loss function

and

where LSTM-DDPG utilizes the two network technology to improve the stabilization of the actor and critic, $\mathcal{Y}$ and $\acute{\mu}(\mathbf{s}_{\tau+1}|\acute{\bm{\theta}^{a}})$ are the output of the target network of actor and critic, respectively.

In addition, the parameters of the target networks $\acute{\bm{\theta}^{a}},\acute{\bm{\theta}^{c}}$ can be softly updated according to the real-time updated parameter $\bm{\theta}^{a}$ and $\bm{\theta}^{c}$, respectively,

where $\zeta$ is the soft update factor. The details of LSTM-DDPG are shown in Algorithm 1.

If the service caching policy $\mathbf{c}_{i}^{\star}$ is given, the problem can be reduced to a series of one-shot collaborative offloading and computing and bandwidth resource allocation (Off-RA) problem

However, this is an instantaneous one-step MINLP, which cannot be converted into MDP and thus cannot be solved by RL algorithms. To solve it, we introduce the GA, which is widely used to solve single-step resource optimization problems as formulated in our small timescale slots with remarkable performance. And it can be easily extended and embedded into DRL algorithms [25]. To adopt the GA, we design a hybrid coding and cross-mutation scheme for solving the problem with both integer and continuous variables and redesign the fitness function to ensure that the constraints are not violated and proposed the Improved GA. The details of the Improved-GA based Off-RA algorithm are shown in the following.

1) Chromosome and hybird coding. The chromosome expression of an individual represents a solution to the Off-RA problem, which consists of three parts. To satisfy the constraints (34h, 34i, 34j), the chromosome is encoded to a discrete integer part and two continuous real value parts. For example, we denote the chromosome of individual $j$ in the $l$th generation as $g_{j}^{l}(k)=\{0,1,\cdot,1;0,0.3,\cdot,0.2;0,0.2,\cdot,0.3\}_{N\times 3}$.

2) Fitness function. The fitness function represents the quality of an individual’s resource allocation strategy, with higher fitness indicating better performance. To construct the fitness function, we use the objective function of the total QoS of TDs. However, during the process of crossover and mutation, it is possible to generate individuals that violate constraints. To ensure the feasibility of the solution, we design the fitness function in a piecewise form. Specifically, if an individual violates the constraints, its fitness is defined as the minimum possible value. The fitness function is defined as follows

3) Selection, Crossover, and Mutation To preserve the excellent chromosomes in the current population, we introduce the fitness proportionate selection operation. Following the principle of survival of the fittest, certain chromosomes are selected as parent nodes for the next generation. The probability that the $j$th chromosome $g_{j}^{l}$ in the $l$th generation is selected as a parent node defined as

After selection, parent nodes generate offspring through crossover and mutation. However, the original uniform crossover operation is unsuitable for consecutively coded chromosomes. As an alternative to direct swapping, we take the mean of the parents. Furthermore, for our problem, this type of crossover operation may not fully capture the coupling between variables. To address this, we add a self-crossover operation after the original uniform crossover. This operation randomly swaps genes in each block of discrete and continuous genes separately. To obtain the final chromosome of the next generation, a mutation operation is required with a specific probability. Integer random mutation is applied to the discrete gene block, whereas real Gaussian mutation is applied to the continuous gene block. For instance, if the parents have the chromosome $\{0,1,2,0.2,0.4.0.3,0.1,0.2,0.6\}$, when their $(1,5,7)$th genes are altered, the chromosome of offspring is $\{2,1,1,0.2,0.45.0.3,0.05,0.2,0.6\}$. The detailed process is summarized in Algorithm 2, and the framework is shown in Fig. 2.

For the proposed LSTM-DDPG based service caching algorithm, the training process runs offline in the CS where there are sufficient computing resources. Therefore, we focus only on the complexity of the online service caching process. For the LSTM-DDPG agent, only the actor network works online. We take the number of parameters in the model to represent the complexity of the algorithm. The actor network has an input layer, an output layer, two hidden layers, and a LSTM layer. The complexity of the actor network can be written as $O(N_{i}^{a}L_{1}+4(L_{1}L_{2}+L_{2}^{2}+L_{2})+L_{2}N_{o}^{a})$, where $N_{i}^{a}$ is the number of neurons in the input layer, $N_{o}^{a}$ is the number of neurons in the output layer, $L_{1},L_{2}$ are the number of neurons in the hidden layers.

The complexity of Algorithm 2 lies in the design of coding, crossover and mutation options, but it is more directly limited by the generations and size of the population. Suppose the population size is P and the size of the chromosome is $G=3\times N$ and the number of generations is L, the computational complexity is about $O(2L+7LP+3LPN)$.

We consider a scenario consisting of one CS, $M$ ESs, and $N$ TDs within a 500m $\times$ 500m square area. The system has 10 types of tasks and services, each TD generates one task randomly per small-timescale slot according to a Zipf distribution with a parameter of 0.8. For each ES, the available bandwidth, computing, and caching resource are set to 20MHz, 20GHz, and 50GB, respectively. The TD’s transmission power and computing capacity are set to 20dBm and 1GHz, respectively. The fixed transmission rates $R^{back}$ and $R^{co}$ are set to be 200Mbps and 20Mbps, respectively. The additive white Gaussian noise $\sigma^{2}$ is set to 114dBm, with a path loss exponent of 2. Table 1 summarizes the rest simulation parameters. To show the priority of DGL-DDPG algorithm, seven similar algorithms are adopted as baselines.

• •

  DDPG-based service caching (DDPG), where the service caching is optimized based on the traditional DDPG algorithm, and the Off-RA strategy is determined by using the proposed Improved-GA.

• •

  Genetic algorithm (GA), where the service caching, offloading, bandwidth, and computing resource allocation decisions are generated by traditional GA. This approach employs discrete coding scheme.

• •

  No-cooperation, the No-cooperation scheme does not consider the collaborative computing and caching among ESs. If the associated ES has related services, the TD can offload, otherwise, it will be processed locally or offloaded to the CS.

• •

  Ave-resource, the communication and computing resources are allocated equally to TDs, while the service caching and task offloading are optimized based on our scheme.

• •

  Popular-cache, the Popular caching scheme caches service models at ESs based on statistical popularity information, while Off-RA decisions are made based on our proposed scheme.

• •

  Random-cache, the Random caching scheme caches service models randomly at ESs under the constraint of cache capacity, while Off-RA decisions are made based on our scheme.

• •

  Random-off, the Random offloading scheme generates offloading decisions randomly, while the other decisions are made based on our proposed scheme.

The proposed algorithm consists of both the large-timescale LSTM-DDPG based service caching and the small-timescale Improved-GA based Off-RA. The service caching strategy is learned by the LSTM-DDPG agent, and the agent gives a service caching decision at the beginning of each large-timescale slot, and then the Off-RA policy is obtained by the Improved-GA. First of all, we make sure that the small-timescale Improved-GA based Off-RA converges for any given service caching strategy. It is then embedded into the large timescale LSTM-DDPG based service caching to train the LSTM-DDPG agent. Therefore, when the large-timescale LSTM-DDPG converges, it means that the designed two-timescale DGL-DDPG algorithm converges. So, in the following, we first analyze the convergence of the small-timescale Improved-GA based Off-RA, and then analyze the convergence of the large-timescale LSTM-DDPG based service caching.

The convergence property of the Improved-GA and GA is shown in Fig. 4. It is seen that Improved-GA converges faster, and converges within about 10 steps, while GA converges within about 25 steps. This improvement is brought by the implementation of hybrid coding and self-crossover operations, which require less exploration steps but yield superior results. This also reduces the training complexity and time of the large-timescale LSTM-DDPG. Therefore, in the training process of LSTM-DDPG agent, we set the iteration number to be 20 steps.

Fig. 5 shows the convergence property of LSTM-DDPG and DDPG. It is observed that LSTM-DDPG performs better and more stable than DDPG. In the early stages of experience accumulation and exploration, both the two curves grow rapidly, indicating that DRL algorithms can efficiently acquire policies by interacting with their corresponding environments. As the number of training episodes increases, both the two algorithms converge, the difference is that LSTM-DDPG is faster and a bit more stable. This is because that the LSTM-based network structure can better leverage historical information to predict unknown network observation transitions. Effective predictions reduce the number of invalid explorations and therefore make it converge faster.

To fully verify the efficiency of DGL-DDPG in dealing with the two-timescale problem, we compare its long-term system performance, $U_{large}$, with that of DDPG and GA algorithm under different environment settings. The simulation is all in the setting of $I=20,K=5,M=2,N=20$ and the result takes 100 averaging operations.

Fig. 6 presents the long-term utility of DGL-DDPG compared to other benchmarks under different caching capacity settings. When the caching capacity is set to a low value (20GB-60GB), the long-term utility of DGL-DDPG is significantly better than those of GA and Popular-cache. This is because that GA only optimize single-step cache decisions based on the current offloading and resource allocation state, while Popular-cache always cache a few fixed, highly popular service models. As a result, these benchmarks fail to adapt to dynamic offloading environments and result in poor long-term performance. In the same time, when achieving the same performance, DRL-based schemes occupy less caching resources than instantaneous GA and statistics-based Popular-cache. Moreover, the performance gap between DGL-DDPG and DDPG further validates the effectiveness of adopting LSTM into DDPG for the long-term service caching phase.

The long-term utility for different number of the ES is shown in Fig. 7, where No-cooperation is used as a benchmark. Apparently, the utility will increase as the number of the ESs increases. The reason is that the whole system can schedule more resources. Both DGL-DDPG and DDPG outperform GA, demonstrating the superiority of DRL in adapting to dynamic environment and solving the long-term optimization problem, In addition, that DGL-DDPG performs better than DDPG indicating that the proposed DGL-DDPG is more suited for solving the resource allocation problem over two-timescale, since the LSTM network can enhance the temporal decision-making ability of DDPG.

In this part, we compare the instantaneous performance $U_{small}$ of the DGL-DDPG algorithm at small-timescale slot.

Fig. 8 shows the impacts of the number of TDs on the short-term utility. As the number of TDs increases, the short-term utility of all schemes continuously decreases. This is because the total resources of the ESs are limited. However, DGL-DDPG still performs better than Popular-cache and Ave-resource when the number of TD is large. The performance advantage over No-cooperation, Random-cache and Random-off is more obvious. The reason is that DGL-DDPG can achieve higher resource utilization through the form of collaboration and joint resource optimization.

The short-term utility with different bandwidth resource at the ESs is shown in Fig. 9. It is seen that the bandwidth resources increase, the short-term utility of all the simulated schemes improves, and DGL-DDPG still performs the best. This is because that the bandwidth resources limit the number of the offloaded tasks, and an increment in bandwidth resources enables the ESs to handle more tasks. Moreover, DGL-DDPG achieves a higher gain than No-cooperation and Popular-cache schemes. It reveals that even if tasks can be offloaded to the ESs, they cannot be handled if there are no the corresponding service model and computing resources, which means it is necessary to schedule the multi-dimensional resources together.

Fig. 10 shows the impacts of the computing capacity of the ESs on the short-term utility. It is seen that the short-term utility of all schemes improves as the computing capacity of the ESs increasing from 20GHz to 30 GHz. The reason is that more tasks can be processed at the ESs with the lower task processing delay and energy consumption when there is more computing resources. This reveals that DGL-DDPG realizes a reasonable service caching strategy along with the offloading decisions to make full use of the computing resource. Moreover, the gain compared to No-cooperation verifies the advantage of collaboration.