Efficient Task Offloading Algorithm for Digital Twin in Edge/Cloud Computing Environment

With the big success of AI, it is possible to achieve more sophisticated, intelligent functions on various mobile devices such as drones, self-driving cars, and robots. DT technology, as a bridge between physical entities and virtual systems, has been used in many different areas.

Industrial IoT is an important concept for Industry 4.0. To further improve the performance of the IoT devices in the industry, DT technologies have been adopted in many works. For example, Sun et al.[10] proposed an architecture based on DTs to assist the federated learning process of the Industrial IoT scenario. Song et al. [11] designed a federated learning framework for the DT driven Industrial IoT, where DRL and Lyapunov dynamic deficit queue are used to improve the communication efficiency. Therefore, data security and training accuracy of federal learning are guaranteed with the help of DT. In addition to Industrial IoT, IoV is an important IoT extension. DTs can provide a digital simulation model of the vehicles and optimize the allocation of resources. For instance, Wang et al. [12] and Sun et al. [13] used dynamic DT of aerial-assisted IoV to handle the unified resource scheduling and allocation problem. Alternating Direction Method of Multipliers (ADMMs) and Stackelberg game theory are employed to improve the overall efficiency of the model. Besides, Sun et al. [14] designed a dynamic DT and federated learning driven system for the air-ground networks: different incentive mechanisms are applied to improve the accuracy and energy efficiency of the system. Non-Orthogonal Multiple Access (NOMA) is an important part of next-generation wireless communications. Hence, Wang et al. [15] developed an energy-efficient DT system based on NOMA transmission. All the devices collaborate via federated learning to update a universal DT model, and the action model is updated to optimize the system. Numerical results also prove the viability of the algorithm.

With the help of edge computing techniques, MEC has become a novel approach for meeting the strict real-time requirements of DTs. Digital Twin Edge Networks (DITEN), which emerged as a novel DT architecture, have attracted significant attention. Sun et al. [16] first proposed a new vision of DITEN that considers user mobility and the variability of MEC environments, where Lyapunov optimization method and DRL algorithm are leveraged to handle the long-term migration cost. IoT is one of the most common application scenes of DT. For example, Lu et al. [17] integrated DTs with edge networks and proposed DITEN, while a blockchain-empowered federated learning scheme alleviates the data privacy protection and communication security problem in DITEN. A DT-assisted MEC model was proposed in [18] to minimize the end-to-end latency of industrial automation where multiple IoT devices and servers were considered. Additionally, UAVs can be applied as MEC nodes, providing low consumption and high flexibility for mobile-edge services. Li et al. [19] studied the intelligent task offloading problem in UAV-enabled MEC assisted by DT: Double Deep Q-network algorithm and closed-form expression method were deployed to optimize the system’s energy consumption. A social-aware vehicular edge caching mechanism was designed in [20], and a new concept of vehicular cache cloud was developed. The system utility is optimized by a deep deterministic policy gradient learning approach intelligently. In the Industrial IoT scenario, DITEN and federated learning were applied to construct DTs for IoT devices in the digital space. The energy and time cost of the communication process is minimized with the deep neural network model [21].

From the above work, it is undeniable that MEC can improve the performance of DTs. Since the cloud servers are always far from the physical entities, the transmission in mobile networks will lead to higher latency for MCC. However, MCC still has its advantages. For example, the cloud server has higher computational power, and the highly integrated framework makes it more energy efficient. Therefore, the cloud server can provide a lower time and energy cost for entities with huge workloads during execution. Hence, more work should be done on integrating DTs with MEC and MCC heterogeneous environments. Moreover, current DT system models mainly focus on improving performance by reducing the total consumption of each DT in the system. However, the DT with multiple data resources has yet to be discussed. No consideration has been given to the synchronization problem between various devices when formula the system cost.

In addition to the DT system models, there is still plenty of work that mainly focuses on the offloading decision-making problem of MEC and MCC. Much of the current work is based on traditional optimization methods such as Lyapunov optimization [22, 23, 24, 25, 26, 27], Graph Theory [28, 29], Stalberg Game Theory [30], and Successive Convex Approximation [31]. These schemes can take optimal or near-optimal offloading decisions for various offloading environments. However, the problem complexity increased exponentially when the number of devices increased. Therefore, the time consumption during generating offloading decisions using traditional optimization methods will be unaffordable when dealing with complicated scenes. To better solve this NP-hard problem within a relatively short time, machine learning-based intelligent algorithms have been the main research direction in recent years.

Most machine learning algorithms require large amounts of data to train the networks. However, the optimal offloading decision data is challenging to obtain when facing dynamic heterogeneous MEC/MCC environments. So, algorithms that can be trained without data, such as RL and DRL based algorithms, have attracted much attention from many researchers. Huang et al. [32] first proposed a Deep Q-Network based offloading algorithm to minimize the system cost in the MEC environment. In order to jointly optimize the task assignment and radio resource allocation in the MEC/MCC scenes, Dab et al. [33] proposed the QL-JTAR algorithm based on Q-learning. Simulations conducted in NS3 have proved the performance of the approach. Double Deep Q-Network algorithm and a Q-function decomposition technique were combined in work [34]. The problem of stochastic computation offloading is formulated as a Markov decision process, and two novel offloading algorithms for the MEC system, Deep-SARL and DARLING, are designed. When considering the application scenario of vehicular edge computing, which is a typical application of MEC, Zhan et al. [35] presented a new DRL architecture based on proximal policy optimization method called DRLOSM. A convolutional neural network is also used to approximate policy and value functions and extract representative features. Qiu et al. [36] studied a blockchain-based collective Q-learning approach in a networking integrated cloud–edge–end resource allocation in IoT.

RL and DRL-driven algorithms are promising solutions to problems that lack enough training data. However, another issue emerges when considering the dynamic environments in MEC/MCC: the model needs to be retrained when the offloading environment varies. Meta-learning is an excellent solution to improve the portability of different methods. Huang et al. [37] combined Meta-learning with deep learning in the MEC network and proposed an algorithm called MELO. Through training the DNNs with historical MEC task scenarios, the model can achieve 99% accuracy via one-step fine-tuning when facing a new MEC task scenario. Wang et al.[38] first applied Meta-Reinforcement Learning in a multi-access edge computing network. They synergized the first-order MRL algorithm with a sequence-to-sequence neural network and proposed the MRLCO algorithm. The typical meta-learning algorithm MAML and Reptile are also applied in MRL and combined with DRL in [39, 40]. Simulations have proved that the training time can be remarkably reduced, and the model’s portability can be improved significantly.

Apart from the meta-learning approach, distributed deep learning is another encouraging way to generate offloading decisions despite few training data. DDL algorithms use multiple parallel DNNs to generate offloading decisions. Each DNN receives offloading environment as input and output offloading decisions for each task through the forward propagation process. In contrast to traditional DNNs, which require large amounts of data for training, distributed deep learning can make the DNNs train each other without data. Huang et al. [41] first proposed the DDLO algorithm based on DDL for the MEC networks. As a result, near-optimal offloading decisions can be generated in less than one second. More work about the online computation offloading issue in MEC networks was studied in [42]. Thorough experiments and evaluation works proved the feasibility of the designed DROO algorithm. Moreover, Wu et al. [34] extended the algorithm and proposed the distributed deep learning based offloading algorithm for the heterogeneous MEC/MCC system. Numerical results indicate that the average error can be controlled under 6% compared to optimal decisions.

DDL has shown significant potential when facing MEC and/or MCC scenarios. Nevertheless, few works have been done for applying DDL in the DT system model. In addition, most approaches ignored the relationship between different devices during the deployment. The hidden information should be extracted and utilized to simplify the network structure. Here, we proposed a novel DT system model that considers the synchronization issues for the DT with various devices for data collection. The distributed deep learning algorithm is combined with a fully connected network layer to further extract the features from each group. The offloading decision-making model takes features as input instead of all the offloading environment information.

Figure 1 elaborates the framework of our DT system model. The framework can be divided into two main parts: the physical environment and the procedure of DT maintenance. In the physical environment, we assume that there are multiple edge servers and various data collection devices in the field, where all the edge servers and devices are deployed. Each device can connect to the edge servers via wireless transmission, and one cloud server can also be utilized through the mobile network.

Besides, we assume that several DTs are maintained by the devices and servers in the physical environment. The maintenance of each DT requires data from various devices, i.e., the vehicles in the same street, Metaverse users in the same virtual space, and cameras for intelligent robots. Each DT is placed and maintained in one of the servers, where all the data is synchronized, and computational tasks are executed, such as simulation, prediction, and analysis.

DTs can achieve better performance with the advantages of MEC and MCC, but the placement of DTs in the dynamic scenario still matters. On the one hand, the utilization of computation power and bandwidth should be maximized in order to further reduce the delay; on the other hand, the average latency and energy consumption should be balanced accordingly. Therefore, the resource allocation problem during the DTs maintenance will be determined according to the environment.

Our system model contains one cloud server, S edge servers, denoted as $\mathcal{S}=\{1,2,\cdots,S,S+1\}$, and various devices used for data collection, which can be denoted as $\mathcal{N}=\{1,2,\cdots,N\}$. Each device belongs to one of the DT and all the DTs can be represented as $\mathcal{M}=\{1,2,\cdots,M\}$. In addition, the maintenance of the DTs always requires great computational capacity, which is much higher than the capacity of the calculation unit in each device. The workload, including the newly collected information such as location and movement, is transmitted to the server via wired or wireless transmission techniques. The DTs are maintained in different servers instead of computing all the tasks locally. The whole process can be called offloading. We assume that the workload for the $n^{th}$ devices is $w_{n}$. The DTs can be maintained on either a cloud server or edge servers, and workloads are offloaded to the servers according to the offloading decision.

Our framework can be divided into two parts, namely, the feature extract module based on a fully connected network and the decision module based on distributed deep learning approach. For any given workloads information $\mathcal{W}$, location $\mathcal{L}$ and devices information $\mathcal{G}$, we aim to output the respect near-optimal offloading decision $\mathcal{X}$.

Before the decision-making module, for each DT we first input $\mathcal{W}$ and $\mathcal{W}$ to the feature extractor module according to $\mathcal{G}$. Features for each DT are extracted, which then be regarded as inputs of the decision module to reduce the dimension of the problem. Although $\mathcal{W}$, $\mathcal{L}$ and $\mathcal{G}$ can be leveraged directly in the DDL process, the relationship between the devices of the same DT is ignored, and the number of trainable parameters is increased significantly, which will cause unnecessary computational cost. Besides, it is hard to use machine learning approaches to extract features. The training process always needs adequate training data, which is hard to get, and the features have to be pre-decided by experience. But in our scheme, the feature extraction module can be trained together with the decision module without data by utilizing the advantage of distributed deep learning, which will be elaborated in the next section.

The decision module consists of $\mathcal{K}$ parallel DNNs with the same structure but different initial weight parameters. Each DNN gets the data given by the feature extraction module as input and output offloading decisions of the application scenario. In the hidden layers, we use Relu as the activation function, and in the output layer, we use the Sigmoid activation function to map the output into (0,1). Since the offloading decision is represented by a sequence of binary integers, we have to map the values given by the network into the desired format. For the $k^{th}$ DNN, we use $\hat{\mathcal{V}}^{k}$ to represent the output values and $\mathcal{V}^{k}$ as the corresponding binary output. For any element $\hat{V}_{i}$ in $\hat{\mathcal{V}}^{k}$, the binary offloading decision $V_{i}$ can be determined by:

As a result, The offloading decision $\mathcal{V}^{k}$ of each DT is represented by a sequence in a binary format. According to the definition of the offloading decision in problem ($\mathcal{P}$), each sequence is then converted from the binary format into one-hot format and composed into the required offloading decision $\mathcal{X}^{k}$.

During the decision-making procedure, the input data is replicated and fed into each DNN. Since the multiple DNNs have different weights, we can obtain $k$ possible offloading decisions $\{\mathcal{X}^{1},\mathcal{X}^{2},\cdots,\mathcal{X}^{K}\}$. The performance in terms of time delay and energy consumption of each decision can be calculated by using the cost function $Q(\mathcal{W},\mathcal{L},\mathcal{G},\mathcal{X})$. Then the best decision $\tilde{\mathcal{X}}$ with the lowest system cost can then be determined, which can be either used as the final decision or regarded as training data.

Taking the complexity and variability of the MEC and MCC hybrid environments into account, we notice the difficulty of collecting abundant training data of the aiming scenario for the traditional deep learning based approaches. Besides, generating optimal decisions via conventional methods is also time-consuming when the server numbers and the DT numbers increase. In our approach, multiple DNNs are used to train the DDL model regardless of training data. The main procedure is depicted as follows.

We first randomly initialize the weight parameter of each DNN differently. To be specific, each layer obeys a normal distribution but the same layer of different DNNs have different weight parameter. Then, we randomly generate a set of input data consisting of $\mathcal{W}$, $\mathcal{L}$, and $\mathcal{G}$ to simulate one possible situation. After that, we input all the data to each DNN and these $\mathcal{K}$ parallel DNNs will output $\mathcal{K}$ different decisions. Based on the generated situation and decisions, we can calculate the current best decision $\tilde{\mathcal{X}}$. Although the DNNs have not been trained well for the same input, the best decision still represents better performance, which can be regarded as labeled data. Improvements will be achieved theoretically if we use the selected decision to train other DNNs. Thus we regard $\{\mathcal{W},\mathcal{L},\mathcal{G},\tilde{\mathcal{X}}\}$ as one training data and stored in the training data set. The process is iterated until the upper limit of data size is reached.

To train the model, we randomly choose a batch of training data from the data set for each DNN. The batches are different from each other since the weights of each neural network will soon become the same as the iteration progresses if all the DNNs are trained by the same data, which will be a hindrance to the exploration of possible decisions. Then cross-entropy loss function is applied for each DNN, which is minimized based on gradient descent. We denote the training data set of the $k^{th}$ DNN as $\mathcal{R}^{k}=\{(\mathcal{W}_{i}^{k},\mathcal{L}_{i}^{k},\mathcal{G}_{i}^{k},\tilde{\mathcal{X}}_{i}^{k})\big{|}1\leq i\leq U\}$, where U is the size of each data batch. Then the loss of the $k^{th}$ DNN can be calculated as:

where $\pi_{k}$ to represent the parameter of the $k^{th}$ DNN and $f_{\pi_{k}}$ is the corresponding output.

After updating the parameter $\{\pi_{1},\pi_{2},\dots,\pi_{K}\}$ using Adam optimizer [43] according to the calculated loss value, the DNNs should have better performance and higher decision-making level. Therefore, we repeat the data generation process described in the previous part and replace part of the data set with these new samples that have higher accuracy.

The parallel DNNs keep learning from the best decisions within each other, and the data set is constantly updated to ensure the training performance. Iterations are conducted, and the neural networks are continuously trained to approximate the global optimal offloading decisions. The pseudo-code of the proposed DRL algorithm is provided in Algorithm 1.

For traditional deep learning based approaches, the model’s accuracy can be tested using the labeled data. However, the distributed deep learning based model is trained without labeled data. Besides, the optimal decision is hard to obtain using both traditional optimization methods and intelligent methods. To evaluate the performance and verify the feasibility of the proposed algorithm, we need to adopt a plausible verification method based on the characteristics of DDL. The testing process of the model can be separated into two parts, the convergence performance during training and the accuracy performance after the model is trained well.

Firstly, we need to ensure that the whole model converges after iterations. More specifically, after finite training iterations, the generated decisions of different inputs remain the same before and after extra iterations. Although the loss value can show the convergence performance of each DNN, the convergence of the model mainly focus on the best decision chosen from all the DNNs. So, we use $U_{0}$ randomly generated input data as the test data, which can be denoted as $\mathcal{R}^{\mathcal{T}}=\{(\mathcal{W}_{i},\mathcal{L}_{i},\mathcal{G}_{i})\big{|}1\leq i\leq U_{0}\}$. We input $\mathcal{R}^{\mathcal{T}}$ into the model before and after each training epoch and generate the corresponding decisions. The system cost of each decision is then calculated, including the cost for the old decisions before training and the new decisions after training, which can be written as $C^{old}=\{C_{i}^{old}|1\leq i\leq U_{0}\}$ and $C^{new}=\{C_{i}^{new}|1\leq i\leq U_{0}\}$. The convergence rate can be derived as follows:

where $\mathcal{C}\in(0,1]$ and the $\mathcal{C}$ will be close to one of the model convergent better.

After the model is proven to converge well, we still need to make sure that the model can make near-optimal decisions rather than converge in any local optimal solutions. So several baselines are applied to make comparison evaluations due to the lack of optimal decisions. we use the randomly generated dataset as the test data, too. For the same test data set, we will measure the average cost under given decision-making schemes and our approach, which can show the improvement and drawbacks of our approach. Besides, we also change the weight coefficient $\alpha$ into different value to ensure the proposed algorithm have reliable accuracy in various trade-off situations.

In our simulation, we consider a heterogeneous edge/cloud environment consisting of one cloud server and three edge servers. Considering the system congestion and the number of threads, we set the equivalent clock frequency of the cloud server $f_{c}=3.5$ GHz. Similarly, the clock frequency of the edge servers is randomly distributed between $1.8\sim 3.0$ GHz. In addition, since the cloud server has better energy efficiency than the edge servers, we set the calculation energy consumption parameter $\theta_{c}=0.1$ mJ and $\theta_{e}^{s}=0.125$ mJ, respectively.

Then we assume that the DT edge/cloud system contains up to 120 data collection devices, which belong to 15 different DTs at most. The locations and device ownership are randomly simulated in different situations, and the workload of each device is derived from the range $[10,40]$ MB. The transmission cost for one unit of data is set as $e_{t}^{c}=0.15$ mJ and $e_{t}^{e}=0.125$ mJ. Then we assume that the bandwidth between each device and base station is 1000 Mbps. The summary of our evaluation parameters is demonstrated in Table II.

For each DNN in our model, we considered a network with two hidden layers as the decision-making module and a fully connected network with one hidden layer to extract the feature of each group. We implement the proposed algorithm in Python 3.8.0 with TensorFlow 2.9.1. All the simulations are performed based on an Intel Core i7-12700H CPU and 16 GB memory.

In this part, we measure the convergence performance under different learning rates, DNN numbers, and database sizes. We also generate a test set consisting of 1024 different scenarios as the test data set. After each iteration, we input the test set into the model and calculate the average cost after each iteration, which can represent the accuracy change of our approach.

By evaluating the convergence and accuracy performance under different hyperparameter settings, we have proven that the proposed algorithm can be trained without labeled data and significantly reduce system consumption. To have a better insight of the effectiveness of our approach, we choose several traditional schemes as baselines. We then change the weight coefficient from 0 to 1 and test the average system cost of different schemes. The following schemes are tested for comparison analysis:

• •

  Random Offloading(RO): In this method, all the decisions in the decision space are chosen randomly with the same probability.

• •

  Cloud Only(CO): In this method, all the DTs are maintained by the cloud server.

• •

  Average Distribution(AD): In this method, all the DT is divided according to the size of the workload. Each server shares an average amount of workload.

• •

  Our algorithm: In this method, we applied the proposed algorithm to generate offloading decisions.

The comparison results of different offloading schemes are depicted in Figure 9. The weight coefficient $\alpha$ of the cost function (17) represents the trade-off between system delay and total energy consumption. It can be seen that our model can converge and decrease the system cost significantly under different weight coefficients. Compared with several baselines, our approach can achieve better performance, especially when tackling the system delay. Besides, our algorithm also has better capabilities to handle wider application scenarios. We do not need to train the model again when the scenario changes, e.g., when each device’s workload and location information are different. The decision can be obtained through a forward propagation process, which can be achieved within 0.01s.