Efficient Large-Scale Multiple Migration Planning and Scheduling in
SDN-enabled Edge Computing

In this section, we introduce the framework of migration scheduling in edge computing. Compared with periodically arrived multiple live migrations in cloud data centers, the arrival of live container migration induced by user mobility is stochastic. Therefore, we design the scheduling framework for the planning and scheduling of live container migration in edge computing with stochastic environments. As shown in Fig. 2, when a migration request arrives, it enters the WAITING state if it is feasible for scheduling, which means the container is not in migration. Otherwise, it will enter into the FAILED waiting migration list of the corresponding container. The migration planning event is triggered periodically within a short interval (such as every 1 second). It will generate the migration scheduling plan according to both waiting and running migrations. Based on the migration plan, the SDN-enabled on-line scheduler starts the migration with the allocated bandwidth and routing. Then, the live container migration will start the pre-migration phase to extract the container procedure tree. This will trace the dirty memory in the userspace of the source server and create an empty container instance in the destination for state synchronization [9]. In MEM_COPY, the dirty memory is transferred iteratively to the synchronizing instance in the destination. In the post-migration phase, the network communication of the migrated service will be redirected to the new instance in the destination. Then, the migrated container will recover at the destination. It will also trigger the start of subsequent resource-dependent migrations in the plan and change the feasibility flag of the first migration request of the same container in the FAILED migration waiting list.

The performance of single live migration $T_{mig}$ can be categorized into three parts: pre-migration computing, memory-copy networking, and post-migration computing overheads, i.e., ${T_{mig}}={T_{pre}}+{T_{mem}}+{T_{post}}$. Due to the smaller footprint and fast start up of containers compared to VMs, the pre-migration and post-migration is much shorter [31]. Based on the iterative pre-copy container migration implemented in CRIU [9], the migration performance in terms of memory-copy can be represented as [30]:

where the ratio $\sigma=\rho\cdot{R\mathord{\left/{\vphantom{RL}}\right.\kern-1.2pt}L}$, $\rho$ is the data compression rate of dirty memory, $Mem$ is amount of kernel memory the container uses, $L$ is allocated available bandwidth, $R$ is dirty page rate which is the memory difference in pagemap per second compared to the previous copy iteration, $i$ is the total migration round.

We consider three conditions to enter the stop-and-copy phase: (1) reach the threshold of memory copy iteration; (2) the transmission time of remained memory difference is less than the downtime threshold; and (3) the allocated bandwidth is less than the dirty page rate. The overhead of disk transmission for the container data and image can be ignored when shared network storage is available. The total iteration rounds of memory copy can be represented as:

where $\Theta$ denotes the maximum allowed number of iteration rounds. The $\Theta=0$ when the dirty page rate is larger than the allocated bandwidth at the start of migration. $V_{thd}=T_{dthd}\cdot L$ is the remaining dirty pages need to be transferred in the stop-and-copy phase, and $T_{dthd}$ is the configured downtime threshold.

We first explain the network sharing competition overheads in multiple migration scheduling. Two migrations may share the same source, destination, or part of network routings. Therefore, performing multiple live migrations in arbitrary order can lead to service degradation and unacceptable migration performance [21, 22]. A smaller bandwidth during the live migration means a longer migration time and more dirty pages need to transfer in order to limit the state difference between two instances for the last stop-and-copy phases which contributes as the downtime. Thus, the sum of the individual migration time of several live migrations is less than the total live migration time [30]. For example, based on the live migration model, Fig. 3(a) and 3(b) show the situation when several identical migrations sharing the same network path. The container’s initial memory size is 1 GB with a 20 MB/s dirty page rate. The downtime and iteration threshold is configured at 0.5 seconds and 30 times, respectively. In this example, for the sake of a clear comparison between the sum of individual migration time and the total migration time, we start all migrations at the same time. In this case, the average migration time as shown also equals the total migration time.

The average execution time of live migrations scheduled sequentially with 10 Gbps and 1 Gbps is 0.8482 and 7.5241 seconds. The average downtime is 0.0082 and 0.1048 seconds. The iteration rounds are 3 and 4, respectively. However, the average migration time or total multiple migration time of 5 live migrations sharing 10 Gbps and 1 Gbps is 3.604 and 88.43 seconds. The average downtime is 0.2048 and 0.3689 seconds with 3 and 12 iterations, respectively. As the number of migrations increases (Fig. 3(a)), the allocated bandwidth decreases linearly. However, to achieve the required migration downtime, the average migration time will increase exponentially. At 7 and 80 migrations sharing of 1 Gbps and 10 Gbps respectively, the iteration rounds reach the threshold as 30 (Fig. 3(b)). Then, with more bandwidth-sharing migrations, the dirty page rate is larger than the allocated migration bandwidth. The downtime exceeds the 0.5 seconds threshold and increases significantly from 0.78 to 64.0 seconds and from 1.85 to 640 seconds. For the time-critical live migrations, a longer migration time will increase the possibility of migration deadline violation and QoS degradation. Therefore, it is optimal to sequentially schedule the resource-dependent migrations while concurrently schedule the independent ones. If there is a set of independent migrations and no other resource-dependent migrations are running, we can start all migration in such a concurrent scheduling group. The objective of migration scheduling is to maximize the number of migrations that can be scheduled concurrently.

Figure 4 shows an illustrative example with twelve live migrations requests on the edge network topology of 5 total EDCs. Let $m^{sd}_{i}$ denote the migration request that migrating container $i$ from EDC $s$ to EDC $d$. For the sake of a concise example, we limit the network interfaces used by the migration traffic. In other words, migration traffics share the same interfaces when the source or the destination is the same. It can be easily extended to the set of network interfaces in source $\{s\}$ and destination servers $\{d\}$ and the corresponding network paths $\{p\}$. The network routing policy considers the shortest network path with the minimal number of migration flows. For example, there are two network routes between EDC1 and EDC3. As there is one migration from EDC2 to EDC3, it chooses network path $\{EDC1,EDC4,EDC3\}$ in this case.

Resource-independent migrations from one concurrent scheduling group can be scheduled at the same time. The planning algorithm needs to generate a scheduling plan consists of several concurrent migration groups that each group size is as large as possible. A larger concurrent scheduling group indicates that there are more migrations could be performed at the same time. As a result, the better performance of multiple migrations in total migration time and the QoS of migrating service can be guaranteed. Note that two migrations from different migration groups are not necessarily resource-dependent. As shown in Fig. 5, based on the network topology provided by the SDN controller, we create an undirected graph of resource dependency among migrations based on the source, destination, and network routing of migration requests. Each node $v^{sd}_{p}$ represents a list of migrations sharing the same source $s$, destination $d$, and network path $p$. In other words, migrations in one node list form a complete graph as all migrations are resource-dependent to all others in the list. For example, the migration list of node $v^{13}$ is $\{m_{1}^{13},m_{12}^{13}\}$. It significantly limits the problem complexity as the number of migration requests increases. The edge of the dependency graph indicates resource competition (network interfaces at source or destination, or bandwidth sharing along network routes) between migrations. A concurrent group equals to an independent set of the resource dependency graph. A maximal concurrent scheduling group is a set of resource-independent migrations that is not a subnet of any other concurrent group. In other words, there is no other migration outside the concurrent group can be added to it so that all migrations can be performed at the same time. Therefore, it equals a maximal independent set (MIS). The largest size MIS is a maximum independent set. As shown in Fig. 5, there are several combinations of migrations for a maximal concurrent scheduling group. In the first iteration, one of the maximum group is ${\left\{{m_{1}^{13},m_{7}^{32},m_{5}^{25},m_{9}^{41}}\right\}}$ and one of the maximal group is $\left\{{m_{3}^{23},m_{11}^{45},m_{6}^{31}}\right\}$. Thus, the maximum group is a better choice. After selecting the migrations from the nodes of the maximal independent set, we delete these migrations and update the dependency graph. One node is deleted from the graph when there is no migrations left in its migration list. For example, after the first iteration, we only delete nodes $v^{25},v^{41},v^{32}$, because there is still one migration $m_{12}^{13}$ left in node $v^{13}$ list. Thus, it is essential to select migrations carefully to achieve the maximum size of the concurrent groups. At the end, the on-line scheduler schedules all migration in the first group. Then, when there is one migration finishes, the scheduler starts all migrations blocked by the finished migration following the order of migration groups.

Before discussing how to get the Maximum Independent Set, the largest Maximal Independent Set (MIS), of the resource dependency graph, we first review some basic graph concepts [33], such as clique $C$ and independent set $I$. A clique is a subset of vertices of an undirected graph G such that every two distinct vertices in the subset are adjacent. The maximal clique is a clique that cannot be extended by including one more adjacent vertex. On the other hand, an independent set of a graph G is the opposite of a clique that no two nodes in the set are adjacent. The maximum clique or independent set is the maximal clique or independent set with the largest size. $\alpha\left(G\right)$ denotes the size of the largest MIS of graph $G$. Therefore, an independent set of the resource dependency graph equals a concurrent migration group. The migrations from the nodes in an independent set $I$ can be scheduled concurrently. Meanwhile, migrations from the nodes in a clique are resource-dependent which need to be scheduled sequentially.

There are few multiple migration planning and scheduling algorithms for live VM migration in cloud data centers [21, 22]. However, the processing time of the scheduling sequence of multiple live migrations based on the algorithms in cloud data centers is not suitable for the real-time requirement of mobility-induced migrations in edge computing. For example, the processing time of FPTAS [22] and CQNCR [21] for migration planning is about 5 and 10 seconds for 100 migrations. The processing time increases to 44.56 and 968.46 seconds for FPTAS and CQNCR to generate the scheduling plan of 500 migrations. For traditional dynamic resource management, the algorithm triggered every 10 minutes or 30 minutes. This leaves enough time budget for algorithms to generate the optimal scheduling sequence. However, in the edge computing environment for mobility-induce live migrations, the live migration requests arrive at any time stochastically. Most of the migration requests are also time-critical. Thus, the processing time of the planning and scheduling algorithm for mobility-induced migrations should be adapted to suit the real-time scenario.

The planning and scheduling algorithm is triggered periodically after every time interval ${\Delta_{sch}}$. We let $M_{arriv}^{t}$ denote the set of arrival migration requests at planning time $t$. $M_{wait}^{t}$ is the set of migration requests waiting for planning at time $t$. $M_{fail}^{t}$ is the set of infeasible migrations, such as its requested container is in migration. $M_{plan}^{t}$ is the set of migrations that have been planned but not finished at time $t$, and $M_{finish}^{t}$ is the set of finished migrations at time $t$.

The input of migration requests at every migration planning time $t$ is $M_{input}^{t}=M_{plan}^{t}\cup M_{wait}^{t}$. For each live container migration $m_{j}$, we have source and destination edge data center and allocated network routing, $(s_{j},d_{j},p_{j})$, available bandwidth $l_{j}$, arrival time $a_{j}$, estimated migration time $T_{j}$, relative deadline $D_{j}$, start time $b_{j}$, and finish time $f_{j}$. Therefore, the response time can be represented as ${r_{j}}={f_{j}}-a{}_{j}$. The slack time of migration scheduling, the remaining scheduling window that one migration will not miss its deadline, is ${\tau_{j}}={a_{j}}+{D_{j}}-{T_{j}}-t$. The objective of live container migration planning and scheduling is to maximize the number of running resource-independent live migrations until the next planning time $t+\Delta_{sch}$.

At every planning and scheduling time $t$, the resource dependency graph $G=(V,E)$ denotes the acyclic undirected graph where $|G|=|V|$. Each node $u\in|V|$ represents the list of migrations $M(u)$ where migration shares the same source $s$, destination $d$, and network routing $p$. By sharing the same source and destination and network routing, migrations in the list of a node are all resource-dependent. Let $(u,v)\in E$ denote the edge between node $u$ and $v$. It indicates the resource dependency between migrations from both nodes. $V\left(G\right)$ denotes the set of nodes of graph $G$.

We model the multiple migration planning problem as generating the maximal independent set of the dependency graph iteratively. In other words, in each iteration, we get the maximal independent set of the remaining graph, then update the graph by deleting corresponding migrations. Let ${G_{i+1}}={G_{i}}\left[{V\left({{G_{i}}}\right)-{S_{i}}}\right]$ represent the remained graph by directly deleting vertex from set of nodes $S_{i}$. Let $I_{i}$ denote the maximal independent set of graph $G_{i}$. Then, the remained graph $G_{i+1}$ in each iteration can be represented as:

by deleting set of nodes $S_{i}=\{u|u\in I_{i},M(u)=\emptyset\}$, where the migration list of the deleted node $u$ is empty. Therefore, for each migration planning, the objective is to generate the iterative maximum independent set of dependency graph:

where $\left\{{I_{iter}^{i}}\right\}=\left\{{{I_{1}},{I_{2}},...,{I_{K}}}\right\}$ is the total K iterative independent sets and there is no vertices left in the $K+1$ remaining graph as ${G_{K+1}}=\emptyset$. In other words, each iterative independent set size equals the size of maximum independent set of remaining graph $\left|{{I_{i}}}\right|=\alpha\left(G_{i}\right)$.

We extend the model to generate the iterative maximum weighted independent set for migration with different priorities, such as migration deadline. The weight of an independent set is $W\left(I\right)=\sum\nolimits_{u\in I}{W\left(u\right)}$. The largest weight of migration $\hat{m}$ in the node migration list is the weight of its corresponding node in the dependency graph $W\left(u\right)=W\left(\hat{m}\right)$ that

Then, the objective of multiple migration planning can be represented as:

The weight of node $W(u)=1$ when there is no need to differentiate migrations in different nodes. Generating the maximum (weighted) independent set of an undirected acyclic graph is a well known NP-hard problem [34, 35]. Therefore, generating the iterative maximum independent set as the subset is also NP-hard.

Because an independent set of $G$ is a clique in the complement graph of G and vice versa, the independent set problem and the clique problem are complementary [33, 34, 36]. In other words, listing all maximal independent sets or finding the maximum independent set of a graph equals listing all maximal cliques or finding the maximum clique of its complement graph. Thus, in each iteration, we can equivalently find the maximum independent set by getting the maximum clique ${C_{i}}$ of the complement graph ${C_{i}}\left({{{\bar{G}}_{i}}}\right)={I_{i}}\left({{G_{i}}}\right)$.

It is known that all maximal cliques can be calculated in a total time proportional to the maximum number of cliques in an n-vertex graph [36]. In other words, each clique is generated in a polynomial time in all maximal cliques listing [34]. When we only consider vertex, the maximal cliques listing algorithm (CLIQUES) [36, 35] based on Bron-Kerbosch [33] is the optimal algorithm. The worst-case running time of CLIQUES is $O({{3^{n/3}}})$. The upper bound of all maximal cliques or independent sets of a graph is ${{3^{n/3}}}$ [37]. For the problem of finding one maximum independent set, the time complexity is improved from $O({{2^{n/3}}})$ in [38] to $O({{2^{0.276n}}})$ [39]. Based on the work [39], the best-known time complexity is $O\left({{2^{n/4}}}\right)$ [40]. Therefore, it is computationally impossible to solve the exact problem of listing all maximal cliques (maximum clique) of its complement graph ${{\bar{G}}_{dep}}$ or all maximal independent sets (maximum independent set) of $G_{dep}$ for the real-time live container migration scheduling in edge computing which exhibits an exponential time complexity.

For the direct iterative MIS generation, we follow the rationals based on the planning model as follows: (1) Create dependency graph $G_{dep}$ based on the source, destination, and network routing of the input migrations and the network topology; (2) Generate the Maximum Independent Set (MIS) $I$ of $G$; (3) Delete the nodes $u\in I$ from $G$ if its migration list $M(u)$ is empty; and (4) Repeat the procedure 2 and 3 until there is no vertices left $G_{dep}=\emptyset$.

In this section, based on the observation of the density property of migration resource dependency graph, we propose the iterative Maximum Cliques (MCs)-based algorithm. We first discuss the rationals of the proposed algorithm.

The degeneracy of a graph G is the smallest number d such that every subgraph of G contains a vertex of degree at most d. It is a measure for the graph spareness. For an n-vertex graph with degeneracy $d$, by introducing the sequence ordering based on degeneracy, Bron-Kerbosch Degeneracy algorithm [43] can list all maximal cliques in time $O({dn{3^{d/3}}})$. With a spare graph that $n\geq d+3$, the upper bound of all maximal cliques number is $\left({n-d}\right){3^{d/3}}$. Figure 6 illustrates the nodes and the density (degeneracy) of the resource dependency graph of WAN network topologies [44] and its complement. It shows that the degeneracy of the complement graph ${{\bar{G}}_{dep}}$ is 4.34 times that of $G_{dep}$. For $G_{dep}$ and its complement graph, the average ratio of dependency $d$ to the total number of nodes $n$ is 0.153 and 0.714, respectively. The resource dependency graph is considerably more sparse than its complement graph. Therefore, for $G_{dep}$, there are much fewer maximal cliques than the total MIS. As a result, according to the theoretical time complexity, the running time of listing all maximal cliques or maximum clique of $G_{dep}$ is much smaller than that of listing all maximal independent sets or maximum independent set of $G_{dep}$. Therefore, the iterative Maximum Cliques (MCs)-based heuristics algorithm has two steps: (1) calculate the list of iterative maximum cliques and (2) generate the iterative maximal independent set based on the list. As nodes from one maximal clique can not be included into the same independent set, the iterative maximum cliques serve as a heuristic pruning decider to speed up the algorithm.

In this section, we describe the base stations coordinates provided by Shanghai Telecom dataset¹¹1http://sguangwang.com/TelecomDataset.html and Shanghai Qiangsheng taxi GPS trace dataset (April 1, 2018)²²2http://soda.shdataic.org.cn/download/31 used in our experiments. The given data is preprocessed by limiting the range of the longitude and latitude from ${30.40^{\circ}}$ N to ${31.35^{\circ}}$ N and ${120.51^{\circ}}$ E to ${122.12^{\circ}}$ E as there are some taxis travel to nearby cities. The Shanghai Telecom dataset contains the longitude and latitude coordinates of a total of 3233 base stations as shown in Fig. 10(a). We use K-means algorithms [47] to generate the location of a total of 200 Edge Data Centers (EDCs) based on the longitude and latitude of the given base stations. Figure 10(b) illustrate the taxi GPS trace in the first hour. Besides the GPS coordinates and timestamp, each data record of the taxi dataset also includes the taxi id, service status, such as alarm, occupation, taxi light, road type, and breaking, as well as vehicle speed, direction, and the number of connected satellites.

Figure 10(c) illustrates base stations are clustered and connected to one of the regional Edge Data Centers. There is no information on the physical network topology and connectivity between EDCs. As shown in Fig. 10(c), for the geometric spanner, we choose Delaunay Triangulation [48] to generate links between the gateway of each EDC. For the network routing within the generated network topology, we consider the shortest path, which is no longer than $4\pi/3\sqrt{3}$ times the Euclidean distance between source and destination. As a result, the boundary of EDC regions (Fig. 10(d)) is a Voronoi diagram [48] where the Euclidean distance of any point to its corresponding EDC region is less than or equal to its distance to any other EDC.

Similar to other research regarding the generation of mobility-induced live migrations in edge computing [17, 18], we combine these two datasets to simulate the scenarios where the user needs to connect to the services and maintains the low end-to-end latency through live container migration in edge computing environments. Figure 11 demonstrates an example that the request of live container migration is induced when a taxi moves across the boundary between two clusters of EDCs. The deadline of each live container migration is generated based on the average mobility speed of users in the last 3 GPS records, travel direction, and the signal strength of base stations.

In this section, we describe details of the experiment setup. The end-to-end delay between the user and the service is the time interval from the user (taxi) sent workload to the container assigned in the EDC to the result is received by the user. To generalize computer vision use case workloads, the service task generated during the experiments follows the Poisson distribution with a mean of 24 per second (24 FPS). In each task, the network packet size sent from a user is 16384 bytes ($128*128$ bytes). The processing workloads in the container are randomly generated from 500 to 1000 cycles per bit [18]. The result packet sent back from the container to the user is 128 bytes. The sum CPU power frequency for each EDC with multiple CPUs is 25 GHz [49, 50]. To simulate the limited network resources for migrations in the edge, we consider that the reserved network bandwidth between a container and its user is 3 Mbps. The physical network bandwidth is 1 Gbps. The network delay between any based station to its regional EDC is 5 ms and the delay between two EDCs is randomly generated in the range 5 to 50 ms [18].

According to the evaluation results of container memory and dirty memory size during live container migrations [16], we generate the container memory from 100 MB to 400 MB. The dirty page rate for each dirty memory transmission is from 2MB/s to 8MB/s and the data compression rate is 0.8 [51]. We configure the downtime threshold and the maximum iterations for live container migration at 0.5 seconds and 30 times [30], respectively. Based on the SDN controller, the remaining network bandwidth between the source and destination EDC which is not utilized by services is allocated to the live container migration traffic. If several live migrations are sharing part of their routings, the bandwidth will be allocated evenly to each of the network flows.

From the experimental scenario S1 to S3 (Table III), 1000, 2000 and 4000 vehicles are selected randomly. We consider the GPS trace of selected vehicles within 1 hour. For the initial placement at the start of the experiment, we allocate corresponding containers for each vehicle at the same edge data center according to its GPS coordinates. The nearest edge data center first policy is considered for our experiment to generate the live container migration requests. According to the user mobility, one live container migration will be triggered when one vehicle exits the coverage area of its current edge data center. There are 9933, 19522, and 37822 migration requests induced by these vehicles’ movement, respectively. During the live container migration, the dirty memory of migrating container will be copied iteratively from the source edge data center to the nearest edge data center through the shortest network path. For the evaluation sensitivity, the results of each scenario are an average of 10 individual experiments. In this experiment, we consider that the container image as a universal service is already available in all edge data centers or shared by the network storage between EDCs. CloudSimSDN-NFV [46], an event-driven simulator, is extended with corresponding components to support the live container migration and user mobility in edge computing environments.

In Section 6, we compare the algorithm performance in terms of the size of iterative MISs and processing time. Thus, we only evaluate the two best algorithms in this section. We compare the experimental results between no migration scheduler, iter-GWIN, single-MCs, and the current state-of-the-art migration planning and scheduling in clouds FPTAS [22]. In FPTAS, to maximize the total bandwidth utilized by migrations, one migration can be started even there is considerably limited bandwidth which is much lower than the dirty page rate per second. The solution can cause devastating migration performance. Thus, we improve FPTAS by adding a bandwidth threshold (FPTAS-BW) that the available bandwidth must larger than the dirty page rate as the migration start condition. As the vehicle number increases from scenario 1 (S1) to scenario 3 (S3), the density of live migration requests in certain areas increases dramatically. The resource competition or resource dependency among live container migration requests will also increase. As a result, the complexity of the dependency graph may also increase. When the requirements of live container migration requests exceed the resource capacity provided by the edge computing, it is inevitable that some of the deadlines of some migration requests can not be satisfied.

Table IV shows the total processing time of migration planning and scheduling algorithms within 1 hour in milliseconds. From S1 to S3, the average processing time of single-MCs for each migration planning is 0.1175, 0.1936, and 0.4904 milliseconds. Compared to iter-GWIN, the processing time of single-MCs decreased by $61.36\%$ in scenario S3. The results are consistent with the algorithm evaluation in Section 6. Furthermore, compared to FPTAS [22], the performance of our solution in terms of processing time has been improved by more than 3000 times. In S3, the processing time of FPTAS is about 78 minutes. As a result, even with any weight modification in the algorithm, the migration deadline in seconds will be missed. Therefore, as the results of FPTAS-BW in deadline violation are off the limit of chart comparison, we only compare it in the migration performance.

From S1 to S3, without migration planning and scheduling, more live migrations compete with each other on the network routing and the available bandwidth. As a result, the average migration time increases dramatically from 2.25 and 4.59 seconds to 299.89 seconds (Fig. 12(a)). Particularly, in S3, the allocated bandwidth may either be smaller than the dirty page rate and cause a large downtime for some migrations. Or, it causes a much longer migration time due to a large number of memory-copying iterations. As a result, the migrating service suffers a devastating consequence. Furthermore, for FPTAS-BW, by maximizing the total migration bandwidth rather than the resource competitions, it suffers smaller average bandwidth per migration. Thus, as shown in Fig. 12 the performance of our purposed solution in terms of average migration time, average downtime, and total transferred data are increased by up to $30.24\%$, $51.56\%$, and $2.06\%$, respectively. Meanwhile, for the proposed planning and scheduling algorithms iter-GWIN and single-MCs, the performance of live migration can be guaranteed even with severe resource competitions. Results (Fig. 12(a), 12(b)) show that the average migration time and downtime are optimal at 1.9 sand 0.13 seconds as there is no bandwidth sharing between resource-dependent migrations. Furthermore, for all the migrations that arrive within the 3600 seconds time interval in S3, iter-GWIN and single-MCs can finish the scheduling of all migrations in 3603.91 and 3601.43 seconds. However, the total migration time of no scheduler is 3603.43 seconds in S2 and 48878.65 seconds in S3. A shorter average migration means less possibility of QoS degradation and less occupation time on the network resource. A smaller downtime equals fewer disruptions on the migrating services.

Another critical migration performance is the transferred data of the live migration. It is also highly related to network energy efficiency. In S1 and S2, although average migration time and downtime increase due to less allocated bandwidth, there is no surge in the transferred data for the no migration scheduling situation (Fig. 12(c)). Because of the container’s small memory footprint, the shared bandwidth can still satisfy the downtime threshold with relatively small memory-copying iterations. However, when the bandwidth becomes the bottleneck, a large number of memory-copying iteration needed to meet the downtime threshold. Therefore, the total transferred data in S3 increase by $114.47\%$ compared with the optimal result from single-MCs.

The deadline of a live migration request is highly related to the QoS and SLA requirement of the real-time migrating service. For iter-GWIN and single-MCs, the ratio of migration violation numbers to the total migration number is $0.071\%$ and $0.107\%$ in S2 and $0.756\%$ and $1.002\%$ in S3 (Fig. 12(d)). However, the ratio for no migration scheduler is 3.07 times in S2 and 8.46 times compared to the best result from iter-GWIN. The ratio of total violation time to the service time of all containers in one hour is $0.00127\%$ and $0.00148\%$ in S2 and $0.0425\%$ and $0.0720\%$ in S3, respectively (Fig. 12(e)). In S3, although migration performance in terms of migration time and downtime is optimized by the migration scheduler, the network resource is insufficient to schedule all 37822 migration requests on time with the live migration competitions. It is inevitable to violate the deadline of certain migrations with lower priority to satisfy the deadline for others. As a solution, one needs to increase the network resource by providing duplicate EDCs and additional network routing or available bandwidth in the hot spot to alleviate the deadline violation of real-time migrations.

The end-to-end delay for the migrating edge service is affected by the migration downtime and the duration of deadline violation. For the network transmission time, we compare the results of no user movement and no migration, no migration requests with user movement, no scheduler, iter-GWIN and single-MCs (Fig. 12(f)). In the scenario that all vehicles stay at the s and do not move during the experiment time (nomov), the average network transmission time to the service or the end-user is from 17.4 to 19.9 milliseconds from S3 to S1. Without the live migration requests (nomig), the end-to-end delay can be not guaranteed due to the network delay between the EDC and the end-user. Specifically, the average network transmission time is around 56 milliseconds. The live migration planning and scheduling algorithm (iter-GWIN and single-MCs) can guarantee the average service network transmission time. In S3, without a migration scheduler, the downtime and deadline violation have a considerable impact on the service network delay. The network delay increases by 6.62 times compared to the result of iter-GWIN.

In summary, our proposed algorithms can efficiently plan and schedule large-scale mobility-induced live container migrations in edge computing. Even in the case of a migration request surge, it guarantees the performance of live container migrations and maintains the QoS of migrating services. It significantly reduces average migration time (up to $99.36\%$), average down time (up to $99.94\%$), total deadline violations (up to $88.18\%$) and violation time (up to $99.94\%$).