Mirage: Towards Low-interruption Services on Batch GPU Clusters with Reinforcement Learning

As scientists and DL practitioners investigate more complex problems, they often utilize larger models and datasets. For example, the 345 million parameter BERT (Devlin et al., 2018) model takes 5 days to train with 8 NVIDIA A100 GPUs on the English Wikipedia dataset (Pauloski et al., 2021). The 175 billion parameter GPT-3 model is estimated to take 34 days to train with 1,024 NVIDIA A100 GPUs (Narayanan et al., 2021). Scientists are also deploying trained models for real-time data processing or object detection. Some examples include transient celestial object detection (Ivezić et al., 2019), point-scanning electromagnetic imaging super-resolution (Fang et al., 2021), and disease detection in digital agriculture (Mohanty et al., 2016). These applications require running long inference jobs as a service, which is problematic since modern GPU clusters frequently impose 48- or 72-hour limits to ensure responsiveness.

Reinforcement learning (RL) is a machine learning method based on trial and error. At a time step $t$, an RL agent takes an action $A_{t}$ to react to the current system state $S_{t}$, and thereby, shifts the system to the next state $S_{t+1}$. The agent receives an immediate reward $R_{t}$ for triggering the transition, and this process of making decisions and collecting rewards repeats to until a terminal step $T$. The RL agent is trained to learn an optimal that maximizes the accrued rewards within the time horizon ($t=1,\dots,T$). When the learned policy is non-deterministic and the state transition is stochastic, the trajectory of produced states, actions and rewards is uncertain and we denote it with sequence of random variables $S_{1},A_{1},R1$, …, $S_{T},A_{T},R_{T}$.

At time step $t$, we use function $Q_{\pi}(S,A)$ to indicate the the expected return (cumulative reward) starting from $t$. $\pi$ itself is function that maps an encountered state to its decided action, and the discounting factor $0\leq\gamma\leq 1$ translates future rewards to present values.

with Monte Carlo simulations, one can rollout different trajectories from a policy, and then, compute an average return for each visited state as a sample estimate of $Q(S,A)$. This would converge if every state and action is visited infinite times. Learning can also happen online by bootstrapping with the $Q$ values of the next state in the sequence,, which is referred to as Q-learning (Mnih et al., 2015):

$s^{\prime}$ is the next state transitioned from $s$, and $k$ represents the iteration of learning that updates $Q$. The Q-values are captured in a tabular manner but this could get cumbersome for large state-spaces. Recent studies (Silver et al., 2016, 2018) use a deep neural network with weights $\theta$, namely a deep Q network (DQN), to represent $Q$ and its loss function and gradients for the $k$-th iteration are defined as below. Rather than computing the full expectation, the loss function can be optimized with mini-batch Stochastic Gradient Descent (SGD).

Deep Q-Networks (DQN) is a value-based RL method that is known for a few limitations. First, DQN lacks a direct representation of the policy, and policy inference requires solving an optimization problem $\text{argmax}_{a}Q(s,a)$ and this challenges when the action space is high-dimensional. Second, DQN does not automatically trade off exploit and exploration, and usually relies on human-crafted strategies such as the $\epsilon$-greedy strategy (Mnih et al., 2015) to guarantee exploration. However, manually tuning the $\epsilon$ threshold is difficult. To overcome these limitations, policy gradient (Silver et al., 2018, 2016) is widely used; it parameterizes a policy with a deep neural network and directly outputs the probability of the action to take. Moreover, policy gradient can autonomously arbitrate exploit and exploration.

We briefly formulate policy gradient as below. First, let $\tau$ denote a random trajectory that comprises of a sequence of state, action, reward triples, i.e., $s_{1}^{\tau},a_{1}^{\tau},r_{1}^{\tau},\dots,s_{T}^{\tau},a_{T}^{\tau},r_{T}^{\tau}$, to until the terminal time step $T$; let $\pi_{\theta}(\tau)$ denote the probability distribution of $\tau$ when it is realized with a stochastic policy $\pi$. $\pi$ is implemented using a deep neural network with parameters $\theta$; let $r(\tau)$ represent the cumulative reward $\sum_{t=1}^{T}r_{t}$ and let $J(\theta)$ represent the expectation of $r(\tau)$, i.e., $J(\theta)$=$\mathbb{E}[\sum_{t=1}^{T}r_{t}^{\tau}]$. Note that, unlike value based methods, policy gradient directly maps a state $s_{t}$ to its decided action $a_{t}$ with (softmax) probability $\pi_{\theta}(a_{t}|s_{t})$. The policy gradient theorem states that

, which can be approximated with Monte Carlo rollout,

, and the policy network can be update with gradient ascent and learning rate $\alpha$, i.e., $\theta\leftarrow\theta+\alpha\nabla_{\theta}J(\theta)$, which iteratively converge to a (local) maximum. Note that although policy gradient solves some of the issues of deep Q-learning, it also incur some other issues; its learned policy may converge to a local optimum (Pierrot et al., 2022), and in real-world applications, it is found to suffer from high-variances (Papini et al., 2018). As there arguably is no clear winner, we adopt both the DQN and the policy gradient method.

Scaling up model capacity is one of the key success of deep learning, as increased model capacity usually renders higher predictive power. There are three learning strategies to enhance model capacity, i.e., scaling up a monolithic model (Li et al., 2021), ensemble learning (Codella et al., 2016) and MoE (Mixture of Experts) (Yuksel et al., 2012). We try all these three different strategies to build and train RL provisioners, and evaluate them in Section 4. Note that we increase model capacity but cautiously prevent over-fitting using cross-validation.

Scaling up a monolithic model. One strategy to increase model capacity is to train a huge, monolithic deep neural network that may include hundreds to thousands of layers and billions of parameters (Li et al., 2021). However, training a huge monolithic model is a herculean task. First, training efficacy deteriorates as gradient vanishes when back-propagated through a deep architecture. Residual bypasses and advanced activation functions mitigate this issue but with no complete solution. Second, a huge, complex model has a higher degree of curvature and contains more saddle points and local minima, and gradient descent is more prone to stuck. Third, training a huge neural network is computationally prohibitive.

Ensemble learning. The other strategy to level up model capacity is ensemble learning such as Random Forest (RF) (Breiman, 2001) and Gradient Boosting Decision Trees (GBDT) (Friedman, 2001), which average the outputs of multiple learners, each trained on a sub-datasest. However, aggregating a large (infinite) number of weak learners (each underfits the complex data and incurs high biases), increases model complexity but does not effectively increase model capacity. Thus, RF or GBDT usually have difficulties in modeling complex supervised learning tasks such as language modeling. Moreover, classic ensemble learning adopts a static mixture weights for its member learners. However, learners usually perform differently for varied input regions, and the mixture weights should be adaptively different for different input samples.

Mixture of Experts (MoE). We believe the philosophy that instead of training a complex, monolithic model or an ensemble of many weak models, one should train and combine several models of intermediate complexity (Geoffrey Hinton, [n. d.]). Following this philosophy, the MoE deep neural networks (Yuksel et al., 2012) has achieved remarkable success. An MoE neural network contains many experts that share the same network architecture, each of which is trained on a partition of the training samples. There is a mixture (or gating) layer that adaptively routes inputs to their best-fit experts, and the best-fit experts’ out are aggregated (averaged) as the final output. This MoE scheme scales up model capacity with no significant increase of computation overhead. We adopt the softmax gating layer (Shazeer et al., 2017), which computes the weighted average of the Q-values across $E$ different DQN experts.

As shown in Table 1, the V100, RTX, and A100 traces have 65,017, 175,090, and 17,570 jobs respectively. Figure 2 depicts the job count distribution over time. The average job count is $2,955\pm 1,289$, $8,378\pm 20177$, and $4,377\pm 659$ per month for the V100, RTX, A100 cluster respectively. As we can see from the figure, there is no clear pattern of job arrival at a month granularity. It is also worth noting that there are 96,780 short jobs (less than 30 secs) on RTX. We do not remove such short jobs from our job traces, as they reflect the real machine usage at that time.

For the node count distribution, the average nodes per job is 2.5, 1.3, and 1.6 on V100, RTX, and A100, respectively. Although multi-node jobs take only a small portion of total job count, their share in node hour consumption is more significant as shown in Figures 3(a),  3(b), and 3(c). For example, in 2021-2 on V100, the percentage of multi-node jobs is 23.4%, but they take 76.9% of the total node hours. Similarly in 2017-11, 12.0% of jobs are multi-node but take 82.5% of total node hours. This observation aligns with the trend of the time-consuming multi-node DL training.

Finally, the statistic most directly relevant to this research, the queue wait time, is depicted in Figure 4. The average queue wait time on the V100 and RTX clusters is shown in Figure 1. The distribution from individual months are presented in Figure 4(a), Figure 4(b), and Figure 4(c). In 2020-10 and 2021-2, $\sim 30-41\%$ of the jobs on V100 have to wait for longer than 24 hours. On RTX, the percentage of jobs waiting for longer than 24 hours is $\sim 12-24\%$. On A100, 92-98% of jobs experienced a wait time less than 12 hours across the five-month period except 2023-2, where 26% of the jobs are waiting for more than 12 hours, and 3% are waiting for longer than 36 hours. Long wait time significantly limits the service provided by batch GPU clusters for long running DL training and inference jobs.

We manually filter out jobs that 1) request more nodes than the available and 2) sub-jobs that are submitted within one Slurm job. In the early-production phase of these clusters, all nodes are in the same partition, then they are split into multiple partitions to address the needs of development and test and production run. Since our job traces are collected in the production partition, there are early jobs that request more nodes than that are available in the partition. So we remove those jobs from the job traces. There are also jobs, though recorded in the Slurm database, that are sub-jobs within one Slurm job. These jobs have an identical prefix followed by the sub-job id. So we combine them as one single job with the starting time of the first sub-job as the start time, and the end time of the last sub-job as the overall end time. Table 1 summarizes the statistics of the job traces.

The machine downtime, e.g., maintenance, leaves blank time ranges in job traces. We take the blank time ranges as no users submit any jobs during that time. Since the scheduled maintenance is one day every one or two months, it does not impact model training significantly. Slurm allows users to submit jobs with dependencies, and it releases the next job to the queue upon the finish of the previous job. This dependency is not reflected in the job traces, so we take jobs with dependencies as independent jobs submitted at different time. This approach does not change the queue wait time of the subsequent jobs.

We represent the system state at each instant $t$ with an $m$-dimensional vector $v_{t}$. By default, $m$ is 40 and the vector comprises the following variables to encode queue, server and job information.

(a) Queue State: First, queue state includes the number of queued jobs currently waiting for scheduling ($var1$). Second, it includes summary statistics (i.e., 0th, 25th, 50th, 75th, and 100th percentiles) of the sizes (number of requested nodes) of the queued jobs ($var2-var6$). Third, queue state includes summary statistics of the ages (time since submission) of the queued jobs ($var7-var11$). Fourth, it includes summary statistics of the runtime limit (maximum duration that each job is allowed to run in the cluster) of the queued jobs ($var12-var16$).

(b) Server State: First, server state includes the number of actively running jobs across all servers in the cluster ($var17$). Second, it includes summary statistics of the sizes ($var18-var24$), elapsed runtime ($var25-var29)$ and runtime limit ($var30-var34$) of all running jobs.

The model only considers submission time prediction for a single successor job, after the submission of its predecessor job. For example, suppose a user job $J$ is partitioned as four 48 hour long sub-jobs $J1$, $J2$, $J3$ and $J4$. The model only considers optimizing the submission time of the successor sub-job $J2$ after the predecessor sub-job $J1$ is submitted. When $J2$ is submitted per model’s decision, $J2$ becomes the predecessor and $J3$ becomes the successor, and so on, until $J4$ is submitted. Therefore, the model maintains a current Predecessor-Successor pair for each group of chained sub-jobs, and at each instant, the state representation also includes the status quo.

(c) Predecessor job state and (d) Successor job information: The predecessor states includes the size ($var35$), time limit ($var36$), queue time ($var37$), and elapsed runtime ($var38$). The successor information includes its size ($var39$), and time limit ($var40$). The successor job has not entered the cluster yet and thus only static information is included. It should be noted that we consider jobs’ internal state, such as say an ML job’s epoch progress to be private to the user. That is, the agent does not know or require users to annotate or expose any job-specific internal information for training.

The state matrix. The 40-dimensional vector $v_{t}$ is an abstraction of the queue, cluster and job states at an instant $t$, while $k$ consecutive vectors $v_{t}$, $v_{t-1}$, $\dots$, $v_{t-k}$ encode both workload (job arrival and completion) and cluster (the queue and server) history of length $k$. Existing RL work, such as playing Altari games (Mnih et al., 2015), adopts a similar approach in constructing model input, within each the last four video frames (last four snapshots of video game) are taken as model input. In parallel, our model input is a $k\times m$-dimensional state matrix $S_{t}$ at each instant $t$. By default, we set $k$ as 144, and periodically record $v_{t}$ at a default interval of 10 minutes which corresponds to the workload and cluster history of the last 24 hours back-traced from $t$.

Unifying the state and action space for DQN and PG network. The action space for our RL provisioner consists of only two different actions: submit (submission) and no-submit (no-submission). The DQN network takes as input both the state matrix and the action to query, while the PG network takes as input only the state matrix. To render a unified model input, we add an ordinal variable that represents the action to query. For the ordinal action variable, 1 represents submission, -1 represents no-submission, for which the DQN network realizes it as an essential input, while the action is only a placeholder and is always 0 when inputted to the PG network. When training the foundation transformer model, we flatten the state matrix to a long vector and concatenate with the additional ordinal action variable. The flattened states (by default) contains 5761 (144$\times$40+1) variables (cf., Figure 5).

Deterministic Policy. For a learned DQN policy, the neural network is a value function $Q(S_{t},a_{t})$ that predicts, at the current instant $t$, the expected future gains of submitting versus not submitting the successor sub-job, given the current state and the history encoded in the state matrix $S_{t}$. For each targeted predecessor-successor pair, after the predecessor is submitted, the DQN model is periodically invoked at an interval of 10 minutes, and will submit the successor sub-job only when the Q-value of submission is larger than that of no-submission. Non-deterministic policy. For a learned PG policy, the network outputs directly the probability of submission versus no-submission, and the action to take is randomly sampled from this output binomial distribution (cf., 5).

Different HPC practitioners can have their own views of the penalty of overlap and (or) interruption between the predecessor and successor sub-jobs. Performance-sensitive users may consider interruption to have a much larger penalty than overlap, while resource-waste-aversion users may consider overlap to have a larger penalty than interruption. We set two user-configurable penalty coefficients, $e_{I}$ and $e_{O}$ , for users to configure their penalty for interruption and overlap, respectively. Given this, we first set the time horizon of actions. Suppose the current instant is $t^{\prime}$, at which the predecessor sub-job is submitted, and after that, the agent’s trained DQN or PG policy produces a sequence of actions $a_{t^{\prime}+1}$, $\dots$, $a_{t^{\prime}+T}$. We can see that $a_{t}=0$ for all $t^{\prime}+1\leq t<t^{\prime}+T$, and $a_{t^{\prime}+T}=1$ as the terminal action for the successor sub-job is submission, while any other actions prior to $a_{t+T}$ are all non-submission actions.

After the successor job is submitted at instant $t^{\prime}+T$, the agent closely monitors the state of the successor job and when it is dequeued for running at some future instant $t^{\prime}+T^{\prime}$, the outcome of how much the overlap or interrupt is revealed. We use $r_{I}$ to denote an outcome interruption and $r_{O}$ to denote an outcome overlap.

With the settings above, we shape the reward $R_{t}$ for actions $a_{t}$ as below. Note that we use negative penalty to equally represent reward, which means a reward of zero is the best possible reward, and the larger the interruption or overlap penalty is, the smaller the reward.

If the observed interruption is small, the previous actions are rewarded to making the right decisions. That is, the previous no-submission decisions are credited for not causing overlap, and the final submission decision is credited for being proactive and timely. A similar argument applies when the observed overlap is small.

The foundation model - transformer. We build the foundation model with a transformer (Figure 5)) model. Transformers are the state-of-the-art method for sequence modeling and prediction. For example, in NLP tasks like language modeling, a transformer model takes as input a sequence of previous words to predict the sequence of words that follow. This resembles our proactive scheduling task where we parse a history of system (e.g., queue and node) snapshots and forecast the change in system states. The difference lies in that, for language models, the transformer decoder deciphers the intermediate representation into predicted words, while for proactive scheduling, the decoder translates the intermediate representation of future system states into queuing delay, which can be used to compute the resulting interruption (or overlap). Furthermore, system state has both long-range dependencies (e.g., periodic, weekly jobs) and short-range dependencies (e.g., bursty arrival of jobs) over the time horizon. Thus, not all system snapshots in a history window provide dependent, relevant features to forecast the status quo and future. The multi-head self-attention module filters out irrelevant snapshots in history and identifies ones that contribute to prediction. Self-attention is the key technique in Transformer or BERT to improve upon traditional sequence modeling and achieve state-of-the-art results for classification tasks, including language modeling, machine translation and sentiment classification. Thus we propose using self-attention as the key enabler to accurately predict future system states and reinforcement learning rewards.

The V-head and the P-head output layers. The deep foundation model learns intermediate representation of the policy internal values and the it is either followed by a single linear map to reduce to a scalar output (i.e., the Q-value) of the V-head, or followed by a linear decision layer plus a softmax layer, which output the distribution of action sampling for the P-head (cf., Figure 5).

Hyperparmeter tuning. The hyperparameters that define the model structure, such as the number of multi-attention heads per transformer encoder layer, are tuned with RayTune (Liaw et al., 2018), an auto-ML engine for hyper-parameter tuning. Figure 5 shows the tuned hyperparameters for all network layers. This set of tuned hyperparameters is repeatedly used as the default throughout this study.

As discussed in Section 2.4, we further augment the capacity of the transformer foundation model with the MoE scheme. Figure 6 shows the MoE architecture and the workflow to combine experts. We implement and evaluate both the Top-1 sparse MoE (Shazeer et al., 2017) and the weighted average dense MoE (Shazeer et al., 2017) and found that the Top-1 scheme exhibit inferior provisioning performance in comparison to the weighted average MoE, though it enjoys less computational overhead due to sparse activation. Throughout this study, we omit the results of the Top-1 scheme for brevity.

The intuition behind MoE is similar to that of piecewise polynomial fitting such that individual polynomials focus to fit a small region (partition) of the input space (samples), and all the polynomials integrate to fit a complex function over the entire input space (complete set of samples). As cluster workloads and machine busyness change from weeks to weeks and months to months, we temporally split the cluster log records (training samples) to $M$ (10 by default) fractions to train different expert transformers. Different experts focus on optimizing provision policy for different load levels and different workload mixes.

Online RL usually fails for complicated control tasks if there exists correlation between consecutive training samples, which explodes the variance of gradient updates and distorts the a policy’s value estimates. We also employ experience replay (Adam et al., 2011) to break data correlations. Experience instances, each of which includes the submission or no-submission action taken by the trained policy, as well as its resulting reward (interruption or overlap), are stored in a memory pool. The memory pool is random as the experience instances are collected from different time points for each simulation trial, and are randomly shuffled across different simulation trials. Experience instances are then grouped into random mini-batches to perform training of the value (or policy) networks.

The training procedure consists of two phases. For the first phase, we train the foundation model offline while for the second phase, we train the V-head and the P-head decision layers online by running Slurm simulation experiments.

The overall system architecture of Mirage consists of two major components: the agent and the simulator, as shown in Figure 7. It is designed to be generic for both model-free and model-based learning and handles value function approximation and model training.

At a high level, the agent trains the proactive provisioner and the simulator simulates the Slurm scheduler using the provided job traces. The simulator implements the priority-based scheduling algorithm with back-filling (Yoo et al., 2003). and mimics the behavior of the production Slurm deployments used in our GPU clusters. The simulator loads the jobs trace that we collected over a 20-month period and exposes an interface of sample(), step(), and submit().

For each episode, the agent and the simulator will be initialized and run up to the time when the first job should be submitted. Then the agent submits the first job by calling submit(), calls sample() to acquire the cluster state. After that, the agent calls step() to instruct the simulator to move forward for a period of time, then collects the cluster state again. With sufficient states, the agent evaluates the rewards of submitting the second job, and makes a decision. Once a decision has been made, the agent calls submit() to submit the second job. Then the simulator runs till the second job starts and returns its final queue wait time that can be used to compute the reward.

The agent has a reward processor that receives the queue wait time from the simulator and converts it to the reward as required by the RL algorithm.

In order to train the agent, and validate the results, we develop a low-overhead Slurm simulator that supports Slurm’s core scheduling logic, i.e., backfilling and priority scheduling. There are two main roles fulfilled by our simulator. One is to simulate the workload to create a virtual cluster environment, and the other is to provide support for offline learning and online learning of agents. Since the time taken for each iteration of learning depends on the simulation speed, it is crucial that we have a low-overhead, yet high-fidelity simulator design. Our current build simulates a one month workload within one minute.

Our simulator includes three components, a cluster abstraction, a scheduler abstraction, and a simulator module that interacts with the agent. The cluster abstraction is used to consume the workload, while the scheduler determines which jobs will be submitted to the cluster based on its policy. The simulator module exposes a rich API for running workloads and allows controlling a number of factors such as running the simulator a specific length of time or running until all the jobs have been finished etc. The ability for users to customize such factors makes our simulator design general enough to support both offline and online learning.

We evaluate simulation fidelity using an identical workload trace, and compare our simulator with the standard Slurm simulator (slu, 2022; Simakov et al., 2018), which implements the exact Slurm scheduling logic that is used in real-word HPC clusters. We use 5 randomly sampled weeks as input and find that the difference in makespan across the five runs is less than 2.5%. We also compute the differences in job completion time (JCT) and find that the geometric mean of the difference is no more than 15% across all runs. Further, our simulator has a 3-26$\times$ lower overhead than the standard Slurm simulator.

In this experiment, we train each model by uniformly sampling the training time range and ran a pair of consecutive single-node jobs for 48 hours. Then we use the same sampling method in the validation range to examine generality. Single-node jobs are commonly for DL inference service. Figure 8 shows the average interruption/overlap of all techniques on the three GPU clusters.

As shown in Figure 8(a), when the cluster is heavily loaded, i.e., the reactive queue time exceeding 12 hours, random forest, XGboost, transformer, MoE, MoE+DQN and transformer+PG all show improved interruption with an average reduction of 44.1%, 33.7%, and 84.7% on the V100, RTX, and A100 cluster compared to the reactive baseline, respectively. MoE+PG is not as effective as other ensemble learning and RL methods. By comparing the training loss of MoE+PG with transformer+PG, MoE+PG overfits with equally low training loss with transformer+PG but a much lower validation performance.

We see a similar pattern when the clusters are with medium load in Figure 8(b). The ensemble learning techniques, DQN techniques, and transformer+PG show sigfinicant interruption reduction. Across the three clusters, Transformer+PG has the lowest interruption followed by MoE.

In this experiment, we examine the effectiveness of candidate methods using a pair of eight-node jobs, which is a representative scale across the three clusters. Figure 9 illustrates comparison with heavy and medium loads.

With heavy load as shown in Figure 9(a), the XGBoost and Random Forest methods show promising reduction of interruption by 37.5%, 40.0%, and 82.5% across the three clusters. MoE+DQN reduces the interruption by 32.2%, 28.2%, and 77.5%, which is slightly behind the ensemble learning methods. Transformer+PG has an improvement by 43.9%, 34.9%, and 90.1%, which shows the best results if we compare with an average.

With medium load as shown in Figure 9(b), We observe that ensemble learning methods almost completely eliminate the interruption. Transformer+PG shows a similar result. All these interruption elimination is at a cost of overlap when the machines are lightly loaded. We will discuss this in next section.

Now, comparing the effectiveness of MoE+DQN and transformer+PG methods across clusters, we see that they are more effective on the A100 cluster than on the V100 and RTX clusters. This is because the A100 job trace has less noisy jobs (those request hours of wall clock time but run for only 30 seconds), as we discussed in §3.1. The experiments in §6.1 and §6.2 with the A100 cluster clearly show the effectiveness of the RL-based methods, especially with a clean job trace.

Although the ensemble learning and RL-based methods can efficiently reduce the interruption when the machine is heavily or medium loaded. They inevitably introduce overlap between jobs when the machine is lightly loaded, where jobs can easily get compute nodes without a significant long wait time. Figure 10 shows the overlap of all methods with 1-node and 8-node jobs across the three clusters. Given the 48-hour wall clock time, a few hour overlap is not a big problem for the proactive provisioner, as the subsequent job needs to load data, checkpoints, and libraries. Once it is ready, the current job can be released, and the subsequent job will resume from the latest checkpoint, so there is no redundant computation or wasted node hours. However, we want the proactive provisioner to be intelligent to avoid overlap if possible. With this standard, we observe that the ensemble methods and the transformer+PG method introduce a $\sim$2X long overlap as the MoE+DQN method.

To balance the interruption reduction and overlap across machine load level, Mirage uses the MoE+DQN as its default model. We leave transformer+PG as a option for users as an aggressive provisioner, which will be more effective when the machine load is high.