SMDP-Based Dynamic Batching for Improving Responsiveness and Energy Efficiency of Batch Services

To meet the escalating demands for powerful computing capabilities, processors have undergone significant advancements in recent decades. The processors of today, including multi-core processors, graphics processing units (GPUs) and tensor processing units (TPUs), are equipped to better support parallel computing. This enhancement is crucial for efficiently managing large-scale data and executing complex tasks. For instance, GPUs have played a prominent role in accelerating the training and inference of neural networks due to their advantage in parallel computing[2]. These computing resources are now widely deployed across various levels—locally, on edge servers, and on cloud servers, providing computing services that facilitate ubiquitous access to intelligence at any time and from anywhere.

For servers equipped with processors capable of parallel operations, an important factor that affects both performance and cost of computing services is batch processing, or batching[3, 4]. Specifically, batching is usually employed for homogeneous tasks that share common operations, allowing the grouping of data into a unified batch for simultaneous processing on the server. The number of standard units of data or tasks gathered in a batch is referred to as the batch size. It is noteworthy that batching helps to better utilize computing and energy resources due to parallelism. This batching effect has been examined across different hardware platforms for various tasks, including basic matrix computations[5] and diverse machine learning (ML) inference models[6, 7, 8, 9, 10, 11].

However, opting for a larger batch size is not always the preferred approach, especially for online service provisioning where requests arrive in a random pattern, and expect responsive feedback. This gives rise to two main challenges in determining the batch size: (1) Larger batch sizes enhance energy efficiency and throughput by improving resource utilization and reducing per-sample I/O overhead. However, this benefit may come at the cost of decreased responsiveness, as batching can increase request response time due to potential waiting time needed to form a batch and extended processing time for handling multiple requests simultaneously[12, 4, 13]. This creates a tradeoff between efficiency and responsiveness[12]. (2) The use of statically configured batching proves inadequate in realistic scenarios[14], exhibiting poor responsiveness under low load conditions and limited throughput under high load[15]. To address these issues, a dynamic batching scheme is essential, allowing for judicious batch size adjustments to well balance the performance and cost.

In this paper, we study the dynamic batching scheme on batch processing-capable servers, aiming to strike a delicate balance between responsiveness and energy efficiency. This issue has gained increasing significance with the emergence of ML-as-a-Service (MLaaS) platforms like Google Cloud Prediction[16], where trained ML models are published on the platforms to provide inference (prediction) services for massive end users. As illustrated in Fig. 1, batch processing the inference requests becomes a natural strategy, for efficiency and economical concerns.

We commence our exploration with intra-processor parallelism, considering a scenario with a single server equipped with a single parallel computing processor. We leverage the theoretical framework of sequential decision-making to address the sequential batch size decisions. In our context, where we assume Poisson request arrivals and an arbitrary service time distribution, the problem is formulated as a semi-Markov decision process (SMDP). The objective is a weighted sum of the long-term average request response time and average power consumption. Notably, to the best of our knowledge, the optimal control problem for batch service queues with size-dependent service times remains unexplored in the literature. Moreover, the inherent complexities of the formulated SMDP problem—characterized by an infinite state space, an average (non-discounted) objective, and unbounded costs—pose challenges for efficient resolution using traditional methods. To address these challenges, we propose a procedure to solve the SMDP problem and derive an approximate optimal policy, which manifests as the selected dynamic batching scheme. Our main contributions are summarized as follows:

1. 1.

   To the best of our knowledge, our work is the first to rigorously formulate and optimally solve the dynamic batching problem for online computing services. The batching decision is formulated as an infinite-state SMDP, with the objective of minimizing the weighted sum of average response time and average power consumption.

2. 2.

   A new method is proposed, composed of finite state approximation, model “discretization” and relative value iteration, to obtain an approximate optimal policy. The demanding problem of state space explosion is tackled by a novel abstract cost, which reflects the impact of costs in “tail” states.

3. 3.

   We also conclude the theoretical results regarding the optimal policy structure in special cases. The SMDP solutions obtained through the proposed general method are visualized under different parameter settings. On one hand, the computed policies align with the theoretical results in special cases, affirming the effectiveness of the proposed method and the correctness of the theoretical results. On the other hand, certain instances reveal that the theoretical results might not extend to more general scenarios, underscoring the necessity of the general solving approach.

4. 4.

   Extensive numerical results demonstrate that the SMDP-based policies achieve the lowest average cost compared to benchmarks. The latency-energy tradeoff curves show that when having the same average response time, the SMDP-based policies never consume more energy than any other benchmark policy, and vice versa. Moreover, the proposed scheme can adapt to different traffic intensities and flexibly balance the response time and power consumption.

The rest of this paper is organized as follows. We begin by presenting related works in Section II. The system model is introduced in Section III, followed by the SMDP formulation in Section IV. A procedure for solving the SMDP problem is proposed in Section V. Theoretical analyses regarding the optimal policy in special cases of the problem are detailed in Section VI. Numerical results are showcased in Section VII. Finally, Section VIII provides the concluding remarks for the paper.

Dynamic Batching. Some works have explored dynamic batching for systems such as data centers[18] and Spark Streaming[19]. Recently, there has been a notable increase in attention towards dynamic batching, particularly in the context of ML applications. While works like DBS[20] and Zeus[21] explore batching for ML training, they fall outside the scope of our focus. Our exclusive attention is on online services. For example, in applications such as smart healthcare monitoring, data generated by devices can be efficiently processed through cloud, edge, or hybrid computing solutions to ensure timely responses[22, 23, 24]. Plenty of research has been conducted on dynamic batching for ML inference serving on GPU-based platforms[4, 3, 15, 14, 12, 8, 25, 26, 27]. There are also studies addressing the batching issue on a multi-core central processing unit (CPU)[10]. Nevertheless, progress in theoretical analysis remains very limited. For example, SERF[25] models inference serving as an M/D/c queue. However, it does not explicitly account for batching in the model. Another work, BATCH[12], characterizes request arrivals as a Poisson process or a Markov-modulated Poisson process with two phases (MMPP(2))[28], allowing for the estimation of the number of requests that arrive before a timeout. However, this analysis overlooks the cases where requests arrive during processing times. In contrast, the author of [29] presents a closed-form queueing analysis of dynamic batching under the greedy batching policy. Nonetheless, this policy is suboptimal as it does not leverage the potential benefits of larger batches. Additionally, the analysis in [29] assumes an infinite batching capacity, which is generally impractical.

Parallel Batch Processing. Parallel batch processing, or parallel batching, is a classical issue in operational research that finds applications in numerous fields such as manufacturing, transportation, and healthcare[30, 31, 32]. However, the batch processing problem studied in this paper exhibits two distinctive deviations from classical scenarios in the existing literature. Firstly, unlike the ideal parallelism with batch-size independent service times[33, 34], the batch processing time can increase with the batch size[35, 36]. Secondly, the energy efficiency can directly benefit from an increased batch size, rather than remaining unchanged. For instance, as illustrated in Fig. 2LABEL:sub@fig:latency and Fig. 2LABEL:sub@fig:energy, which are based on the statistics from NIVIDIA[7], the processing time and energy consumption for batch ML inference services appear to be affine functions of the batch size. Consequently, the average processing time and energy consumption per batch decrease as the batch size increases, leading to improvements in both computational and energy efficiency, as shown in Fig. 2. For GoogLeNet inference on a TESLA V100, using a batch size of 128 can achieve a speedup of 9.8 times compared to inference without batching (batch size of 1). Additionally, the energy efficiency is improved by 4.4 times.

Queueing analyses for batch servers with size-dependent service times have been conducted in the literature, but they only focus on certain structured policies that are suboptimal[37, 38, 39, 40]. Following the line of optimal control, existing research[33, 41, 32] primarily addresses problems where batch processing times are independent of the batch size. In [42], the author models the load-balancing problem in multiple batch service queues with size-dependent processing times as a Markov decision process and identifies it as an open problem. In fact, optimal batching in one such batch service queue, as the simplest sub-problem of [42], still remains unsolved.

SMDP. Continuous-time Markov decision processes (CTMDPs) and SMDPs are the common formulations for sequential decision-making in continuous-time systems[43]. Given that the processing times of computation tasks often exhibit limited randomness[12], deviating from the characteristics of an exponential distribution, SMDPs appear as more fitting choices for the studied problem. In this work, we need to address an infinite-horizon and infinite-state SMDP problem with unbounded costs, where the objective is expressed in a long-term average form. Analytical results regarding optimal policies for this SMDP are available only in specific cases[33]. An alternative approach is to utilize iteration-based numerical methods, which require finite state approximation due to the intractable infinite state space. In prior research, the authors in [44] demonstrated the convergence of several finite state approximation algorithms for average cost SMDP models, but only with bounded costs. In [45], proofs were provided for the convergence of finite state approximation algorithms for models with unbounded costs, but with a discounted objective. Nevertheless, none of the existing finite state approximation algorithms has been proven effective for the considered SMDP problem.

In summary, the lack of theoretical guarantees in existing dynamic batching schemes necessitates a rigorous formulation. The extracted system model distinguishes from classical batch service queues in its size-dependent service times, a feature that remains unexplored in the literature. The design of dynamic batching can be formulated into an SMDP problem with an infinite state space. Regrettably, none of the existing finite state approximation algorithms has been proven effective in addressing the studied SMDP, highlighting the need for the development of novel methods.

In our prior work[1], we proposed an SMDP-based dynamic batching scheme for ML inference serving on GPU-based platforms. This paper extends the research to encompass general online computing service scenarios. The batch service time and energy consumption take more general forms with respect to the batch size, unlike the deterministic and linear functions considered in [1]. Moreover, additional insights are gained through both theoretical induction and extensive numerical results.

We consider a single server with a single parallel computing processor in the continuous-time setting. Batch processing is implemented on the same type of computing requests, and it cannot be interrupted once the processing is started. The system is modeled as a single service queue, where computing requests (tasks) are assumed to arrive according to a Poisson process with an arrival rate of $\lambda$. The requests awaiting processing are stored in a buffer, which is assumed to have an infinite capacity. This assumption of an infinite buffer is based on the fact that the storage capacity of a computing server, which functions as the buffer in the queuing model, is significantly larger than the memory size. The total number of requests in the buffer as well as being processed at time $t\geq 0$ is denoted by $s(t)\in\{0,1,2,\dots\}$.

Let $b\in\mathcal{B}\triangleq\{x\,|\,x\in\mathbb{N}_{+},B_{\min}\leq x\leq B_{\max}\}$ denote the batch size, where $B_{\min},B_{\max}\in\mathbb{N}_{+}$ and $B_{\min}\leq B_{\max}$. $B_{\max}$ (or $B_{\min}$) is the maximum (or minimum) batch size allowed by the system. The cumulative distribution function (CDF) of the processing (service) time for a batch of size $b$ is $G_{b}(x)\;(x\geq 0)$ with mean $1/\mu^{[b]}$. Assume that for every $b\in\mathcal{B}$, $G_{b}(x)$ has a finite second moment. Moreover, the mean processing time should be finite and larger than zero, i.e., $G_{b}(0)<1$ and $1/\mu^{[b]}<\infty$. Let $l\;:\;\mathcal{B}\rightarrow\mathbb{R}$ represent the function of the mean batch processing time with respect to the batch size, where for $\forall b\in\mathcal{B}$, $0<l(b)<\infty$ and ${l(b)}=\frac{1}{\mu^{[b]}}$. The time required for processing a larger batch should be no less than that for a smaller batch. Therefore, we assume that $l(b)$ is monotonically non-decreasing in terms of $b$. Let $\tilde{g}_{b}(s)$ denote the Laplace transform of the service time distribution $G_{b}(x)$, defined by

with $-\tilde{g}^{\prime}_{b}(0)=1/\mu^{[b]}$.

When operating with a batch size of $b$, define the batch processing computational efficiency, or batch service rate, as the average number of requests processed per unit of time, denoted as $\theta(b)\triangleq\frac{b}{l(b)}$. Since parallelism can enhance computational efficiency, it is assumed that $\theta(b)$ is monotonically non-decreasing with the batch size $b$. Noting that $l(b)$ is also a non-decreasing function with respect to $b$, it follows that the mean batch processing time $l(b)$ should exhibit linear or sublinear growth as $b$ increases, if $l(b)$ is dependent on $b$.

Thus, the maximum service rate is $\theta(B_{\max})=\frac{B_{\max}}{l(B_{\max})}={B_{\max}}\mu^{[B_{\max}]}$. Let $\rho=\lambda/(B_{\max}\mu^{[B_{\max}]})$ denote the ratio of the arrival rate over the maximum service rate. Assume that $\lambda l(B_{\max})<B_{\max}$, or equivalently $\lambda<B_{\max}\mu^{[B_{\max}]}$, then $\rho$ satisfies $\rho\in(0,1)$, which is a necessary condition for the system stability.

The energy consumption of processing a batch of $b$ requests is denoted by ${\zeta}(b)$, $\zeta\;:\;\mathcal{B}\rightarrow\mathbb{R}$. Let $\eta(b)=\frac{b}{{\zeta}(b)}$ denote the batch processing energy efficiency, which is defined as the average number of requests served with one unit of energy consumption. Given the potential for parallelism to improve energy efficiency, we assume that $\eta(b)$ is monotonically non-decreasing with the batch size $b$. Similarly, $\zeta(b)$, the function of the batch processing energy consumption, should exhibit linear or sublinear growth as $b$ increases, if $\zeta(b)$ is not independent of $b$.

Given the server configurations and the specific computing task, the parameters $B_{\max}$ and $B_{\min}$ can be profiled and determined. Additionally, the exact forms of $G_{b}(\cdot),l(b)$, and ${\zeta}(b)$ are established by fitting the latency and energy consumption statistics obtained from prior profiling[12, 29].

We consider two main factors in the objective: One is the request response time (or latency), which includes both waiting and processing time, as the performance metric. The other is the power consumption of the server, as the running cost metric. Our objective is to minimize the weighted sum of average request response time, denoted by $\overline{W}$, and average power consumption, denoted by $\overline{P}$:

where $w_{1}>0$ and $w_{2}\geq 0$ are the weights.

The serving process consists of sequential service rounds. Define $t_{i}\;(t_{i}\geq 0,\;i=1,2,\ldots)$ as the start time of the $i$th service round and $b(t_{i})\in\mathcal{B}$ as the batch size in the $i$th service round. Let $N(t)\in\mathbb{N}$ denote the total number of service rounds until time $t\geq 0$. The objective can then be expressed as

Note that here the average request response time $\overline{W}$ is equivalently transformed to the average queue length $\overline{L}$ through Little’s Law $\overline{L}=\lambda\overline{W}$[46], where $\overline{L}=\limsup\limits_{T\rightarrow\infty}\frac{1}{T}\int_{0}^{T}s(t)\mathrm{d}t$.

The decision-making process for batching in the continuous-time system, as introduced in Section III, naturally lends itself to the formulation as an SMDP[43]. In SMDP, we are only concerned with the states at decision epochs, upon which the decisions are made. Let $\mathcal{T}=[0,\infty)$ be the timeline of the SMDP model. The decision epochs are set as the moments when either the server completes a batch of service, or a request arrives while the server is idle. The $m$th ($m=0,1,2,\ldots$) decision epoch is denoted as $t_{m}\in\mathcal{T}$. Let the state be the number of requests in the system. The state at the $m$th decision epoch is denoted by $s_{m}$, taking values from the state space $\mathcal{S}\triangleq\{0,1,2,\ldots\}$.

At each epoch $m$, the server takes an action $a_{m}$ from the action space $\mathcal{A}\triangleq\{0\}\cup\mathcal{B}$. The action $a_{m}$ is the size of batch to be processed. Note that $a_{m}=0$ means that no requests are served in the $m$th epoch. Let $\mathcal{A}_{s}\subseteq\mathcal{A}$ be the set of feasible actions for a given state $s$. The number of requests to be batched should be no more than the available requests, which means $\mathcal{A}_{s}\triangleq\{0,1,2,\ldots,s\}\cap\mathcal{A}$, and thus $\mathcal{A}_{s}=\{0,1,2,\ldots,s\}\cap\mathcal{B}$.

The state transition is associated with the current state and action. Let $m(j|s,a)$ denote the probability that the semi-Markov decision process occupies state $j$ at the next decision epoch when action $a$ is chosen at the current state $s$. Let $p_{k}^{[b]}$ denote the probability that $k$ requests arrive during the period of processing a batch of $b\in\mathcal{B}$ requests. With the assumption that the arrival of requests follows a Poisson process, we have

A useful method to generate the probabilities $p_{k}^{[b]}$ is by using $\tilde{g}_{b}(s)$, which is the Laplace transform of the service time distribution. Denote the probability generating function (PGF) that corresponds to $p_{k}^{[b]}$ as $A^{[b]}(z)=\sum_{k=0}^{\infty}p_{k}^{[b]}z^{k}$, which can be simplified to

Then the required probabilities $p_{k}^{[b]}$ can be computed by

The transition probability for $\forall j,s\in\mathcal{S},\forall a\in\mathcal{A}_{s}$ is expressed as

Let a random variable $\gamma_{m}$ denote the sojourn time between the $m$th and the $(m+1)$th epoch. The random variables $\gamma_{m},m\in\mathbb{N}$, are conditionally independent given the state $s_{m}$ and action $a_{m}$. Let $\Gamma_{s,a}(x)\;(x\geq 0)$ denote the CDF of $\gamma_{m}$ when action $a\in\mathcal{A}_{s}$ is chosen at the state $s$, given by

Define $y(s,a)=\mathbb{E}[\gamma|s,a]$ as the expected sojourn time until the next decision epoch, given by

Costs are incurred for serving the requests as well as holding them. The cost of serving a batch of $n\in\mathbb{N}$ requests is denoted by $u(n)$, and the cost of holding $n$ requests in the system per unit time is denoted by $v(n)$. Let $c(s,a)$ denote the expected cost until the next decision epoch when action $a\in\mathcal{A}_{s}$ is taken in state $s$. We have $c(s,0)=u(0)+v(s)y(s,0)$, and for $a\neq 0$,

The cost functions corresponding to the objective in (3) are $u(n)=w_{2}{\zeta}(n)\;(n>0)$, $u(0)=0$ and $v(n)=\frac{w_{1}}{\lambda}n$. This leads to a detailed description of $c(s,a)$, which is

where $G_{a}$ denotes a generic random variable that follows the CDF $G_{a}(x)$, and $\mathbb{E}[G_{a}^{2}]=\tilde{g}^{\prime\prime}_{b}(0)$.

We can also generalize $c(s,a)$ in the form

where $d(s,a)$, referred to as the cost rate, represents the holding cost averaged over the sojourn time:

The formulated SMDP can be fully described by the set of objects $\mathcal{P}\triangleq\{\mathcal{T},\mathcal{S},\mathcal{A}_{s},m(\cdot|s,a),\Gamma_{s,a}(\cdot),c(s,a)\}$, and we will use the symbol $\mathcal{P}$ to represent this SMDP model in the subsequent text.

Let $\mathcal{B}(\mathcal{A})$ denote the set of probability distributions on Borel subsets of $\mathcal{A}$. A Markovian decision rule $\bm{d}^{(m)}:\mathcal{S}\rightarrow\mathcal{B}(\mathcal{A})$ specifies the probabilities of taking each action at epoch $m$. It is called Markovian since it relies solely on the current state $s_{m}$ for its decision-making. The decision rule is deterministic if it selects an action with probability 1. A policy $\bm{\pi}=\{\bm{d}^{(0)},\bm{d}^{(1)},\bm{d}^{(2)},\ldots\}$ is a sequence of decision rules. Furthermore, $\bm{\pi}$ is called stationary if $\bm{d}^{(m)}=\bm{d},\forall m\in\mathbb{N}$.

Our goal is to find a policy that minimizes the long term average expected cost $g^{\bm{\pi}}(s_{0})$, given $s_{0}\in\mathcal{S}$ at $t=0$, which is

The objective (14) and the SMDP model $\mathcal{P}$ together constitute the SMDP problem. The objective focuses on the long-term average, and for every history-dependent policy, there exists an equivalent Markovian policy with the same objective value (referred to Theorem 8.1.2 in [43]). This implies that only Markovian policies need to be considered. Moreover, in this paper, we restrict our consideration to stationary deterministic policies. A stationary deterministic policy $\pi:\mathcal{S}\rightarrow\mathcal{A}$ is a function that maps the state space $\mathcal{S}$ to the action space $\mathcal{A}$. This type of policy is concise and clear, helping to reduce the solution space. For instance, the static batching policy and the greedy batching policy are both stationary deterministic policies.

The greedy batching policy is a representative dynamic batching policy, defined as follows:

The considered model $\mathcal{P}$ is an infinite state SMDP with non-negative, unbounded costs and finite action sets. The existence of an optimal stationary deterministic policy for such a model requires further discussions.

The equations corresponding to the optimal stationary deterministic policies are provided as follows.

Through Proposition 1, the existence of an optimal stationary deterministic policy for this problem is ensured. We try to acquire such an optimal policy by solving the optimality equations (17) in the next section.

Iteration-based algorithms, such as value iteration and policy iteration, are widely used to solve the optimality equations for most common discrete-time, finite-state and discounted Markov decision process (MDP) problems. However, the standard procedure is not readily applied to our problem, which is a continuous-time SMDP with an infinite state space and a long-term average objective.

We solve this problem in three steps. In Section V-A, we approximate the infinite state space by a finite state space through “tail” state aggregation. In Section V-B, we transform the finite state SMDP to an equivalent discrete time MDP. Finally in Section V-C, we use the relative value iteration (RVI) algorithm to solve the average-cost MDP problem.

In the previous section, we introduced a general approach to address the formulated SMDP problem. However, in existing literature, specific properties of optimal policies have been discussed for certain special cases. Research studies[33, 47] have shown that in scenarios with size-independent service times, optimal policies exhibit a threshold-based structure known as a Q-policy or control limit policy, given certain assumptions.

The concept of the control limit policy is explained as follows:

The necessary assumptions for the specific cases discussed in this section are listed as follows:

The conclusion regarding the structure of the optimal policy for the specific scenario is as follows.