DeepRT: A Soft Real Time Scheduler for Computer Vision Applications on the Edge

We mainly focus on the latency and throughput performance of CNN inference. When batching is enabled, the latency $l$ of performing CNN inference upon some input data consists of two parts:

where $l_{e}$ is the real execution time of performing CNN inference upon input data on GPU, and $l_{q}$ is the queuing time spent by some input data that arrive early in waiting for input data that arrive late. $l_{q}$ hinges on the specific design of the inference system, so throughput this section we only measure $l_{e}$ and call it execution time to avoid ambiguity.

All the performance measurements are carried out using a mature cloud and edge inference solution, Triton Inference Server (triton, ), developed by NVIDIA. The hardware setting is introduced in Section 5. The way we measure execution time and throughput is as follows. We use Triton’s perf_analyzer to send requests to Triton server and record median execution time and throughput. Each time we send one or several requests to the server for inference, depending on the concurrency number, and each request may contain one image or a batch of multiple images; when inference result is received from the server, we immediately send out another request(s). This process is repeated over a fixed time interval, $20$ seconds. We use images sized $3\times 224\times 224$ (RGB channels $\times$ height $\times$ width), which is the default image size in Triton. We choose $6$ widely used deep learning models – ResNet50, ResNet101, ResNet152, VGG16, VGG19, and Inception-v3. They belong to $3$ types, ResNet (he2016deep, ), VGGNet (simonyan2014very, ), and Inception-v3 (szegedy2016rethinking, ). In our setting, ResNet and VGGNet models are built upon the ONNX framework (bai2019, ), and Inception-v3 is of the GraphDef format (tensorflow, ), which is a tool used by Tensorflow to represent models. Our setting covers different types of models, different model sizes in each type, and different frameworks to show the universality of our conclusions.

We first study the performance characteristics when executing multiple models concurrently on GPU. To be more specific, there are multiple model instances loaded on GPU, and these models process image frames received from clients all at the same time; we study how concurrency affects the execution of each model. This analysis can be further divided into two parts: concurrent execution of multiple instances of the same model on GPU, and concurrent execution of different models.

Concurrent execution of the same model. When different clients request to process their data with the same model, except for batching the data and processing them as a whole in one model instance, another common approach taken by Triton and some platforms is to replicate the same model on GPU to get several model instances, and to process each request with one of the model instances concurrently. In this part we study the performance of running different number of duplicate instances of the same model concurrently (see Figure 2(a) and 2(b)). For Inception we show its performance up to a concurrency number of $6$, as more concurrent instances overload the system. We have also recorded the variance of different executions, but the variances are too small to display.

From Figure 2(a) we can see the linear relationship between execution time and the number of concurrent instances. We think this is due to how CUDA schedules multiple programs. When multiple programs run concurrently on GPU, CUDA schedules their warps (a warp contains multiple CUDA threads which can be executed in parallel) with a time sliced scheduler (wang2017quality, ). Warps from different contexts cannot be executed simultaneously and some warps have to wait in queue for their time slice share. When we execute multiple instances of the same model, since all instances have exactly the same warps but the warps are in different contexts, the execution time of each warp grows linearly with the concurrency number. Figure 2(b) shows how throughput changes with the concurrency number. We can see that if we increase concurrency number to $2$, there is a slight increase in throughput, but afterwards the throughput stays at a stable value.

A crucial conclusion we draw in this part is that, although increasing concurrency to a certain level can improve inference throughput by a small margin, execution time increases linearly as concurrency number grows. Let’s look at an example to see how this observation affects scheduling algorithm design. Imagine there are two images to be processed with a specified model, say, ResNet50. Processing the two images one by one inside one single model instance instead of processing them concurrently in two model instances, achieves a slightly lower throughput but the average latency is reduced by $25\%$, since the first image is finished early and its execution time is not affected by the concurrent execution of the second image. Therefore, the latency of the first image is reduced by half compared to running concurrently, while the latency of the second image remains approximately the same as when running concurrently.

Concurrent execution of different models. A common practice of Triton Inference server and other solutions when different clients send requests to process data with different models is to execute the models simultaneously. This part analyzes the performance characteristics when different models run concurrently on GPU. Among the $6$ models introduced previously, in each experimental run we choose two models and execute them concurrently¹¹1In this analysis, we study how the execution of a model is affected by another model instance. We don’t analyze the situations where there are three or more instances since the current setting already shows the complicated interference between model executions.. The other experimental setups are the same as in the previous part.

In Table 1, we show the median execution time and throughput when executing a model specified by the leftmost column concurrently with another model specified by the uppermost row. As a comparison, we use the columns marked by “-” to show the performance when a model is executed alone. The data show that when a model M is executed along with different other models, its execution time and throughput performance vary greatly²²2There is an observable trend, however, that when a model is executed concurrently with models of the same type (e.g., a ResNet101 model and a ResNet152 model), the performance tend to be similar, but performance discrepancy still exists. This trend is a piece of evidence for our following hypothesis. The kernels of models belonging to the same type have similar sizes, thus they cause similar amount of interference.. Explaining such an observation requires a detailed study of how internal scheduling happens at a low level on GPU, which is very difficult since GPU drivers are not open sourced. Our hypothesis is that different kernel (a CUDA function) sizes of different models cause different slowdowns. CUDA uses a time sliced scheduler to schedule programs from different contexts (corresponding to different models) and the kernels are the smallest scheduling unit. They get scheduled non-preemptively on GPU. Different models are composed of distinct numbers and types of kernels; when a model M is executed concurrently with model N and the kernels of M are larger in sizes but smaller in quantity, model M will have more GPU time share and thus exhibit higher throughput and smaller execution time.

The observation we make from this part, is that concurrently executing multiple instances of different models results in complex interference between models. If we want to design a real time system, we need to figure out the interference in any subset of all admitted models in order to obtain their worst-case execution times. When there are $z$ models, there are ${z\choose 2}$ combinations of models to profile if we only execute two model instances concurrently. In fact, if we allow any number of concurrent model instances, we need to profile $\sum_{k=1}^{z}{z\choose k}=2^{z}$ combinations of models, which is practically impossible. However, if we manage to execute the different model instances sequentially instead of concurrently, there won’t be any interference between the sequential model executions.

A well-known conclusion about GPU-based deep learning model training and inference is that batch processing can boost their throughput. In this part we show the performance characteristics when we do inference in batches. The experimental setup of this part is the same as in the previous two parts, except that each time we only execute one model instance instead of concurrently executing two or more instances. The comparison between different batch sizes is shown in Figure 2(c) and Figure 2(d). As expected, batching increases throughput with a cost of higher execution time.

Since concurrent execution of multiple instances and batching inputs into larger tensors can both process multiple requests simultaneously, in this part we compare how these two approaches affect system performance. On each of the aforementioned $6$ models, we fix the number of concurrently processed requests (concurrent model instances $\times$ batch size) and see how the execution time and throughput vary. The result is shown in Figure 2(e) and Figure 2(f), where we compare concurrency - batch combinations of $4\times 1$, $2\times 2$, and $1\times 4$. We can see that batch processing should be favored over concurrent model instance execution in terms of increasing system throughput and reducing request execution time.

In summary, We make three observations:

• •

  Executing two or more models simultaneously does not notably improve system throughput, and it increases latency of each request due to increased execution time.

• •

  When executing multiple instances from different models on the same GPU, there exists interference between concurrent executions, making it cumbersome to profile or estimate the worst-case execution time, hindering the design of a real time system.

• •

  Processing input data in batches increases system throughput, much more than concurrent model execution does, with a sacrifice of increased request execution time.

Two key takeaways from these observations are: (1) Concurrently executing multiple model instances does not provide high value to our goal of guaranteeing maximum latency while maintaining high throughput; instead, the interference it introduces makes designing a real time system difficult. So we would like to execute model instances sequentially instead of concurrently to avoid interference. (2) Batching increases throughput , but we need to make sure that the increased latency (increased execution time plus queuing time when an image frame waits for other frames belonging to the same batch) does not cause deadline misses.

Data Model. This paper assumes that the GPU resources on the edge are shared by multiple users. Each user sends a request to DeepRT, and each request corresponds to a video, the frames of which need to be processed by a user-specified CNN model. Different users may request different CNN models. When a request comes, if DeepRT confirms that the new request and all the existing requests in the system can be scheduled, DeepRT will admit the new request³³3In this paper, we use admitted requests and requests interchangeably, and we call requests that wait to be verified pending requests.. The video frames of an admitted request arrive at DeepRT in an online manner, and the interval between two frames is determined by a frame rate, or period⁴⁴4We only consider the scenario where all frames are raw, uncompressed video frames. Decompression of compressed frames should be coordinated by a CPU scheduler and is not the scope of DeepRT. Also, we assume that the frames in a video arrive one by one, but DeepRT can also handle the situation where a chunk of frames arrive at a time, used by video coding techniques like H.264. . Each request also has a relative deadline, indicating the desired maximum latency of performing inference on each of the video frames. The relative deadline of a request is not necessarily equal to its period. Different videos may have different input shapes, which equal to the number of channels (most videos use RGB channels so this number is $3$) $\times$ frame height $\times$ frame width.

Execution Model. The two key takeaways from Section 2 drive us to avoid concurrent model execution and leverage batching to increase system throughput. We propose the following execution model for DeepRT. When a frame from a request arrives, DeepRT does not process this frame immediately. Instead, DeepRT waits for frames of the same shape requiring the same CNN model, batches these frames together, and processes the batched data on GPU. In fact, all frames that are of the same shape and require the same model can be batched together regardless of which request they belong to, while frames with different shape or require different models should not be batched because GPU cannot execute in parallel the same kernels for these frames. Since the batch of frames from multiple requests is what the GPU actually executes, we call the job of processing a batch of frames a job instance. DeepRT processes multiple job instances one at a time instead of concurrently. In DeepRT, the frames with the same shape requiring the same model are said to be of the same category. Likewise, we call the requests containing the same category of frames to be of the same category. Executing job instances sequentially instead of concurrently makes Equation 1 become $l=l_{q_{b}}+l_{q_{j}}+l_{e}$, where $l_{q_{b}}$ denotes frame queuing time waiting for other frames in the same batch, and $l_{q_{j}}$ denotes queuing time of a job instances. In order to provide a guarantee on $l$, two questions need to be answered: (a) How many frames should be in each batched job instance, or how does DeepRT determine which frames should be put into the same batch? (b) GPU operations are non-preemptive, in the sense that operations already launched on GPU cannot be preempted. How to schedule the non-preemptive batched job instances?

This subsection answers the first question. There are already research efforts (crankshaw2017clipper, )(fang2019serving, ) or industrial solutions (triton, ) which process batches containing fixed or adaptive number of images from multiple requests concurrently. While these approaches are capable of increasing inference throughput and reducing latency, their approaches don’t support real time services, since they don’t have a deadline-centric soft real time design. A real time system with batching enabled has to make sure the queuing time of a frame arriving early waiting for frames arriving late does not cause a deadline miss of the early frame.

In order to guarantee real time processing of frames, instead of answering the question “what is the right number of frames inside a batch”, we try to answer another question “when should a frame be put inside a batch”. We propose the DisBatcher approach to answer this question. DisBatcher divides time into contiguous time intervals with the same length, called time windows. The end time of a window coincides with the start time of next time window. Frames that fall into the same time window should be batched together into one job instance regardless of which request the frames belong to, as long as the frames are of the same category, and the batching action happens at the end of a time window. DisBatcher sets the relative deadline of a job instance to be the length of one time window, which means that a job instance generated at the end of one time window, should be completed before the end of the next time window. For simplicity, We call the end time of a time window, which is also the start time of the next time window, a time window joint. As we previously mentioned, frames of different categories should not be batched together. DisBatcher creates different and independent time windows for each category of frames. With this time window approach, we transform the processing of one category of requests into a series of non-preemptive job instances, or a task instance. A task instance’s relative deadline is equal to its period, which is the length of the time window used to generate the task instance. This task instance is not strictly a periodic task, since each job instance has different execution times due to different number of frames inside the batches, and some job instances have $0$ as their execution times (no frames in this batch). An example of using time windows to batch frames is shown in Figure 3.

How to set appropriate time window lengths in order to ensure that for all frames their latencies do not exceed their deadlines remains a challenge. DisBatcher utilizes a deadline-centric design. For one category of requests, DisBatcher sets the time window length to be half of the smallest relative deadline among all requests, regardless of other parameters such as request arrival periods. In fact, we have the following theorem.

With the DisBatcher approach, we transform the problem of how to batch and schedule frames in real time to the problem of scheduling a set of non-preemptive periodic task instances. The new task instances are not strictly the traditional non-preemptive periodic tasks, since job instances of the same task instance have variant execution times.

The non-preemptive periodic tasks whose jobs have variant execution times are called non-preemptive multiframe tasks (moyo2010schedulability, )(chakraborty2002schedulability, )(baruah2010preemptive, ). Non-preemptive periodic tasks with fixed execution times are a special case of non-preemptive multiframe tasks. There are two types of scheduling algorithms for non-preemptive multiframe tasks — algorithms that do not permit the processor to be idle when there are jobs that have been released but have not completed execution (non-idling), and algorithms that allow idle times (idling). For DeepRT, we choose non-idling Earliest Deadline First (EDF) as its scheduling algorithm, the reasons of which are twofold.

First, the inserted idle times lead to a waste of precious GPU computation power, sacrificing the total throughput of the system. Second, although some idling algorithms can perform better than non-idling algorithms in terms of the number of schedulable tasks (ekelin2006clairvoyant, ), they often depend on complicated heuristics, and their performance gains are only demonstrated through simulations. In fact, finding an optimal schedule in idling non-preemptive context is an NP-hard problem (gary1979computers, ). On the contrary, EDF has been shown to be an optimal scheduling algorithm for non-preemptive multiframe tasks in non-idling context (george1995optimality, )(baruah2006schedulability, )(chakraborty2002schedulability, ). In Section 4, we will discuss how DeepRT performs admission control in order to make sure that all job instances are schedulable.

It is worth mentioning that the scheduling scheme of DeepRT makes it easy to support non-real time requests. We treat non-real time requests in a similar way with real time requests. We also use DisBatcher to batch non-real time requests to transform the frames into task instances. However, we don’t batch non-real time requests together with real time requests for performance isolation; we use a large time window for non-real time requests to make sure they obtain a low deadline priority, and we impose them a large arrival period regardless of their true arrival periods so that they don’t aggregate to large batches and cause too much priority inversion.

Our Performance Profiler works offline. For each deep learning model that we want to execute at the edge server, for each frame shape that DeepRT permits as legitimate shape, and for different batch sizes, the Performance Profiler executes each batch of frames on GPU multiple times, and records the execution time of each run. For each setting, we obtain a list of running times and take the worst-case running time⁵⁵5In practice we take 99 percentile running time to filter out outliers.. In this way, we create a lookup table containing the execution times of different sized batches of frames with different shapes processed by different deep learning models. Whenever a new request comes to the system, we look up this table, find the corresponding model and shape, and feed the results to Admission Control Module to make admission decisions.

When a new request arrives at DeepRT, it is first routed to the Admission Control Module. Since we target building a soft real time system, DeepRT is selective with the requests in case too many requests cause serious deadline misses. The Admission Control Module decides whether a pending request is admitted to DeepRT. As we discussed in Section 3, DeepRT uses DisBatcher to transform frames into task instances, which are intrinstically non-preemptive multiframe tasks. Therefore, performing admission control for DisBatcher based DeepRT is equivalent to performing admission control for non-preemptive multiframe tasks.

Some past works propose to perform admission control for EDF under the non-preemptive multiframe workload scenario using demand-bound functions (baruah2010non, )(baruah2010preemptive, )(baruah2006schedulability, )(chakraborty2002schedulability, ). A demand bound function represents the maximum execution demand of a task set in any time interval of a given length. Then this approach compares the demand bound with available resources to decide whether a task set is schedulable. This approach suffers from pseudo-polynomial complexity and inaccuracy of their approximate algorithms in calculating the demand bound functions. Another approach performs simulation based feasibility analysis (moyo2010schedulability, ). Basically this approach represents time with a clock variable. When the clock reaches the arrival time of a job, the job is released to a deadline queue. It simulates the execution of the job by simply incrementing the clock by the job’s worst-case execution time. Then it compares the virtual completion time of the job, which is the current value of the clock variable, with the job’s deadline, to determine whether there is a deadline miss. Since different tasks may have different initial release times, this approach uses a tree to represent all possible execution sequences, making its time complexity non polynomial.

Since the goal of DeepRT is to provide real time inference while maintaining high throughput, the Admission Control Module should admit as many requests as possible but not too many requests which overload the system. Therefore, it has to perform an exact analysis of schedulability. DeepRT adopts the simulation based approach as it is an exact analysis. The time complexity of this approach can be greatly reduced in DeepRT to linear with respect to the number of frames. The reason is that in DeepRT the requests for inference all have specific release times, because users’ video frames occur at specific times instead of at arbitrary times, and the release times are communicated with the Admission Control Module. Since we also know the time when every time window starts, we can know the release times of all job instances. In this way, we know exactly when and in what order each job instance “arrives” at the GPU, so we can build an exact execution schedule in linear time instead of building a tree of execution sequences. Note that in building such an execution schedule, DeepRT requires the execution times of different job instances obtained in Section 4.1. The assumption for the exact schedulability analysis is thus accurate job instance execution time profiling.

To further improve the Admission Control Module by reducing its complexity, before we run the simulation based schedulability test, we first use a utilization based test to filter out obviously infeasible requests. The goal of the utilization based Phase 1 test is to reject a pending request as fast as possible which will obviously cause deadline misses if accepted. The simulation based Phase 2 test is an exact test, which refines the results from Phase 1 test and ultimately decides whether the new request gets admitted.

Phase 1. In Phase 1, DeepRT uses the utilization of task instances to reject a pending request that will obviously cause deadline violations. We calculate and evaluate the utilization of task instances because task instances are actually executed by the GPU . We define the average utilization of a task instance $s$ to be

where $N_{s}$ denotes the total number of job instances in task instance $s$, and $P_{s}$ is the period of $s$ which equals to the time window length used to generate $s$. Within each period there is at most one job instance. $E_{i}$ denotes the execution time of a job instance $i$ belonging to $s$. We further define the average utilization of a task instance set $\Sigma$ to be $U=\sum_{s\in\Sigma}U_{s}$. The task instance set comprises task instances generated from all existing requests and the new pending request. Naturally in order not to overload the system, we should have $U\leq 1$, or $\sum_{s\in\Sigma}\frac{\sum_{i=1}^{N_{s}}E_{i}}{N_{s}P_{s}}\leq 1$. The complexity of calculating $\sum_{i=1}^{N_{s}}E_{i}$ is linear with respect to $N_{s}$, since we need to estimate the number of frames that fall inside each time window, and look up the profiling execution time table to decide $E_{i}$.

In order to boost the speed of Phase 1, we make an approximation when calculating $U_{s}$. As the complexity of calculating $\sum_{i=1}^{N_{s}}E_{i}$ comes from the fact that we don’t know how many frames fall into each batch, we use an average number of frames over all time windows to estimate the exact number of frames in each time window. We denote the set of requests of the same category with $I^{g}$. The average number of frames inside a time window of a request $I^{g}_{m}\in I^{g}$ is $\frac{W_{g}}{p^{g}_{m}}$, where $W_{g}$ is the period of the corresponding time window, and $p^{g}_{m}$ is the period of request $I^{g}_{m}$. Therefore, for all the requests of category $g$, the average number of frames in one time window, denoted by $n_{g}$, can be represented by $n_{g}=\lfloor\sum_{I^{g}_{m}\in I^{g}}\frac{W_{g}}{p^{g}_{m}}\rfloor.$ We look up the execution time $E^{n_{g}}$of a batch of $n_{g}$ frames from the table and we can get an estimation of $U_{s}$:

Please note that Phase 1 accepts more pending requests than DeepRT can handle while not rejecting feasible requests by underestimating total workload. The reasons of this underestimation are twofold. First, it’s clear that the sufficient necessary condition for a set of periodic preemptive tasks to be schedulable is that the total utilization is no larger than 1; but this is only a necessary condition for preemptive multiframe tasks (mok1997multiframe, ), let alone non-preemptive multiframe tasks in our scenario. Second, we use an estimated average utilization of a task instance which doesn’t consider peak utilization. Also, we use a floor operator in calculating $n_{g}$ above. In this way, Phase 1 admission control can give prompt responses to some clients and reduces some workload for Phase 2.

Phase 2. While DeepRT’s Phase 1 test admits requests generously, the Phase 2 test does an exact schedulability analysis to control admission to the system. The Phase 2 test consists of three sequential steps — system state recording, pseudo job instances generation, and an EDF imitator algorithm.

In the first step, system state recording, the Admission Control Module captures the current system state, which includes four parts: (1) the number of frames of each category that have already arrived at DeepRT and wait to be batched by the DisBatcher, (2) the already batched job instances in the deadline queue waiting to be processed by GPU, (3) the periods of all time windows, and (4) the period and number of remaining frames of each request. Essentially these four parts describe the existing system workload, and how to batch image frames to job instances.

Once the Admission Control Module learns about the current system state, it proceeds to the second step, where it simulates the process of DisBatcher and generates pseudo job instances from all the requests, including the pending request being tested. It implements a virtual representation of the DisBatcher mechanism introduced in Section 3, where the time and workload are both simulated. For each request, the Admission Control Module estimates the arrival time of each frame using its period, and compares the arrival time with the starting time and end time of the time windows to see which window each frame falls into. In this way the Admission Control Module is able to know the number of frames to be batched in each window. The Admission Control Module looks up the execution time of each batched job instance from the execution time table to get a list of virtual job instances. This list contains the “future” job instances from all task instances, and the job instances are ranked according to their release times, which is the time when a job instance is pushed to the deadline queue. This is done through simultaneously running the aforementioned DisBatcher simulator over all categories of requests and appending the job instance with the smallest release time to the list each time.

With the system current state captured from Step 1 and the future virtual job instances obtained from Step 2, the Admission Control Module moves to step three, where it uses an EDF imitator algorithm to determine whether these job instances can be scheduled by EDF. The EDF imitator algorithm is shown in Algorithm $1$. In this algorithm, $Q$ is a sorted deadline queue storing job instances, and $L$ is the list of job instances obtained from Step 2 which are released in the “future”. Note that $L$ is already ordered by release times. We use $R_{i}$ to denote the release time of a job $i$, $E_{i}$ to denote the execution time of a job $i$, $D_{i}$ to denote the relative deadline of $i$, and $L[1]$ to denote the $1^{st}$ element of list $L$. The complexity of this algorithm is $O(N)$, where $N$ is the total number of frames.

DisBatcher is the core component of DeepRT. It is responsible for transforming frames received from admitted user requests to job instances that are suitable to execute on GPU. It’s an implementation of the batching approach presented in Section 3.

Once a request is admitted by the Admission Control Module, the Admission Control Module sends request-related metadata including frame shape, requested model, period, and relative deadline to the DisBatcher. Then clients will directly send frames to DisBatcher. DisBatcher keeps track of all the admitted requests in DeepRT, together with their metadata. DisBatcher manages a frame queue for each category of requests. These queues store frames which arrive during their time windows and wait to be aggregated into batches of frames, or tensors. DisBatcher utilizes reccurent countdown timers to implement time windows. It keeps a timer for each category of requests with the timer’s countdown interval equal to the time window length. When a timer expires, DisBatcher batches all the frames in its corresponding queue to a tensor and immediately starts timer countdown again. Whenever a new request is admitted, DisBatcher updates the countdown interval of the corresponding timer if the new request’s relative deadline is smaller than the current smallest deadline.

The DisBatcher wraps a tensor inside a new job instance, the relative deadline of which is equal to the corresponding time window length, and it pushes this job instance onto an execution queue. A Worker on GPU subscribes to this execution queue and processes the job instances according to EDF. We implement the execution queue with a priority queue. The priority is determined by the job instances’ absolute deadlines (release time + relative deadline).

The Worker is the execution engine which actually processes the batched job instances with a requested model on GPU. It repeatedly consumes the execution queue mentioned above whenever there are job instances inside. It processes the job instances one after another. The Worker is also responsible for monitoring the performance of execution. It detects and records deadline misses. It also detects whether the execution time of a job instance is larger than the profiled worst-case execution time and report overruns to the Adaptation Module.

We employ an optimization technique to further reduce frame latency. Occasionally GPU is idle while there are frames which have already arrived but wait to be batched by DisBatcher. When DeepRT detect this situation, it batches the frames before their timer expires and immediately sends the batched job instance to GPU for processing. In this way, DeepRT can reduce the latency of these frames and meanwhile increase the utilization of GPU.

Since the hardware devices upon which DeepRT operates are commercial off-the-shelf computing devices that don’t have any hard deadline guarantees, DeepRT is only able to provide soft real time services. The response time of processing the same request under the same setting may vary between different runs, occasionally leading to job overruns. As the GPU executes jobs non-preemptively and we use EDF, one single job overrun can possibly cause unpredictable deadline misses of many other jobs in the system. When overruns happen, we need a method to “punish” the overrun job, and more importantly, to avoid deadline misses as much as possible.

In DeepRT, each job instance category has a penalty initialized to be $0$. When the Worker observes that the execution time of a job instance exceeds the profiled execution time, the Adaptation Module will increase the penalty of the job instance category by the excess part. Meanwhile, the Adaptation Module informs the DisBatcher to decrease the shape (resolution) of tensors belonging to that category. These tensors will not be batched with other tensors with the same smaller shape in order not to disturb job instances’ priorities. The Worker will record the execution time of the new job instance and subtract the saved execution time from penalty. When penalty becomes non positive, the Adaptation Module commands the DisBatcher to resume the original shape of the overrun job instance and sets its penalty back to $0$.

We use the DAVIS dataset (perazzi2016benchmark, ) as the workload. This dataset consists of video frames of 480p ($480\times 854$) and 1080p ($1080\times 1920$) resolution. We also downsample the video frames to various resolution formats in order to enrich the request data. On the desktop computer with an RTX 2080 card, we set the request video data to have 3 resolution formats: $1080\times 1920$, $480\times 854$, and $240\times 352$. On the Jetson TX2, as its computing power is smaller, we use frames of $360\times 640$, $240\times 352$, and $224\times 224$. All videos are colored videos and they all have the 3 RGB channels. It is worth mentioning that DeepRT is agnostic of video contents since different videos exhibit the same characteristics when being processed by classification models.

In this part, we compare the performance of DeepRT and the state-of-the-art CNN inference systems with respect to their abilities to meet the latency requirements specified by user applications.

Baseline. We compare DeepRT against these approaches which enable batching or adaptive batching to achieve low-latency high-throughput inference:

• •

  AIMD is an implementation of the dynamic batching scheme used by Clipper (crankshaw2017clipper, ) and MArk (zhang2019mark, ). As the name suggests, when inference latency does not exceed the latency objective, batch size increases additively. If latency objective is violated, a multiplicative reduction of batch size is performed.

• •

  BATCH is the scheme used by Triton Inference Server (triton, ). It performs batching over request data with a fixed batch size determined empirically. We set the batch sizes as small as possible to reduce the latency of each job.

• •

  BATCH-Delay is another scheme provided by Triton Inference Server. Apart from imposing a fixed number as the batch size, BATCH-Delay also imposes a time limit to each model. This scheme batches input data either when the number of frames in a batch reaches the configured batch size, or when the time limit is reached, whichever occurs first.

It is worth mentioning that all of these approaches process multiple requests concurrently under multitenancy situation. For BATCH and BATCH-Delay, we set the batch size as small as possible in order to reduce the latency of each job. However, sometimes a small batch size could drain GPU memory since there are too many jobs executed concurrently on GPU. If this happens, we increase the batch size until the memory problem is alleviated.

Request traces. Each time we run DeepRT or the aforementioned inference systems, we feed the system with multiple synthesized requests to perform inference on their data. The requests are independent of each other and they arrive at the system one at a time. We use tweets traces from Twitter (twitter, ) as a reference to determine the interval between the arrival of requests. Each request contains a video with a fixed number of frames, and each frame is released periodically according to its frame rate. In order to demonstrate the universality of DeepRT on various kinds of applications, we randomly set the period and relative deadline of the frames in a video. The period and relative deadline of the frames are sampled from a Gamma distribution independently. We select Gamma distribution to generate random period and relative deadline settings because the generated random numbers by Gamma distribution start from $0$, and it’s a common distribution in queuing theory. The shape parameter $k$ and the scale parameter $\theta$ of the Gamma distribution are set to $2$ and $5$, respectively. Then we scale the samples to appropriate values. For each request, we randomly choose an input shape and a model from the $6$ models listed in Section 2 plus Mobilenet-v2 (sandler2018mobilenetv2, ), and we limit the number of categories of requests.

On both the desktop computer and Jetson TX2, we synthesize $3$ traces of requests using the aforementioned approach. Each trace contains $20$ to $30$ requests. The periods and relative deadlines of all requests in the same trace are obtained from scaling the randomly sampled values with the same factor. The mean values of periods and relative deadlines of these traces are shown in Table 2.

In each run, we feed an inference system with one trace of requests, wait till all frames are processed, and record frame deadline misses as an indication of the system’s ability to perform real time inference. Since DeepRT’s Admission Control Module admits requests selectively while other approaches don’t have admission control measures, in order to guarantee the fairness of comparison, we record the accepted requests from DeepRT and feed these requests to other systems. Moreover, we disable DeepRT’s Adaptation Module which potentially reduces frame shapes.

Results on achieving real time inference. We first show the deadline miss rates of DeepRT and the other inference systems in Figure 4. To distinguish the traces on Jetson TX2 from the traces on desktop computer, we use “Trace x^′” to indicate it is for Jetson TX2. We can see that for all $6$ traces DeepRT shows the lowest deadline miss rates. When the mean values of period and relative deadline are $50ms$, the deadline miss rate of DeepRT is still $5\%$ while it handles $4$ concurrent requests and a total number of $10$ requests. DeepRT exhibits the lowest deadline miss rates because its design focuses on meeting requests’ deadlines and it considers special characteristics of GPU mentioned in Section 2. The results demonstrate DeepRT’s ability to perform soft real time inference services for multiple requests. Note that DeepRT does not completely avoid deadline misses due to job instance overruns. Interestingly, Clipper shows the highest deadline miss rates in all runs. We think the reason is that the AIMD based adaptive batching scheme assumes abundant resources and is more suitable for cloud-scale inference.

We also analyse the frames that are finished processing after the deadlines. In soft real time systems, jobs whose deadlines are missed still provide some value if they can be completed as early as possible. We examine the distribution of overdue time for each inference approach and show the results in the form of CDFs in Figure 5. We can see that DeepRT performs the best in terms of deadline overdue time due to its utilization of the EDF algorithm.

Peak GPU memory usage. We also measure the peak GPU memory usage of the $4$ approaches under different traces. Peak GPU memory usage is an important metric since the computations on GPU are memory demanding. An inference system consuming too much GPU memory could drain the memory, leading to memory allocation errors. On the desktop computer we use nvidia-smi to measure GPU memory usage, and on Jetson TX2 we use tegrastats. The results are shown in Figure 6.

In this part we evaluate the throughput performance of DeepRT. Since there is no existing real time scheduler for CNN inference on the edge, we implement a real time scheduler, Sequential EDF (SEDF), and compare the throughput performance of DeepRT agaist SEDF. We would like to examine whether DeepRT is able to offer high inference throughput while meeting latency requirements compared to SEDF. As its name suggests, SEDF processes input frames from multiple requests one by one according to the frames’ deadlines. It doesn’t execute multiple models concurrently, nor does it perform batching. We also implement an EDF imitator as the admission control policy of SEDF. It is worth mentioning that we don’t compare DeepRT with the baseline approaches in Section 6.2 since those approaches are not real time schedulers and therefore do not provide latency guarantees. Besides, they do not have admission control modules which reject requests if they can cause deadline misses. Therefore, we compare DeepRT with a soft real time scheduler SEDF to guarantee a fair comparison.

We would like to see (1) how many concurrent requests each inference system can handle and (2) what is the throughput of each system. We use the same method as Subsection 6.2 to generate request traces, except that we increase the frequency of request arrival to saturate the inference systems. Another difference is that, we feed the two systems with the same pending requests, but we let each of them determine which requests to admit. The number of concurrent requests each approach can handle and the average throughput each approach achieves are shown in 7.

We observe that on all the traces DeepRT performs better than or as well as SEDF, due to the fact that the novel batching approach of DeepRT leverages the batching ability of GPU and exhibits high throughput while providing latency guarantee. We can see that on the third traces of both devices, DeepRT largely outperforms SEDF, while the differences on the first traces are smaller. As the mean relative deadlines of the requests in the third traces are larger, more frames can be batched due to our DisBatcher design, boosting the throughput performance of DeepRT. On the first traces, fewer frames are batched so DeepRT doesn’t have a high performance gain compared to SEDF.

In this part we evaluate the performance of the Admission Control Module. Specifically, we would like to examine (1) whether the Admission Control Module is able to accurately model the system to make admission decisions, and (2) what is the running time of the Admission Control Module.

Accuracy of the EDF imitator. Naturally, we would like the Admission Control Module to admit as many requests as possible to increase throughput while not violating any deadline requirements. That is the reason why we employ an EDF imitator as an exact analysis tool to determine schedulability. We evaluate how accurate the EDF imitator is in estimating future job instance executions.

We generate $3$ traces of requests with the method in Subsection 6.2. The only difference lies in the mean values of periods and relative deadlines. For the first trace, we set the mean period to be $100ms$ and mean relative deadline to be $300ms$. For the second trace, we set both values to be $200ms$. And for the third trace we set them to be $300ms$ and $100ms$, respectively. The reason of using these configurations is that we would like to examine the EDF imitator under various batch sizes and various deadlines. We only perform this experiment on the desktop computer as the effectiveness of the EDF imitator is the same across all platforms as long as the profiled worst-case job instance execution times are accurate.

We use the difference between the estimated latency of frame inference from the EDF imitator and the actual frame latency measured during real executions as the metric of accuracy, as the major goal of DeepRT is to provide latency guarantee for users. The result is shown in Figure 9. We can see that the difference is the smallest on the trace with the smallest deadline, and vice versa. On the first trace which corresponds to the largest relative deadline ($300ms$), the difference can be as large as $250ms$. We find that the large latency differences happen on latter frames in a request frame sequence. In fact, when the EDF imitator is performed on some requests, it considers all frames in the requests and latency estimation errors accumulate over the frame sequences. But large latency differences are rare and still smaller than the corresponding relative deadlines ($250ms<300ms$). Overall the EDF imitator is sufficiently accurate to predict whether the deadline of a frame will be missed.

Admission Control Module running time. As mentioned in Section 4, the complexity of the EDF imitator is linear with respect to the number of frames in all requests. We evaluate the running time of the Admission Control Module on both devices under different number of frames. Specifically, we generate $4$ request traces for both devices using the previous method; the requests in the $4$ traces contain videos with $10^{2}$, $10^{3}$, $10^{4}$, and $10^{5}$ frames, respectively. The running times (see Figure 9) are all below $1$ second except the case where Jetson TX2 processes requests with $10^{5}$ frames, where the running time is $5.9$ seconds. If we consider the normal frame rate of a video to be $30fps$, $10^{5}$ frames correspond to a video of approximately one hour. In fact, if DeepRT is used to perform inference on long videos, we can calculate the least common multiple of their periods and run the EDF imitator over twice that time period, significantly reducing the running time.

We evaluate how quickly DeepRT reacts to job instance overruns and eliminate deadline misses caused by these overruns. We generate request traces with periods and relative deadlines to be $200ms$ for desktop and $600ms$ for Jetson TX2. In each run we manually inject a short waiting time to $5$ consecutive job instances and measure the number of deadline misses caused by the injected waiting time. If a certain method reacts to overruns faster, it can reduce the number of deadline misses. We run this experiment on both the desktop computer and Jetson TX2. The lengths of the waiting times are set to be $100ms$, $200ms$, $500ms$, and $1000ms$.

We compare the number of deadline misses between enabling and disabling the Adaptation Module in Figure 10. We can see that even without the Adaptation Module, DeepRT is still able to bring the system back to normalcy after experiencing some deadline misses. The reason is that DeepRT does not achieve $100\%$ utilization of the GPU as it is a real time system. There is idle time between job instance executions which act as buffer absorbing the overruns. The Adaptation Module enhances the ability to absorb the overruns.