A Survey of Multi-Tenant
Deep Learning Inference on GPU

Traditional DL computing optimization in full stack often expands in service-level orchestration [36], graph-level optimization [15, 43], runtime-level scheduling [10], kernel-level tuning [3, 39] and resource-level management [9, 19]. Although there are many previous works for computing optimization in these difference levels, multi-tenant computing shows dramatic characteristics that make these methods ill-fitted. According the the same optimization stack, we summarize the major differences in Table I and analyze the computing challenges in Figure 2.

However, the challenges remains for strategic co-location like that the inter-tenant interference [19] could happen and degrade the quality of service such as service-level objectives (SLA) of tail latency and throughput. This could become worse with increased number of co-located workloads and degrade the overall serving throughput. Therefore, there are many recent service-level orchestration works [19, 42, 5] that design different heuristic-based, modeling-based or prediction-based mechanisms to conduct strategic co-location for efficient multi-tenant computing on GPUs.

By contrast, multi-tenant DL computing with multiple parallel DAGs usually have extensive inter-operator parallelism, as shown in Figure 2, which enables more flexible inter-operator scheduling among tenants [44]. Certain challenges exist in such graph scheduling such as the increased complexity in larger number of operators and scheduling space, and the more complex GPU resource contention analysis, etc.

Previously, one important reason that hinders the adoption and development of multi-tenant DL computing on GPU is the insufficient hardware scheduling mechanisms. But with the increasing attention in this topic, GPU vendors like Nvidia release certain new GPU multi-tenant features to support multi-tenant DL inference, which provides great opportunities for multi-tenant scheduling. The multi-tenant GPU scheduling features could be categorized into two major types: software-level and hardware-level support.

DL service-level orchestration is an important feature in large-scale data centers to improve the GPU utilization. As the top-most scheduling level, such orchestration usually regards one service query as the basic scheduling unit. This reduces the scheduling complexity as there is no need to consider the intra-DNN model structure details (operators and graphs). One example is the Microsoft Deep Learning Inference Service (DLIS) system [36]. The service orchestrator characterizes different models’ resource requirements and then strategically places one or multiple queries onto hosts through the service router. Therefore, it could maximize the served queries per second (QPS) while ensure little inter-query interference so as to maintain similar tail latency.

However, designing a proper co-location strategy or system is a non-trivial task. For example, one challenging factor is the serving dynamics, i.e., undetermined arrival rates and/or distribution of incoming DL queries, different RNN/LSTMs queries with varied inputs and control states. Distinct from static workloads that we can get the full information, such dynamic scenarios require us to either utilize historical data or predict the future workload dynamics. PREMA [5] proposed a predictive multi-task DNN scheduling algorithm that combines off-line records and online token-based job scheduling to determine the best multi-tenant co-location strategy.

Another challenge in multi-tenant co-location is how to accurately predict the inter-model resource interference. This is a critical factor in ensuring QoS such as tail latency. [19] trained a ML-based latency degradation predictor under co-location using offline-profiled hardware-level features such as SM and DRAM usage, PCIE read/write BW, buffer usage, etc. Then the latency degradation predictor is used to evaluate the model placement’s potential influence for each query.

However, these works have certain scalability issues as they mostly targeted at static model types, hardware types, etc., which may not be suitable for dynamic workloads. Meanwhile, as each DNN can have many operators (e.g., layers) that have fluctuated resource consumption, such coarse-grained scheduling (with one entire query as the basic unit) may suffer from resource under-utilization/contention occasionally and thus still hinders the QoS.

Graph and runtime-level scheduling could help address one of the aforementioned challenge of coarse-grained granularity by enabling more fine-grained scheduling, e.g., the DNN operators. This could be done by leveraging the GPU software-level support such as the multi-stream mechanism and scheduling APIs. For example, [44] propose an ML-based scheduling strategy for multi-tenant DNN exeuction acceleration. It first abstracts multiple DNN’s computation graph with all operators into a global intermediate representation (IR), which enables flexible resource sharing between different tenants so as to improve the utilization. To find the optimal concurrent operator execution strategy in the huge scheduling space, they design a ML-based auto-search method by defining three main factors: scheduling search space, profiling-guided latency cost model, and the ML search algorithm. Based on offline profiling records, the search algorithm could find the best scheduling for optimal GPU utilization and throughput.

Such graph and runtime-level operator scheduling could usually achieve better performance due to the fine-grained design, but they also face more scalability issues, e.g., when the number of co-located workloads increase to very large. Meanwhile, it also applies to static or known multi-tenant workload only, which cannot address dynamic model types.

Besides the aforementioned works, another optimization perspective to solve the inter-tenant inference is to conduct fine-grained resource managing [9, 13]. For example, spatial partitioning and allocation of GPU resources to different DL workloads could isolate different jobs’ resource (e.g., stream multiprocessors (SMs), memory bandwidths), thus avoiding the job interference in the hardware resource level. However, as we introduced before, achieving fine-grained resource partitioning is non-achievable until recently GPU vendors release a series of resource sharing and partitioning support like multi-streams, multi-process services (MPS [25]) and multi-instance GPU (MIG [24]). Most recent resource-level management works are built upon these technologies.

For example, GSlice [9] uses MPS to conduct adaptive SM partitioning for different DNN jobs. They design a self-learning method to dynamically adjust the GPU resource allocation ratio for each workload and thus avoid interference among co-located DL workloads and maximize the throughput. [4] utilizes similar spatial partitioning mechanism by MPS while additionally combines temporal scheduling strategies. MIG-Serving [38] is the most recent work that adopts the newly-released MIG feature on A100 to achieve spatial resource management for multi-tenant scheduling.

However, such spatial resource partitioning solutions also have a intrinsic limitation that is the inflexible re-configuration when the workloads change and requires resource partitioning adjustment. For GPUs, re-configuring the resource partitioning requires certain amount of time (e.g., tens of $ms$ or more), which can be even larger than one DL inference workloads’ processing time. Therefore, re-configuring frequently is not practical and thus limits such solutions’ performance when facing dynamic workloads. [9] tries to reduce the stall caused by reconfiguration time of MPS by utilizing a standby/shadow process. However, the minimum time for switching one partitioning configuration to another one still cost several seconds, which is still non-negligible in online serving.

Another way of leveraging ML-based prediction is to conduct light-weight modeling to predict the latency degradation under different multi-model and hardware combinations so that the scheduler can make better decision regarding the latency SLA constraints. For example, the work [19] built a ML model to predict the latency of multi-model inference cases on different machines. As the effectiveness of the final scheduling solution highly depends on the modeling accuracy, the scalability and generality issue across hardware/model types needs to be addressed, which can be also very challenging.

The fast development of multi-tenant DL computing brings many challenges for the system stack optimizations. Besides for the GPU only, this also enlightens the other DL-oriented hardware architecture designers (e.g., TPU, chiplet, neuromorphic and quantum-based accelerators) to optimize for flexibility and agility facing a rapidly changing DL application landscape. Specifically, one important future trend is the “full stack in the loop”, i.e., to remove the boundaries in the vertical DL system stack and conduct full-stack integration to strive for both optimal performance and flexibility.

One example in this line of efforts is the DL compiler renovation by tvm unity [32], as shown in Figure 4 (a). Current DL computing stack conducts separate layer-wise optimization (graph-runtime-kernel-resource) and single directional top-down deployment. This however prohibits feedback loops and cross-layer interactions for SW/HW co-compiling based on model workloads and hardware characteristics. Unifying the abstraction between layers would thus greatly facilitate the new full-stack optimization as a loop, not only for multi-tenant computing, but also for future wider DL application.

Another example is the increasing attention in the versatile and flexible chiplet-based SW/HW co-design [33, 48] that uses multi-chip-modules (MCMs). Compared to traditional large monolithic die, such MCM combines smaller chiplets into a larger system and substantially reduces fabrication and design costs. However, it requires thorough application awareness to optimize the chip design and overall performance. As so, such chiplet modules could also greatly benefit from the full-stack-in-the-loop architecture of DL computing.

Multi-tenant DL computing is a natural generalization result due to the significant computing scaling trend of GPU. However, recently another model scaling trend is observed, that is, designing and training super-scale AI models for general intelligence. For example, the recent SOTA giant AI model Megatron-NLG [35] has reached 530 billions of parameters and requires tens of GPUs to conduct multi-node distributed inference. If we take such model scaling into consideration, even more new DL computing modes could be observed and enrich the future DL & system landscape.

We describe the future large-scale DL system landscape by using a taxonomy in Figure 4 (b). Using Instance (I) to denote one DNN model and Device (D) to denote the hardware, traditional DL system mostly comes within the Single Instance Single Device (SISD) domain and only constitute the top-left quarter in the full spectrum. Multi-tenant computing emerges as the Multiple Instances Single Device (MISD) with the computing scaling trend, as we summarized in this servery.

Whereas diagonally, with the model scaling trend, the Single Instance Multiple Devices (SIMD) interaction mode also emerges and is attracting more attention such as distributed inference for super-scale giant models [18, 47, 50] including language, recommendation models, etc. Finally, Multiple Instances Multiple Devices (MIMD) computing would eventually combine all these modes and become a practical needs for future DL-centric data center optimization.