RT3D: Achieving Real-Time Execution of 3D Convolutional Neural Networks on Mobile Devices

Since the Convolutional Neural Networks (CNNs) were exemplified by the performance improvements obtained by AlexNet (Krizhevsky, Sutskever, and Hinton 2012) in 2012, neural network based computer vision has achieved superhuman performance. Mobile devices are becoming an important carrier for deep learning tasks. However, real-time execution is the most critical requirement given computation/storage resource constraints on mobiles for deep learning tasks.

Recently, many efforts (Han et al. 2016; Yao et al. 2017; Huynh, Lee, and Balan 2017; Chen et al. 2018; Alibaba 2020; TensorFlow 2017; PyTorch 2019) aim to accelerate CNN execution on off-the-shelf mobile devices and some of them achieve significant advancements. However, most of these optimizations focus on traditional 2D CNNs in the image domain. On the other hand, 3D CNNs have been proposed for video domain tasks such as video classification, and action recognition/detection (Ji et al. 2012; Wang et al. 2017; Carreira and Zisserman 2017; Qiu, Yao, and Mei 2017; Köpüklü et al. 2019) It is still an open problem to execute 3D CNNs on off-the-shelf mobile devices targeting for real-time performance. For example, C3D (Tran et al. 2015), a mainstream 3D CNN takes over 2.5 seconds to complete the inference (of 16 frames) on a representative mobile CPU (Kryo 585 in Qualcomm Snapdragon platform) with Pytorch Mobile (PyTorch 2019), which is clearly far from real-time execution.¹¹1Real-time performance requires to compute 30 frames/second according to state-of-the-art industry standard. The extra dimension in 3D convolution (CONV) significantly increases storage size and computation workload comparing with 2D CONV.²²22D CONV is a special case of 3D CONV with the temporal dimension size equal to 1. The large memory footprint of 3D CNN models often exceeds the on-chip cache size of off-the-shelf mobile devices. As a result, 3D CNNs are currently supported only by very few mobile acceleration frameworks i.e., PyTorch Mobile (PyTorch 2019) and Alibaba MNN (Alibaba 2020) with relatively low computation efficiency, let alone real-time execution performance.

A natural way to bridge the gap is to turn to model compression techniques, particularly weight pruning (Wen et al. 2016; Han, Mao, and Dally 2016; Guo, Yao, and Chen 2016; Dong and Yang 2019; Zhuang et al. 2018; Yu et al. 2018; He et al. 2019) which has demonstrated its efficacy on accelerating 2D CNN executions. Nevertheless, generalizing weight pruning methods from 2D to 3D CNNs is more than a straightforward task owing to the higher dimensionality of weight tensors and thus the larger search space of weight pruning. It is especially challenging to derive the best-suited weight pruning method in order to achieve real-time performance on off-the-shelf mobile devices. Two fundamental problems need to be solved: the sparsity scheme and the pruning algorithm. The former refers to the regularity in pruning, i.e., the specific structural characteristics of CNNs after pruning. The two representative cases for 2D CNNs are the most flexible, irregular pruning scheme that can prune arbitrary weights (Han, Mao, and Dally 2016; Guo, Yao, and Chen 2016), and the computing platform-friendly filter/channel pruning scheme that prunes whole filters/channels (Wen et al. 2016; He et al. 2019; Yu et al. 2018). The latter refers to the appropriate algorithm to determine the target weights to remove and train the remaining, non-zero weights. For 2D CNNs, there is also rich literature in heuristic pruning (Han, Mao, and Dally 2016; Guo, Yao, and Chen 2016; Dong and Yang 2019) or regularization-based pruning algorithms (Wen et al. 2016; Yu et al. 2018; He et al. 2019).

This work develops RT3D framework, including the derivation of best-suited (structured) sparsity scheme and pruning algorithm of 3D CNNs, and the design of the associated compiler-aided acceleration, for off-the-shelf mobile devices. We propose and investigate two structured sparsity schemes that are highly mobile acceleration friendly. The first vanilla sparsity scheme achieves sparsity by removing kernel groups in 3D CONV layers. It can achieve straightforward acceleration for on-device inference with the aid of compiler code generation, but it suffers from relatively high accuracy loss as whole kernel groups are pruned. The second, more optimized one is the kernel group structured (KGS) sparsity scheme. It is a more fine-grained structured sparsity that enjoys higher flexibility, and will result in a higher accuracy under the same pruning rate. Moreover, it is important to note that the KGS sparsity scheme is beyond a mere tradeoff of accuracy and mobile performance. In fact, with proper support of compiler code generation, the KGS sparsity can achieve almost the same mobile acceleration (e.g., in frames/second) as the vanilla sparsity, under the same pruning rate. This is owing to the delicate design of KGS sparsity to match the parallelization mechanism in compiler-assisted mobile acceleration, such that the full on-device parallelism can be exploited.

We further present three pruning algorithms to achieve the proposed structured sparsity schemes for 3D CNNs. The first two, i.e., the heuristic algorithm and regularization-based algorithm, are natural generalization from state-of-the-art algorithms on 2D CNN weight pruning. However, they are either greedy algorithm, or suffer from the limitation that all weights will be equally penalized even after convergence of the pruning process. Both result in potential accuracy loss. To overcome these shortcomings, we propose a novel reweighted regularization pruning algorithm. The basic idea is to systematically and dynamically reweight the penalties, reducing the penalties on weights with large magnitudes (which are likely to be more critical), and increasing the penalties on weights with smaller magnitudes. It possesses other advantages, such as not introducing additional hyperparameters, and being flexible for either parameter reduction or FLOPs (floating-point operations) reduction, etc.

Seamlessly integrated with above innovations, RT3D also develops the first end-to-end, compiler-assisted acceleration framework of 3D CNNs on both mobile CPUs and GPUs (the few prior work are limited to mobile CPUs), and also the first to support different structured sparsity schemes. RT3D achieves up to $29.1\times$ speedup in end-to-end inference time comparing with current mobile frameworks supporting 3D CNNs, with moderate 1%-1.5% accuracy loss, on representative CNNs (C3D, R(2+1)D, S3D). The end-to-end inference time for 16 video frames could be within 150 ms.

A brief contribution summary is: (a) sparsity schemes for 3D CNNs which are both flexible and mobile acceleration friendly, (b) highly effective pruning algorithm to achieve such sparsity schemes, (c) compiler-assisted mobile acceleration framework, and (d) for the first time, real-time performance of 3D CNNs can be achieved on off-the-shelf mobile devices using a pure software solution.

This section proposes two structured sparsity schemes of 3D CNNs. We focus on the most computationally intensive convolutional (CONV) layers of 3D CNNs. Let ${\mathbf{W}}_{l}\in\mathbb{R}^{M\times N\times K_{h}\times K_{w}\times K_{d}}$ denote the 5-dimensional weight tensor of the $l$-th CONV layer of a 3D CNN, where $M$ is the number of filters; $N$ is the number of input channels; $K_{w}$, $K_{h}$, and $K_{d}$ are the width, height, and depth, respectively, of the 3D CONV kernels. Different from the 2D CONV kernel, the 3D CONV kernel has an additional dimension on the kernel depth, making ${\mathbf{W}}_{l}$ a 5-dimensional tensor.

Figure 1 demonstrates the proposed two structured sparsity schemes for 3D CNNs: Vanilla Structured Sparsity Scheme and Kernel Group Structured (KGS) Sparsity Scheme. The weight tensor ${\mathbf{W}}_{l}$ is first partitioned into groups of kernels along the filter and input channel dimensions. Each kernel group consists of $g_{M}\times g_{N}$ ($2\times 2$ in Figure 1) 3D kernels. The Vanilla sparsity scheme is shown in Figure 1 (a), where whole kernel groups are determined to be pruned or not. On the other hand, our proposed KGS sparsity scheme as shown in Figure 1 (b) is that for the kernels in the same group, weights are pruned at the same locations. This is illustrated better on the right of Figure 1 (b), where 3D kernels are reshaped into vectors with $K_{s}=K_{h}\times K_{w}\times K_{d}$ weights. Consider the $g_{M}\times g_{N}$ kernels in a group, i.e., kernels ${\mathbf{W}}_{l}(m:m+g_{M}-1,n:n+g_{N}-1,:,:,:)$. Weights at the same location in these kernels i.e., ${\mathbf{W}}_{l}(m:m+g_{M}-1,n:n+g_{N}-1,h,w,d)$ are determined to be pruned or not together, where $(:,:,h,w,d)$ describes the same location (coordinate) in kernels.

The Vanilla sparsity scheme is a relatively straightforward generalization from structured sparsity schemes (Wen et al. 2016; Liu et al. 2017; Luo, Wu, and Lin 2017) for 2D CNNs. It can achieve straightforward acceleration for on-device inference with the aid of compiler code generation, but it will obviously suffer from relatively high accuracy loss as whole kernel groups are pruned. On the other hand, the proposed KGS sparsity scheme is a more fine-grained structured sparsity that enjoys higher flexibility. In fact, the Vanilla sparsity scheme is just a special case of KGS sparsity, and therefore, one can confidently state that the KGS sparsity will result in a higher accuracy under the same pruning rate, as long as effective pruning algorithm has been developed and employed.

It is important to note that the KGS sparsity scheme is beyond a mere tradeoff of accuracy and mobile performance. In fact, with proper support of compiler code generation, the KGS sparsity can achieve almost the same mobile acceleration performance (e.g., in frames/second) as Vanilla sparsity, under the same pruning rate. This is owing to the delicate design of KGS sparsity to match compiler-assisted mobile acceleration. For effective mobile acceleration, the whole kernel group will be transformed into matrix multiplication (with input feature map) (Chetlur et al. 2014) as shown in the reshaping step of Figure 1 (b). Accordingly, the KGS sparsity is equivalent to whole column removals in the weight matrix of a kernel group. The computation overhead in whole column removal is minor and can be mitigated by compilers, and the remaining computation is still based on full matrices (albeit smaller). A key observation is that the parallelism degree on off-the-shelf mobile devices is limited, and thus the smaller matrices of remaining weights have enough size to fully exploit the parallelism provided by mobile devices. As an illustrative example, suppose that the mobile device can execute 10 operations in parallel while the matrix contains 100 remaining operations. Then the reduced-size matrix can be executed in 10 iterations, achieving full parallelism. As the hardware parallelism can be fully exploited in both Vanilla and KGS schemes (if compiler overhead is negligible), the mobile acceleration performance in terms of FLOPs/second will be almost the same for both pruning schemes, and so does the frames/second performance under the same pruning rate (and FLOPs count). As a result, the proposed KGS sparsity can fully enjoy the benefit of high flexibility in terms of higher accuracy or higher pruning rate.

Please note that the $g_{M}\times g_{N}$ group size needs to be determined in Vanilla and KGS sparsity schemes, in order to achieve the maximum on-device parallelism with low computation overhead. The group size is determined offline with actual mobile testings using synthesized CNN layers. In other words, it will NOT become a hyperparameter in the pruning algorithm. $g_{N}=4$ and $g_{M}=4$ or $8$ are preferred to match the SIMD (Single Instruction, Multiple Data) parallelism provided by current mobile CPUs and GPUs. These values are large enough to exploit the on-device parallelism and small enough to provide enough pruning flexibility and accuracy, as shall be seen in the experimental results.

This section describes three pruning algorithms to achieve the proposed structured sparsity schemes for 3D CNNs. The first two are natural generalization from state-of-the-art algorithms on 2D CNN weight pruning, and the last one is specifically designed to address the limitations in the prior two. Consider a general 3D CNN consisting of $L$ convolutional (CONV) layers. Besides the $l$-th CONV layer weight tensor ${\mathbf{W}}_{l}$, the bias is denoted by ${\mathbf{b}}_{l}$. The loss function associated with a 3D CNN can be denoted by $F(\{{\mathbf{W}}_{l}\}^{L}_{l=1},\{{\mathbf{b}}_{l}\}^{L}_{l=1})$. To achieve the proposed group-wise sparsity schemes, weight tensor ${\mathbf{W}}_{l}$ is partitioned into a set of kernel groups along the dimensions of filters and channels, i.e., $\{{\mathbf{W}}_{l}^{\mathcal{G}_{p,q}}\}$, for $p\in[P]$ and $q\in[Q]$, where $P=\lceil M/g_{M}\rceil$, $Q=\lceil N/g_{N}\rceil$, and $[n]$ denotes the integer set $\{1,2,\ldots,n\}$. Figure 2 provides an illustrative example of kernel groups.

1. Heuristic Pruning Algorithm: As discussed in Section 2.1, the prior work has investigated heuristic pruning for 2D CNNs, for both irregular and structured sparsity schemes. The prior work (Luo, Wu, and Lin 2017; Yu et al. 2018) are mostly relevant to this work as we also focus on structured sparsity. Motivated by these work, we assign a similar “neuron importance score” to each kernel group (or the same location of kernels in the group), and perform pruning on the current layer with input information of the next layer in a back propagated manner (similar procedure as (Luo, Wu, and Lin 2017)). This serves as our heuristic pruning algorithm for the proposed sparsity schemes of 3D CNNs.

2. Regularization-based Pruning Algorithm: adds an additional regularization term to the loss function to achieve the Vanilla or KGS sparsity scheme. Then, the regularization-based pruning can be formulated as

$$\underset{\{{\mathbf{W}}_{l}\},\{{\mathbf{b}}_{l}\}}{\text{minimize}}\ \ F\big{(}\{{\mathbf{W}}_{l}\}^{L}_{l=1},\{{\mathbf{b}}_{l}\}^{L}_{l=1}\big{)}+\lambda\sum^{L}_{l=1}R_{g}\big{(}{\mathbf{W}}_{l}\big{)},$$

(1)

where $R_{g}\big{(}{\mathbf{W}}_{l}\big{)}$ is the regularization term for the Vanilla or KGS sparsity and $\lambda$ is the penalty measuring its importance. Motivated by group Lasso (Yuan and Lin 2006), the regularization term can be defined as $R_{g}\big{(}{\mathbf{W}}_{l}\big{)}=\sum_{p=1}^{P}\sum_{q=1}^{Q}\left\|{\mathbf{W}}_{l}^{\mathcal{G}_{p,q}}\right\|_{g}$, where $\left\|\cdot\right\|_{g}$ denotes kernel group $\ell_{p}$ norm. We can choose from $\ell_{1}$ norm (Liu et al. 2017), $\ell_{2}$ norm (He et al. 2018; Li et al. 2019) or their combination for this group-wise regularization.

In more details, the regularization-based pruning can be achieved by

		$\displaystyle\underset{\{{\mathbf{W}}_{l}\},\{{\mathbf{b}}_{l}\}}{\text{minimize}}\ F\big{(}\{{\mathbf{W}}_{l}\}^{L}_{l=1},\{{\mathbf{b}}_{l}\}^{L}_{l=1}\big{)}+$		(2)
		$\displaystyle\lambda\sum^{L}_{l=1}\sum_{p=1}^{P}\sum_{q=1}^{Q}\sum_{h=1}^{K_{h}}\sum_{w=1}^{K_{w}}\sum_{d=1}^{K_{d}}\left\|{\mathbf{W}}_{l}^{\mathcal{G}_{p,q}}{(:,:,h,w,d)}\right\|_{g}.$

3. Reweighted Regularization Pruning Algorithm: As discussed in Section 2.1, the fixed regularization-based pruning algorithm has a limitation, – all weights will be equally penalized even after convergence of the pruning process, resulting in potential accuracy loss. We propose a novel reweighted regularization pruning algorithm to overcome this limitation. The basic idea is to systematically and dynamically reweight the penalties. Especially, we will reduce the penalties on weights with large magnitudes (which are likely to be more critical), and increase the penalties on weights with smaller magnitudes. This shall be performed in a systematic, gradual way to avoid the greedy solution which prunes a large number of weights at the early stage. Moreover, our proposed algorithm does not need to manually set the pruning rate for each layer, as a limitation in prior works based on ADMM or Geometric Median-based regularizations.

For reweighted regularization, we minimize the following objective function:

		$\displaystyle\underset{\{{\mathbf{W}}_{l}\},\{{\mathbf{b}}_{l}\}}{\text{minimize}}\ F\big{(}\{{\mathbf{W}}_{l}\}^{L}_{l=1},\{{\mathbf{b}}_{l}\}^{L}_{l=1}\big{)}+\lambda\bigg{[}\sum^{L}_{l=1}\sum_{p=1}^{P}\sum_{q=1}^{Q}$		(3)
		$\displaystyle\sum_{h=1}^{K_{h}}\sum_{w=1}^{K_{w}}\sum_{d=1}^{K_{d}}\bigg{(}{\mathcal{P}}_{l,t}^{\mathcal{G}_{p,q}}\circ\left\|{\mathbf{W}}_{l}^{\mathcal{G}_{p,q}}{(:,:,h,w,d)}\right\|_{g}\bigg{)}\bigg{]},$

where $\circ$ denotes element-wise multiplication. ${\mathcal{P}}_{l,t}^{\mathcal{G}_{p,q}}$ is the collection of penalty parameters and is updated in every iteration $t$ to facilitate the degree of sparsity. In each iteration, the instance of ${{\mathbf{W}}}_{l}^{\mathcal{G}_{p,q}}$ is denoted by ${{\mathbf{W}}}_{l,t}^{\mathcal{G}_{p,q}}$ and we update ${\mathcal{P}}_{l,t}^{\mathcal{G}_{p,q}}$ by setting

$${\mathcal{P}}_{l,(t+1)}^{\mathcal{G}_{p,q}}=\frac{1}{\left\|{\mathbf{W}}_{l,t}^{\mathcal{G}_{p,q}}{(:,:,h,w,d)}\right\|_{g}^{2}+{\epsilon}},$$

where $\epsilon$ is a small positive number avoiding the zero denominator. The reweighted regularization process updates penalty parameters based on the current weight values, will not incur extra hyperparameters, and has a fast convergence rate as analyzed in (Candes, Wakin, and Boyd 2008). After 3$\sim$4 iterations, we will prune the weights that converge to zero, and perform a slight retraining on the non-zero weights (with a few epochs) to regain accuracy.

While overcoming the limitation in fixed regularization-based algorithms, the advantage and flexibility in such algorithms will be preserved. There is only one $\lambda$ as the major hyperparameter, without the need of manually deciding per-layer pruning rate. Also similar to fixed regularization algorithms, we can multiply the per-layer FLOPs value to each layer $l$ in the above optimization function. In this way we can target at the overall FLOPs reduction, which is more relevant to the actual acceleration. In the experiments, we set the FLOPs reduction as the optimization target, and report the corresponding FLOPs reduction rates and actually measured mobile accelerations.

This paper presents RT3D, a mobile acceleration framework for 3D CNNs that includes two novel, mobile-friendly structured sparsity schemes (Vanilla and KGS) and best-suited pruning algorithms, that can achieve low inference latency and high accuracy, simultaneously. Based on them, RT3D employs a compiler-assisted code generation framework to transform pruning benefits to performance gains. The evaluation results show that RT3D beats two state-of-the-art acceleration frameworks with speedup up to $29.1\times$. RT3D can infer 16 video frames within 150 ms, for the first time, achieving real-time inference of 3D CNNs on off-the-shelf mobile devices with a pure software solution.

As a pure software solution, RT3D is the first to achieve real-time execution of 3D CNNs on mobile devices without notable accuracy loss–which can only be achieved by special (and more expensive) hardware solutions previously. This research will significantly encourage the general research of deep learning acceleration with software-based techniques while reducing the demand for some hardware-based accelerations. RT3D will enable many machine learning applications of behavior/activity detection on mobile platforms that have to run on the cloud previously.

The ethical aspects and future societal consequences of this research are highly application-dependent. This work has the following potential positive impact in society: First, because machine learning applications can run on the mobile (edge) side only without transmitting user data to the cloud server, data privacy is significantly enhanced, thus these applications can run in a more private environment. Second, this work may also significantly broaden the usage of machine learning in many other domains ranging from medical science, biology to geoscience and environmental science, e.g., combining motion sensors and high-precision 3D CNN inferences to support short-latency motion recognition can enable a real-time Parkinson treatment. At the same time, this work may have some negative consequences: due to the low-cost and easy-accessible nature of machine learning on mobile, this work has the potential of increasing the possibility of misusing machine learning techniques. Furthermore, we should be cautious of the result of the failure of the system which could cause wrong decision making, thus jeopardizing the safety of the public and individuals. In addition, all experiments in our work are based on the public dataset and our task/method does not leverage biases in the data.

This material is based upon work supported by the National Science Foundation (NSF) under Grants CCF-1937500, CNS-1932351, Army Research Office Young Investigator Program 76598CSYIP, and Jeffress Trust Awards in Interdisciplinary Research. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of NSF, or Thomas F. and Kate Miller Jeffress Memorial Trust.