1xN Pattern for Pruning Convolutional
Neural Networks

We start with notation definitions. Considering a pre-trained $L$-layer CNN model $F$, we denote its filter set as $\mathbf{W}=\{\mathbf{W}^{i}\}_{i=1}^{L}$ with $\mathbf{W}^{i}=\{\mathbf{W}^{i}_{j}\}_{j=1}^{n^{i}}\in\mathbb{R}^{n^{i}\times m^{i}\times h^{i}\times w^{i}}$, where $n^{i}$, $m^{i}$, $h^{i}$ and $w^{i}$ respectively indicate the number of output channel, input channel, kernel height and kernel width in the $i$-th layer; $\mathbf{W}^{i}$ is the filter set for the $i$-th layer and $\mathbf{W}_{j}^{i}$ is the $j$-th filter in the $i$-th layer. For the fully-connected layer, its weight matrix is indeed an exception of $h^{i}=1$ and $w^{i}=1$. In order to simplify the explanations, in the following contents, we only talk about the convolutional layers as illustrations.

Network pruning can be implemented by imposing a mask $\mathbf{T}^{i}$ upon $\mathbf{W}^{i}$. Here $\mathbf{T}^{i}$ is a binary tensor (0 or 1) with its entries indicating the states of network connections, i.e., whether the corresponding weights are pruned or not. Thus, given an expected pruning rate $p$, network pruning is formally expressed as:

where $\oplus$ represents the masking operation, $\mathcal{L}(\cdot)$ measures the importance of its input, and $K$ denotes the size of $\mathbf{T}^{i}$ that varies according to the basic pruning granularity. We measure the input importance using the $\ell_{1}$ norm of the basic pruning granularity. We find this criterion sufficient, however, other metrics, such as weight gradients [6], activation sparsity [35], can be adopted as well.

Weight Pruning. The studies on weight pruning remove individual weights at any location of $\mathbf{W}^{i}$. Therefore, each mask $\mathbf{T}^{i}$ in weight pruning has the same shape with $\mathbf{W}^{i}$ of $\mathbb{R}^{n^{i}\times m^{i}\times h^{i}\times w^{i}}$ and its size $K=n^{i}\cdot m^{i}\cdot h^{i}\cdot w^{i}$. The specific objective of weight pruning is:

Filter Pruning. The studies on filter pruning remove the entire filter $\mathbf{W}^{i}_{j}$. Thus, each mask $\mathbf{T}^{i}$ in filter pruning has the shape of $\mathbb{R}^{n^{i}}$ and $K=n^{i}$. The specific objective of filter pruning is:

In the following, we introduce a novel pattern of 1$\times$N pruning, whose basic pruning granularity falls into consecutive N output kernels with the same input channel index. We show that our 1$\times$N pruning can be an efficient and effective alternative to simultaneously accelerate the model inference on modern CPUs-based platforms and retain the accuracy performance.

We define the problem of pruning CNNs using our 1$\times$N pattern. In order to simplify the explanations, we reformat the representation of $\mathbf{W}^{i}\in\mathbb{R}^{n^{i}\times m^{i}\times h^{i}\times w^{i}}$ as $\mathbf{\Omega}^{i}\in\mathbb{R}^{m^{i}\times n^{i}}$. Note that each element in $\mathbf{\Omega}^{i}$ is a kernel with the shape of $h^{i}\times w^{i}$, i.e., $\mathbf{\Omega}^{i}_{k,j}=\mathbf{W}^{i}_{j,k,:,:}$. Each column $\mathbf{\Omega}^{i}_{:,j}$ stands for a filter and each row $\mathbf{\Omega}^{i}_{k,:}$ consists of these kernels that have the same input channel index of $k$.

As shown in Fig. 1, we further partition the whole $\mathbf{\Omega}^{i}$ into a collection of smaller blocks. Our partition can be made more precise for an $m^{i}\times n^{i}$ matrix $\mathbf{\Omega}^{i}$ by partitioning $m^{i}$ into a collection of $m^{i}$ row-groups, and then further partitioning $n^{i}$ into a collection of $\frac{n^{i}}{N}$ col-groups. Consequently, each block $(k,j)$ is a 1$\times$N matrix including consecutive N output kernels with the same input channel index $k$, namely ${\mathbf{\Omega}}^{i}_{k,(j-1)\cdot N+1\,:\,j\cdot N}$. Based on this partitioned matrix, the basic pruning granularity of our 1$\times$N sparsity falls into these blocks. Thus, the mask, $\mathbf{T}^{i}$, in our 1$\times$N pruning has the shape of $\mathbb{R}^{m^{i}\times\frac{n^{i}}{N}}$ and its size $K=m^{i}\cdot\frac{n^{i}}{N}$. Finally, the specific objective of our 1$\times$N pruning is:

Furthermore, we realize that weight pruning and filter pruning are two special cases of our proposed 1$\times$N pruning pattern. Specifically, our 1$\times$N pruning degenerates to weight pruning subject to N = 1, $h^{i}=1$ and $w^{i}=1$. Besides, it further degenerates to filter pruning if $N=n^{i}$. When $1<N<n^{i}$, our method provides an intermediate granular level for network pruning, since it is coarser as compared to the fine-grained weight pruning but finer as compared to the coarse-grained filter pruning. Many previous researches [18, 19, 20] also explore removing kernels; however, our pruning pattern has a stronger requirement of continuity on the N removed kernels, with its merits in acceleration since these consecutive kernels can be stored continuously in the memory cache and the convolution with the inputs can proceed using a parallelized block-wise vectorized operation as analyzed in Sec. 3.4. Therefore, it is expected that our 1$\times$N pruning can offer a better performance than the filter pruning, and also an apparent inference speedup compared to the weight pruning.

As a distinguished difference from weight pruning and filter pruning, pruning kernels provides an intermediate granular level for network sparsity, since it is coarser as compared to the fine-grained weight pruning but finer as compared to the coarse-grained filter pruning.

Our 1$\times$N pruning in this paper follows the typical three-step pipeline of network training, then pruning consecutive kernels with smaller $\ell_{1}$ norms, and fine-tuning the sparse one to recover the performance in the end. At the second pruning step, inspired by [10] which conducts channel permutations to preserve more high-magnitude weights in the N:M weight pruning, we realize that the layout of $\mathbf{\Omega}^{i}$ can be altered to further relieve the pruning impact.

To implement the above process, we propose a workflow of filter rearrangement, whose working principle is shown in Fig. 2. Given the original weight matrix $\mathbf{\Omega}^{i}$ in the top-left of (a), simply applying 1$\times$N pruning upon $\mathbf{\Omega}^{i}$ leads to the loss of some kernels with relatively large values of $\ell_{1}$ norms in the top-right of (a). We calculate the $\ell_{1}$ norm of each filter, i.e., one column in $\mathbf{\Omega}^{i}$, then rearrange $\mathbf{\Omega}^{i}$ in the output channel dimension according to the calculated filter norm in a decreasing order as shown in the lower-left of (a). The rearranged weight matrix is denoted as $\tilde{\mathbf{\Omega}}^{i}$ to which our 1$\times$N pruning is further applied. As a consequence, more kernels with larger $\ell_{1}$ norms are preserved after pruning in the lower-right of (a), leading to an overall increasing weight magnitude as verified on MobileNet-V2 of (b). The filter rearrangement requires the outputs to be similarly rearranged as well, so as to maintain the same convolutional results with the next-layer weight matrix. However, frequently rearranging outputs incurs more run-time cost in the inference. Alternatively, we choose to apply a similar rearrangement to the input channel dimension of the next-layer weight matrix as illustrated in the middle of (a), which is accomplished once for all before pruning and thus brings no run-time cost. The effectiveness of filter rearrangement for accuracy improvements is presented in Sec. 4.

Herein, we want to stress the difference of our filter rearrangement against the channel permutation [10]. First, rearranging the columns of $\mathbf{\Omega}^{i}$ is indeed to change the position of each filter in our setting. Second, our goal is to preserve more high-magnitude kernels, while [10] is to preserve more individual high-magnitude weights. Third, we simply accomplish our rearrangement according to the $\ell_{1}$ norm of each filter. The implementation details of channel permutations are discussed in another paper [36] where a bounded regressions-based permutation search is proposed to find a high-quality permutation.

Then, Eq. (4) after filter rearrangement is rewritten as:

Note that the maximization of the above objective can be achieved by setting to $1$s the entries of $\mathbf{T}^{i}$ corresponding to these blocks in $\tilde{\mathbf{\Omega}}^{i}$ with their $\ell_{1}$ norms within the largest top-$(1-p)$, and $0$s otherwise. As a consequence, the pruned weights after applying our 1$\times$N pruning pattern is then derived as:

Given the input activation tensor $\mathbf{X}^{i}$ as illustrated in Fig. 3, the output tensor $\mathbf{Y}^{i}$ calculated by the standard dense-matrix structure is obtained as:

Considering a sparse matrix, operations using the standard dense-matrix structure bear inefficiency as processing and memory are wasted on a large number of zero-valued elements. Thus, it is of great necessity to take advantage of specialized data structures in order to store and manipulate the sparse matrix. However, the irregular sparse matrix resulting from weight pruning requires a great many of indices to record the positions of the reserved weights. Though Compressed Sparse Row Format (CSR) can be used to save index storage, irregular sparsity barely takes advantage of vector processing architectures and memory buses, resulting in little acceleration and even speed deterioration.

Weight pruning leads to an irregular sparse matrix, therefore, a large number of indices are needed to record the positions of the reserved weights. To save the index storage, the relative Compressed Sparse Row (CSR) format is usually adopted [4, 37], which encodes each index by the relative distance (i.e., the number of zeros) between two adjacent non-zero weights. Besides, a decoding process is needed to select the corresponding activations for the reserved weights. The main drawback of irregular pruning is that decoding one index requires a search over the whole activation vector, thus it brings little acceleration, even speed degradation.

In contrast, due to the requirement of continuity on the N removed kernels, our pruning pattern results in a sparse matrix $\bar{\mathbf{\Omega}}^{i}$ with constant-sized blocks. This good property brings two merits: First, the constant-sized blocks are by nature more easily encoded by Block Compressed Sparse Row Format (BSR) [38] to save non-zero elements in $\bar{\mathbf{\Omega}}^{i}$ with significantly less storage for the indices. Second, the output tensor $\mathbf{Y}^{i}$ is derived using a block-wise vectorized operation in parallel to achieve an apparent speedup. Specifically, as illustrated in Fig. 3, the pruned weight matrix $\bar{\mathbf{\Omega}}^{i}$ in our 1$\times$N pruning is encoded by BSR into three components: $1)\mathbf{D}^{i}\in\mathbb{R}^{t\times N}$, $2)\mathbf{I}^{i}\in\mathbb{R}^{t}$ and $3)\mathbf{P}^{i}\in\mathbb{R}^{\frac{n^{i}}{N}+1}$, where $t$ is the number of non-zero blocks in $\bar{\mathbf{\Omega}}^{i}$. The matrix, $\mathbf{D}^{i}$, and vector, $\mathbf{I}^{i}$, contain non-zero blocks and their row indices in $\bar{\mathbf{\Omega}}^{i}$, respectively. We form $\mathbf{D}^{i}$ by firstly stacking up non-zero blocks within the same col-group of $\bar{\mathbf{\Omega}}^{i}$, and concatenating the stacked ones across different col-groups. The vector $\mathbf{I}^{i}$ records the row index of each block item in $\bar{\mathbf{\Omega}}^{i}$. The $k$-th element of the vector $\mathbf{P}^{i}$ encodes the start row index of the $k$-th col-group in $\mathbf{D}^{i}$. The last element of $\mathbf{P}^{i}$ is a fictitious index, which is always equal to $t+1$. Attributed to the block storage format of $\mathbf{D}^{i}$, we can calculate $\mathbf{Y}^{i}$ in a block-wise manner during decoding as follows:

With proper index vectors $\mathbf{P}^{i}$ and $\mathbf{I}^{i}$, we can directly fetch these activations corresponding to non-zero weights for the output computation, through which we avoid a complete involvement of the whole activation tensor as with the dense-matrix structure. Besides, the block storage format of $\mathbf{D}^{i}$ also allows fast row data access since each block $\mathbf{D}^{i}_{\mathbf{I}^{i}_{z},:}$ is stored continuously in the memory. Besides, the block-wise vectorized operation in Eq. (8) can be implemented extremely fast as the multiplication between input item $\mathbf{X}^{i}_{k,\mathbf{I}^{i}_{z}}$ and each entry of $\mathbf{D}^{i}_{\mathbf{I}^{i}_{z},:}$ can proceed in parallel. Thus, our 1$\times$N pruning enables apparent acceleration on the general CPUs-based devices. To ensure end-to-end execution efficiency, we utilize the optimizing compiler TVM [39] to enable optimal code generation. And based on Eq. (8), we use Ansor [40] for automated tensor program generation to search best sparse convolution implementation.

For fair comparison, similar to our 1$\times$N pruning, we re-implement the compared baselines of weight pruning and filter pruning using $\ell_{1}$ norm as the importance evaluation. Besides, given the expected pruning rate $p$, we simply remove per-layer weights/filters/blocks with their corresponding $\ell_{1}$ norms within the smallest top-$p$. All experiments are performed using the pre-trained light-weight MobileNet-V1 [21], -V2 [22], -V3 [23] and large-scale ResNet-50 [2] on ILSVRC-2012 [24] that contains over $1.2$ million images for training and $50,000$ validation images from $1,000$ classes. After pruning, we fine-tune the sparse models for 90 epochs on two NVIDIA V100 GPUs with the settings: Stochastic Gradient Descent (SGD) optimizer with a momentum of 0.9, weight decay of 4e-5 for MobileNets and 1e-4 for ResNet-50, and initial learning rate of 0.1 with a cosine annealing. The data augmentation includes random cropping and horizontal flipping.

We first analyze the influence of filter rearrangement in Sec. 3.3. Table I compares the performance of our 1$\times$N pruning for pruning MobileNet-V2 with and without filter rearrangement. The pruning rate $p$ is set to $50\%$. As can be seen, rearranging filters consistently enhances the accuracy performance in both the top-1 and top-5, even with various block size (N). For example, the top-1 classification accuracy of pruned MobileNet-V2 with 1$\times$16 pruning is increased by $0.381\%$ (69.352% with and 68.971% without filter rearrangement). To dive into a deeper analysis, by rearranging the weight matrix in the output channel dimension, more blocks with larger $\ell_{1}$ norms are preserved after applying our 1$\times$N as validated in Fig. 2(b). These results well validate the effectiveness of filter rearrangement in boosting the performance of pruned models.

We continue the study on our consecutive kernel removal. Recall that consecutive N output kernels with the same input channel index are grouped into one block and our 1$\times$N removes these blocks with a smaller $\ell_{1}$ norm. In Table II, we compare our consecutive kernel removal with two variants including (1) removing kernels randomly and (2) removing kernels with smaller $\ell_{1}$ magnitudes. From these results in Table II, we can see that removing random kernels performs worse than removing smaller magnitude kernels, indicating the importance of preserving larger magnitude weights. Our stronger constraint of block-level magnitude leads to performance increase compared with the kernel-level magnitude. Also, our 1$\times$N merits in its acceleration since the consecutive kernels can be stored continuously in the memory cache. Moreover, the convolution with the inputs can proceed using a parallelized block-wise vectorized operation as analyzed in Sec. 3.4 and verified in the following Sec. 4.3.

In this section, we compare the performance of our proposed 1$\times$N pruning with traditional weight pruning and filter pruning. We show that the advantages of pruning pattern are reflected from two perspectives: $1$) maintaining better accuracy than filter pruning, and $2$) achieving apparent CPUs acceleration compared to weight pruning.

Accuracy Performance. We first study the performance of 1$\times$N sparsity across different networks. Table III displays the pruning results of our 1$\times$N pruning and existing weight pruning and filter pruning using MobileNet-V1/-V2/-V3 with the pruning rate $p$ set to $50\%$. Table III shows that filter pruning suffers the most performance degradation of $5.806\%$, $5.007\%$, $8.171\%$, and $5.143\%$ when pruning MobileNet-V1, V2, V3-small and V3-large, respectively. Such severe performance losses are attributed to its coarse-grained pruning granularity. Consequently, the poor performance barricades the using of filter pruning in practical model deployment. In contrast, due to its fine-grained pruning granularity, weight pruning presents the best performance with top-1 accuracy losses of $0.390\%$, $0.591\%$, $0.849\%$, and $1.383\%$ when pruning MobileNet-V1, V2, V3-small and V3-large, respectively. Despite its ability to maintain high accuracy, weight pruning achieves rare acceleration as analyzed in the following. The poor speedup also disables the using of filter pruning. With respect to our proposed 1$\times$N pruning, we have two observations: $1$) our method well boosts the performance of filter pruning regardless of the block size N. Taking MobileNet-V2 as an example, our 1$\times$4 pruning achieves $69.706\%$ top-1 accuracy, significantly better than $66.730\%$ of filter pruning. Though it is slightly poorer than $71.146\%$ of weight pruning, our 1$\times$4 pruning obtains an apparent speedup as detailed in the following context. $2$) The performance of 1$\times$N pruning degenerates as the block size N increases. The rationale behind this is that a larger N indicates coarser pruning. As analyzed in Sec. 3.2, our 1$\times$N pruning degenerates to weight pruning with a small N and filter pruning with a large N.

Table III also provides the performance comparison when using ResNet-50 as the network backbone with the pruning rate $p=50\%$. We can observe similar phenomena to MobileNets that filter pruning suffers the most performance drops and weight pruning presents best performance while our 1$\times$N provides a trade-off. Besides, our performance decreases as the block size N increases.

Fig. 4 further shows the performance comparison when applying different pruning rates to sparsifying MobileNet-V2 and ResNet-50. We can see that the increasing pruning rate results in decreasing accuracy performance for all methods. However, filter pruning degrades drastically if $p>50\%$. In contrary, our pruning pattern maintains a similar decreasing tendency and close performance to weight pruning even if the pruning rate $p$ is very high¹¹1The raw data for plotting Fig. 4 can be found from our project at https://github.com/lmbxmu/1xN..

CPUs Acceleration. Fig. 5 presents the experimental results, which are conducted to further explore the acceleration capacity of different methods on CPUs-based platforms. In Sec. 3.4, we adopt TVM [39] to compile the pruned model of our pruning pattern. For fair comparison, we also consider TVM compiler for weight pruning and filter pruning. Besides, additional experiments by TFLite compiler [41] are also presented for weight pruning and filter pruning. After model compiling, we respectively deploy the sparse models to obtain network latencies on the mobile platform of Pixel2 equipped with a Snapdragon 835 CPU and the embedded platform of Zeropi equipped with a Cortex-a7 CPU.

From Fig. 5, we observe no speedup from weight pruning, despite its ability to preserve good performance. As analyzed in Sec. 1, weight pruning leads to irregular sparsity that hardly utilizes the vector processing architectures and memory buses. Thus, weight pruning often results in little acceleration and even speed deterioration. Filter pruning leads to the most significant speedups since it does not modify the network structure, so that the pruned network can be well fitted by regular hardware to achieve acceleration. Nevertheless, the severe performance degradation disables the using of filter pruning in the model deployment. In contrast, our 1$\times$N (N=4) pruning achieves noticeable latency reductions across various pruning rates such as $56.04$ms inference savings on Cortex-A7 CPU over weight pruning when $p=50\%$, while maintaining comparable top-1 accuracy performance. Compared with weight pruning and filter pruning, our 1$\times$N pruning shows a better capacity of keeping a trade-off between latency and performance.