RMNet: Equivalently Removing Residual Connection from Networks

Since AlexNet Krizhevsky et al. (2012) came out, the state-of-the-art CNN architecture has become deeper and deeper. For example, AlexNet only has 5 convolutional layers, it is soon extended to 19 and 22 layers by VGG network Simonyan & Zisserman (2015) and GoogLeNetSzegedy et al. (2015; 2016; 2017), respectively. However, deep networks that simply stack layers are hard to train because of the gradient vanishing and exploding problem — as the gradient is back-propagated to earlier layers, repeated multiplication may make the gradient infinitely small or large. This problem has been largely addressed by the normalized initialization LeCun et al. (2012); Glorot & Bengio (2010); Saxe et al. (2013); He et al. (2015) and intermediate normalization layers Ioffe & Szegedy (2015), which enable networks with tens of layers converging. Meanwhile, another degradation problem has been exposed: with the increase of the network depth, accuracy gets saturated and then degrades rapidly. ResNet He et al. (2016a; b) addresses the degradation problem and achieves 1K+ layers models by adding a residual connection from the input of a block to the output. Instead of hoping each stacked layer directly fit a desired underlying mapping, ResNet lets these layers fit a residual mapping. When the identity mapping is optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers. As ResNet gains more and more popularity, researchers propose many new architectures, e.g. Hu et al. (2018); Li et al. (2019); Tan & Le (2019; 2021), based on ResNet and interpret the success from different aspects.

However, ResDistill Li et al. (2020) points that the residual connections in ResNet-50 account for about 40 percent of the entire memory usage on feature maps, which will slow down the inference process. Besides, residual connections in the network are not friendly for ‘network pruning’ Ding et al. (2021b). By contrast, the VGG-like model, also called the plain model in this paper, has only one path and is fast, memory economical, and parallelization friendly. RepVGG Ding et al. (2021b) proposes a powerful approach to remove residual connections via re-parametrization at inference. Specifically, RepVGG adds 3$\times$3 convolution, 1$\times$1 convolution, and identity together for training. The BN layer was added at the end of each branch and the ReLU was added after the addition operation. During training, RepVGG only needs to learn the residual mapping, while in the inference stage, re-parameterization was utilized to transform the basic block of RepVGG into a stack of 3 $\times$ 3 convolution layers plus ReLU operation, which has a favorable speed-accuracy trade-off compared to ResNet. However, we find that the performance of RepVGG suffers severe degradation when the network goes deeper, we provide experiments to verify this observation in Section 3.

In this paper, we introduce a novel approach named RM operation, which can remove the residual connection with non-linear layers inside it and keep the result of the model unchanged. RM operation reserves the input feature maps by the first convolution layer, BN Layer and ReLU Layer, then merge them with the output feature maps by the last convolution in the ResBlock. With this method, we can equivalently convert a pre-trained ResNet or MobileNetV2 to an RMNet model to increase the degree of parallelism. Besides, the architecture of RMNet makes it friendly for pruning since there are no residual connections. We summarized our main contributions as follows:

• •

  We find that removing residual connection by re-parameterization methods has its limitation, especially when the depth of the model is large. It lies in the non-linear operation that can not be put inside of residual connection for re-parameterization.

• •

  We propose a novel method named RM operation which can remove residual connection across non-linear layers without changing the output by reserving the input feature maps and merging them with the output feature maps.

• •

  With RM operation, we can convert the ResBlocks to a stack of convolutions and ReLUs, which help get a deeper network without residual connection and make it very friendly for pruning.

Advantage of Residual Networks over Plain Model. He et al. He et al. (2016a) introduce ResNet for addressing the degradation problem. In addition, the gradient of layers in PreActResNet He et al. (2016b) does not vanish even when the weights are arbitrarily small, which leads to nice backward propagation properties. Weinan (2017) shows the connection between dynamical systems and deep learning. Lu et al. (2018) regards ResNet as an Euler discretization of Ordinary Differential Equations (ODE). NODEs Chen et al. (2018) replace the residual block with neural ODEs for better training extremely deep networks. Balduzzi et al. (2017) identifies shattered gradients problems making the training of DNNs difficult. They show that the correlation between gradients in the standard feedforward network decays exponentially with depth. In contrast, the gradients in ResNet are far more resistant to shattering, decaying sublinearly. Veit et al. (2016) shows that residual networks can be seen as a collection of many paths of differing lengths. From this view, $n$-blocks’ ResNet have $O(2^{n})$ implicit paths connecting input and output, and that adding a block doubles the number of paths.

DNNs Without Residual Connection. A recent workOyedotun et al. (2020) combined several techniques including Leaky ReLU, max-norm, and careful initialization to train a 30 layer plain ConvNet, which could reach 74.6% top-1 accuracy, 2% lower than their baseline. A mean-field theory-based method Xiao et al. (2018) is proposed to train extremely deep ConvNets without branches. However, 1K-layer plain networks only achieve 82% accuracy on CIFAR-10. Balduzzi et al. (2017) use a ”looks linear” initialization method and CReLUs Shang et al. (2016) to train a 198 layer plain model to 85% accuracy on CIFAR-10. Though the theoretical contributions were insightful, the models were not practical. One can get DNNs without residual connection in a re-parameterization manner: re-parameterization Ding et al. (2019; 2021a) means using the params of a structure to parameterize another set of params. Those approaches first train a model with residual connection and remove residual connection by re-parameterization when inference. DiracNet Zagoruyko & Komodakis (2017) uses the addition of an identity matrix and convolutional matrix for forwarding propagation, the parameters of convolution need only to learn the residual function like ResNet. After training, DiracNet adds the identity matrix to the convolutional matrix and uses the reparameterized model for inference. However, DiracNet can only train up to 34 layer plain model which has 72.83% accuracy on ImageNet. RepVGG Ding et al. (2021b) deploys residual neural network only at the training time. At the inference, RepVGG can transform the residual block into a plain module consisting of a stack of 3 $\times$ 3 convolution and ReLU via re-parameterization. The difference between DiracNet and RepVGG is each block in DiracNet has two branches (identity without BN and 3 $\times$ 3 ConvBN) while RepVGG has three branches (identity with BN, 3 $\times$ 3 ConvBN, and 1 $\times$ 1 ConvBN). However, those re-parameterization methods that used commutative property can only apply to linear layers, i.e., the non-linear layers have to be outside of the residual connection, which limits the potential of neural networks for large depths.

Filter Pruning: Filter pruning is a promising solution to accelerate CNNs. Numerous inspiring works prune the filters by evaluating their importance. Heuristic metrics are proposed, such as the magnitude of convolution kernels Li et al. (2017), the average percentage of zero activations (APoZ) Hu et al. (2016). There also exists some work to let networks automatically select important filters. For example, Liu et al. (2017a) sparsify the weights in the BN layer to automatically find which filters contribute most to the network performance. However, For ResNet-based architecture, the existence of residual connection limits the power of pruning since the dimension of the input and output through the residual connection must keep the same. Thus pruning ratio of ResNet is not larger than the ratio of the plain model. Since RM operation can equivalently transform ResNet to a plain model, the transferred model (RMNet) also has a great advantage on pruning.

We depict the basic blocks used in ResNet and RepVGG in Figure 1. In ResBlock (Figure 1(a)), two ReLUs are inside and outside of the residual connection, respectively. While in RepBlock (Figure 1(b)), since re-parameterization is based on the commutative law of multiplication, two ReLUs have to be both outside of the residual connection. One can build RepVGG from a basic ResNet architecture by replacing each ResBlock with two RepBlocks. Next, We analyze why RepVGG can not be trained very deep like ResNet from both forward and backward passes:

$\bullet$ forward path: Veit et al. (2016) assumes that the success of ResNet can be contributed to the ”model ensemble”. we can regard ResNet as an ensemble of many paths of different lengths. Thus, $n$-blocks’ ResNet have $O(2^{n})$ implicit paths connecting input and output. However, unlike ResNet that two branches in the block are separable and can not be merged, multi-branches in RepVGG can be represented by one branch, which can be shown as:

where $Conv^{{}^{\prime}}$ is the merged convolution of each $Conv$ in the branches. Thus RepVGG does not have the implicit ”ensemble assumption” of ResNet, and the representation gap between RepVGG and ResNet increases as the number of blocks raises.

$\bullet$ backward path: Balduzzi et al. (2017) analyzes ”shattered gradients problem” in deep neural networks. The correlation of gradients behaves like ‘White Gaussian Noise’ when there are more ReLUs in the backward path. Suppose ResNet and RepVGG both have $n$ layers. From Figure 1(a), Information in ResNet can pass through the residual without going through inside ReLU in each block. However, each ReLU in RepVGG is located in the main path. Thus the number of ReLUs for ResNet in the backward path is $\frac{n}{2}$, while the number of ReLUs for RepVGG in the path is $n$ which suggests that the gradients in ResNet are far more resistant to shattering when the depth is large, leading to better performance than RepVGG.

In Figure 2, we examine how network depth influences network performance on ResNet and RepVGG. The datasets we use are CIFAR-10/100. For a fair comparison, the same hyper-parameters are deployed for each method: mini-batch size (256), optimizer (SGD), initial learning rate (0.1), momentum (0.9), weight decay (0.0005), number of epochs (200), learning rate decay 0.1 at every 60 epochs.

From Figure 2, as the depth increases, ResNet can get better accuracy, which is consistent with the previous analysis. In contrast, the accuracy of RepVGG will decrease, e.g. compared to ResNet-133 which achieves 79.57% accuracy on CIFAR-100, RepVGG-133 only has 41.38% accuracy.

Table 1 shows the performance of RepVGG and ResNet on ImageNet dataset. We can see that, under the same network structure, ResNet-18 is 0.5% higher than RepVGG-18, and ResNet-34 is 0.8% higher than RepVGG in terms of top-1 accuracy. Thus, RepVGG increases the speed at the cost of losing representation ability.

This section aranges as follows: in Section 5.1, we verify the effectivenss of RMNet on network pruning task. in Section 5.2, we show RM Operation can help train deeper RepVGG; in Section 5.3, we show RMNet is applicable for light-weight models; in Section 5.4. we show RMNet can achieve better speed-accuracy tradeoff on CIFAR10/100 and ImageNet datasets;

The speed of each model is tested on a Tesla V100 GPU with a batch size of 128 by first feeding 50 batches to warm the hardware up, then 50 batches with time usage recorded.

In this paper, We propose RM operation to remove residual connections and perform RM operation to convert ResNet and MobileNetV2 into plain models (RMNet). RM operation allows input feature maps to pass through the block while reserving their information and merges all the information at the end of each block, which can remove residual connection without changing original output. Experiments have shown that RMNet can achieve a better speed-accuracy trade-off and is very friendly for network pruning.

There are some future directions to be considered: 1) Analyzing the residual connection on the Transformer based architectures and removing such connections via RM operation. 2) Searching networks with less residual connection with NAS and converting them to efficient RMNets.