Greedy Network Enlarging

The modern CNN backbone architectures usually consists of a stem layer, network body and a head [23, 10, 31]. The main MACs and parameters burdens lie in the network body, as typically the stem layer is a convolutional layer and the head is a fully-connected layer. Thus, in this paper we focus on the scaling strategy of network body. The network body consists of several stages [23], which are defined as a sequence of layers or blocks with the same spatial size. For example, ResNet50 [10] body is composed of $4$ stages with $56\times 56$, $28\times 28$, $14\times 14$, and $7\times 7$ output sizes, respectively.

Scaling up convolutional neural networks is widely used to achieve better accuracy. Network depth, width and input resolution are three key factors for model scaling. Deeper convolutional neural networks capture richer and more complex features, and usually have high performance in contrast to shallow network. With the help of shortcuts, very deep network can be trained to convergence. However, the improvements on accuracy become smaller with the increase of depth. Another direction is scaling the width of network. More kernels mean more fine grained features can be learned. However, the MACs is squared with the width. As a result, the network depth will be constrained and high level features maybe loss. EfficientNet [31] showed that the accuracy quickly saturates when networks become wider. Higher resolution provide rich fine-grained information. In order to match high resolution, more powerful network is wanted. Deeper and wider network can acquire large receptive field and capture fine grained features.

As a result, the network depth, width and resolution are not independent. These three dimension have various combinations. And one unified principle can not acquire the best configuration for all tasks. In this paper, we decompose the network depth, width and resolution into stage depth, width and input resolution. This will maximize the utilization of computations for each stage and the whole network.

Given a base network $\mathbb{N}$ with $L$ stages, width and depth are $\mathbf{w,d}=(w_{1},w_{2},...,w_{L},d_{1},d_{2},...,d_{L})$, and input resolution is $\mathbf{r}$. The objective is to acquire the network architecture with best performance by optimizing the allocation of target MACs $T$:

where $\mathbf{\theta}$ is the trainable parameters of the network. $\mathrm{ACC}_{\mathrm{val}}$ denote the validation accuracy, $T$ is the target threshold of MACs and $\delta$ is used to control the difference between the MACs of the searched model and the target MACs.

Search space. We consider the combinations of input resolution, width and depth of each stage. Suppose a base network with $L$ stages and configurations of width and depth $\mathbf{w,d}$ = ($w_{1}$, $w_{2}$, …, $w_{L}$, $d_{1}$, $d_{2}$, …, $d_{L}$), and input resolution $\mathbf{r}$. For each tieration, the growth rate for width, depth and resolution is $s_{w}$, $s_{d}$ and $s_{r}$, respectively. Under constrained target MACs, we enlarge the width, depth for each stage and the network input resolution step-by-step. For example, ResNet18 contains $4$ stages, if we constrain the search upper bound is $3$ times in both depth and width for each stage, and the growth rate is $1$ for depth and width. The total number of combinations is $128\times 256\times 512\times 1024\times 4\times 4\times 4\times 4\approx 4.4\times 10^{12}$ without considering the variation of input size.

Figure 3 presents our framework. Our intention is to find the optimal allocation of computations by enlarging depth, width and input resolution for each stage under constrained computations. So as to maximize the utilization of MACs, as shown in Eq. 1. For each stage $i$ in the network, we try to find its optimal depth $d_{i}^{\star}$ and width $w_{i}^{\star}$. The optimal input size $r^{\star}$ of resolution is searched to match the specialized width and depth. In the problem1, $\mathbf{r,d,w}$ have discrete values and massive combinations. Deep learning is both time and resource consuming. Due to the extreme complexity, traditional mathematical optimization method is impracticable. So we turn to efficient neural architecture search method.

To simply the search complexity, we first introduce an assumption. Finding the global optimal model is difficult with the massive search space, so we can smooth the target to find a top-performing configuration of target MACs. We introduce an assumption that the top-performing smaller CNNs are a proper subcomponent of the top-performing larger CNNs, as shown in Assumption 1. Resnet [10], VGG [29], EfficientNet [31] etc., fit this assumption perfectly. This assumption enables the idea of efficient search algorithm via greedy network enlarging.

With the above assumption, we transform the optimal network architecture search problem into a series of interrelated single-stage optimal sub-network architecture search problems, and then solve them one by one. Decisions need to be made at each stage to optimize the process. The selection of decisions at each stage depends only on the current state (here, the current state refers to the resolutions, widths and depths of the current stage). When the decision of each stage is determined, a decision sequence is formed, which determines the final solution. The overall algorithm is illustrated in Algorithm 1.

In the algorithm, we use exponential increment of MACs in the process of search. This way make the changes of network more gentaly in contrast to uniform increment. For each iteration in Algorithm 1, in order to find the local optimal architecture configuration, we have to search and evaluate the candidate architectures. This step contains two targets: the first is to find the candidate architectures under limited increase of MACs; the second is to find the local optimal architectures with maximum validation accuracy. In the step of acquiring candidates, we consider the increase of resolution separately, which reduces the candidates. The increase of width and depth of each stage is on the basis of corresponding resolution.

In order to reduce the searched candidates, we take use of proportional control factor to assign the MACs between depth and width for each stage. Specifically, the ratios of MACs between depth and width are in one set $\bm{P}$. Under this setting, we search depth first and then width for each stage. The algorithm is illustrated in Algorithm 2.

To guide the search process, we have to estimate the performance of a given architecture. The most accurate method is to train the candidates on the whole training data and evaluate their performance on validation data. However, this way requires great computational demands in the order of thousands of GPU days. Developing methods for speeding up the process of performance estimation is crucial.

We turn to proxy tasks to estimate performance. Including shorter training times [23], training on a subset of the data [17], on proxy data [40] or using less filters per layer and less cells [25]. These low-fidelity approximations reduce the cost, they also introduce bias in the performance estimation. Proxy data and simplified architecture have large deviation which leads to poor rank preservation.

In this section, we determine the optimal proxy task for performance estimation with empirical experiments. Firstly, we get the proxy sub-dataset by evaluating the performance of different sub-datasets. Secondly, the hyper-parameters of training are acquired with parameter search. Spearman’s rank correlation coefficient $\rho$ is a non-parametric measure of rank correlation, which is used as the measure of proxy task.

For the proxy sub-dataset, we create two sub-datasets ImageNet1000-100 and ImageNet100-500 by random selecting images from ImageNet. To evaluate these datasets, $12$ network architectures with different width, depth and input sizes are generated on the basis of EfficientNet-B0. We train all the $12$ networks and EfficientNet-B0 on the whole train set of ImageNet for $150$ epochs, the Top-1 accuracies on the validation dataset are used as the comparison object. We finetune the $12$ networks for different epochs. Besides, we train the $12$ networks from scratch for few epochs. On ImageNet100-500, the average Spearman value is $\rho=0.16$. On ImageNet1000-100, the average Spearman value is $\rho=0.23$. So we choose ImageNet1000-100 as the proxy sub-dataset. More details are presented in the supplementary materials.

After determining the proxy dataset, we try to improve the correlation between the proxy task and original task by searching the hyper-parameters. $24$ network architectures with different width, depth and input sizes are generated on the basis of EfficientNet-B0. We train all of the $24$ networks on the whole train set of ImageNet for $150$ epochs, the Top-1 accuracies on the validation dataset are used as the comparison object. Two pretrained EfficientNet-B0 models on the ImageNet and ImageNet1000-100 are provided, respectively. The learning rate, mode of learning rate decay and training epochs are considered. Among these hyper-parameter combinations, the top-2 Spearman value $\rho$ is $0.57$ and $0.54$, these values indicate moderate positive correlation. They both use cosine decay method and the initial learning rate is $0.01$ for training $20$ epochs. The difference is that the first use the ImageNet1000-100 pretrained model and the second use the ImageNet pretrained model. More details are presented in the supplementary materials. Figure. 4 presents the consistency of different networks. In the next section, we take use of initial learning rate is $0.01$ and cosine decay for finetuning $20$ epochs on the ImageNet1000-100 pretrained model.

We extensively evaluate our methods on popular classification datasets ImageNet(ILSVRC2012) [5], which contains $1.3$M images and $1000$ categories, the validation set contains $50$K images. On ImageNet, in order to speed up the search process, we create proxy ImageNet1000-100 dataset, which contains $100$K train images and $50$K validation images randomly sampled from ImageNet train set. Two baseline networks are considered: EfficientNet [31] and improved GhostNethttps://gitee.com/mindspore/models/tree/master/research/cv/ghostnet [9].

To accelerate the search process, we set the growth rate of resolution and depth as $s_{r}=8$ and $s_{d}=1$, respectively. For the growth rate of width, we use $s_{w}=2$ for small model and $s_{w}=4$ for large model. The ratios of MACs between depth and width are in one set $\bm{P}=\{0.0,0.1,0.2,\cdots,1.0\}$ The error rate of MACs is $\delta=0.01$. We take use of exponential growth of MACs. We set different number of search iterations $\mathbf{N}$ for small and large models. The finetune method comes from function preserving algorithm [3].

After the process of search is completed, we retrain the acquired network architecture on the whole ImageNet from scratch. The train setting is from timm [34] under its license and EfficientNet [31]. RMSProp optimizer with momentum 0.9; weight decay 1e-5; multi-step learning rate with warmup, initial learning rate 0.064 that decays by 0.97 every 2.4 epochs. Moving average of weight, dropblock [7], random erasing [39] and random augment [4] are used.

ImageNet has noise labels and the method of crop augmentation introduces more noisy input and labels. To prevent this, we use the relabel method [36] to get higher accuracy.

For EfficientNet, we take EfficientNet-B0 as the baseline, and we search the models with MACs similar to EfficientNet-B$1$ to B$4$. Besides, we enlarge GhostNet with the principle of EfficientNet and search GhostNet architectures with greedy search method. For GhostNet, we add Squeeze-and-Excitation [13] module for each block. Table.1 shows the main results and comparison with other networks. The searched models are marked with ’S-’.

GhostNet-B1 and GhostNet-B4 in 1 are obtained by the compounding scale rule of EfficientNet. Their performance is lower in contrast to greedy search methods. This suggests that the rule on EfficientNet is not fit for GhostNet. We need to resample and optimize for new networks to get suitable rules. Besides, the compounding scale principle ignores the difference of stages, which leads to the loss of elaborate adjustment.

In Table.1, Top-1 accuracies of all searched architectures outperform the compound scaling tricks of EfficientNet [31] and RegNet [23]. On $600$M MACs, our searched architectures get $79.91\%$ and $80.08\%$, improve performance $0.81\%$ and $0.97\%$, respectively. On EfficientNet-B2 and B3, our searched EfficientNet architectures achieve $80.92\%$ and $81.98\%$. We search $2$ networks on $4$B MACs level, S-EfficientNet-B4 gets $83.002\%$ and S-GhostNet-B4 gets $83.2\%$, respectively.

The relabel training trick improve the accuracy further. The Top-1 accuracy improves $0.6\%$ to $1.1\%$ on all searched architectures. We achieve a new SOTA 80.87% and 84.3% ImageNet top-1 accuracy under the setting of $600$M and $4.4$B MACs, respectively. All searched network architectures are presented in the supplementary materials.

Figure 5 is used specifically to show the changes of accuracy and input resolution of the search process. With increase of MACs, the resolution rises wavily, which verifies the role of dynamic search. The accuracy increases slowly and steadily.

Furtherly, the schematic diagram of greedy search for EfficientNet-B1 is shown in Figure 6. Under constrained MACs, we show the candidate network architectures. The green box means the best architecture in current iteration, and the gray box are discarded. Besides, the best architecture of each iteration are delivered to the later iterations.