PP-PicoDet: A Better Real-Time Object Detector on Mobile Devices

Manually Designed Backbone. Based on many experiments, we find that ShuffleNetV2 is more robust than other networks on mobile devices. To further improve the performance of ShuffleNetV2, we follow some methods of PP-LCNet[36] to enhance the network structure and build a new backbone, namely Enhanced ShuffleNet (ESNet). Figure 3 describes the ES Block of ESNet in detail. The SE module[37] does a good job of weighting the network channels for better features. Therefore, we add SE modules to all blocks. Like MobileNetV3, the activation functions for the two layers of the SE module are ReLU and H-Sigmoid, respectively. Channel shuffle provides the information exchange of ShuffleNetV2 channels, but it causes the loss of fusion features. To address this problem, we add depthwise convolution and pointwise convolution to integrate different channel information when the stride is 2 (Figure 3a). The author of GhostNet[38] proposes a novel Ghost module that can generate more feature maps with fewer parameters to improve the network’s learning ability. We add the Ghost module in the blocks with stride set to 1 to further enhance the performance of our ESNet (Figure 3b).

Neural Architecture Search. At the same time, we present the first effort on one-shot searching for object detectors. Object detectors, equipped with high-performance backbones for classification, might be sub-optimal due to the gap between different tasks. We do not search for a better classifier, but train and search the detection supernet directly on the detection datasets, which leads to significant computational savings and optimization of detection instead of classification. The framework simply consists of two steps: (1) training the one-shot supernet on detection datasets, (2) architecture search on the trained supernet with an evolutionary algorithm (EA).

For convenience, we simply use channel-wise search for backbone here. Specifically, we give flexible ratio options to choose different channel ratios. We choose the ratio randomly and coarsely in [0.5, 0.675, 0.75, 0.875, 1]. For example, 0.5 represents that the width is scaled by 0.5 of the full model. The channel numbers divisible by 8 can improve the speed of inference time on hardware devices. Therefore, instead of using the channel numbers in the original model, we first trained the full model with channel numbers [128, 256, 512] for each stage block. All ratio options can also keep the channel numbers divisible by 8. The chosen ratio works on all prunable convolutions in each block. All output channels are fixed as the full model. To avoid tedious hyper-parameter tuning, we fix all original settings in the architecture search. For the training strategy, we adopt the sandwich rule to sample the largest (full) and the smallest child model and six randomly sampled child models for each training iteration. There are no more additional techniques adopted in the training strategy, such as distillation, since different techniques perform inconsistently for different models, especially for detection tasks. Finally, the selected architectures are retrained on ImageNet dataset[39] and then trained on COCO[40].

We use the PAN[41] structure to obtain multi-level feature maps and the CSP structure for feature concatenation and fusion between the adjacent feature maps. The CSP structure is widely used in the neck of YOLOv4[2] and YOLOX[4]. In the original CSP-PAN, the channel number in each output feature map is kept the same as the input from the backbone. The structures with large channel numbers have expensive computational costs for mobile devices. We address this problem through making all channel numbers in all feature maps equal to the smallest channel number by $1\times 1$ convolution. Top-down and bottom-up feature fusion are then used through the CSP structure. The scaled-down features lead to lower computation costs and undamaged accuracy. Furthermore, we add a feature map scale to the top of CSP-PAN to detect more objects. At the same time, all convolutions except $1\times 1$ convolutions are depthwise separable convolution. Depthwise separable convolution expands the receptive field through $5\times 5$ convolution. This structure brings a considerable increase in accuracy with much fewer parameters. The specific structure is shown in the Figure 2.

In the detector head, we use depthwise separable convolution and $5\times 5$ convolution to expand the receptive field. The numbers of depthwise separable convolution can be set to 2, 4, or more. The overall network structure is shown in the Figure 2. Both the neck and the head have four scale branches. We keep the channel numbers in the head consistent with the neck module and couple the classification and regression branches. YOLOX[4] uses a decoupled head with fewer channel numbers to improve accuracy. Our coupled head performs better without reducing the channel numbers. The parameters and the inference speed are almost the same as the decoupled head.

The label assignment of positive and negative samples has an essential impact on the object detectors. Most object detectors use fixed label assignment strategies. These strategies are straightforward. RetinaNet[44] directly divides positive and negative samples by the IoU of the anchor and ground truth. FCOS[5] takes the anchors whose center point is inside the ground truth as positive samples, and YOLOv4[2] and YOLOv5[3] select the location of the ground truth center point and its adjacent anchors as positive samples. ATSS[26] determines the positive and negative samples based on the statistical characteristics of the nearest anchors around the ground truth. The above-mentioned label assignment strategies are immutable in the global training process. SimOTA is a label assignment strategy that changes continuously with the training process and achieves good results in YOLOX[4].

We use SimOTA dynamic label assignment strategy to optimize our training process. SimOTA first determines the candidate area through the center prior, and then calculates the IoU of the predicted box and ground truth in the candidate area, and finally obtains parameter $\kappa$ by summing the n largest IoU for each ground truth. The cost matrix is obtained by directly calculating the loss for all predicted boxes and ground truth in the candidate area. For each ground truth, the anchors corresponding to the smallest $\kappa$ loss are selected and assigned as positive samples. The original SimOTA uses the weighted sum of CE loss and IoU loss to calculate the cost matrix. To align the cost in SimOTA and the objective function, we use the weighted sum of Varifocal loss and GIoU loss for cost matrix. The weight of GIoU loss is the $\lambda$, which is set to 6 as shown to best through our experiments. The specific formula is:

In the detector head, for classification, we use Varifocal loss to couple classification prediction and quality prediction. For regression, we use GIoU loss and Distribution Focal Loss. The formula is as follows:

In all the above formulas, $loss_{vfl}$ means Varifocal Loss, $loss_{giou}$ means GIoU loss, $loss_{dfl}$ means Distribution Focal Loss.

In recent years, more activation functions have emerged that go beyond ReLU. Among these activation functions, H-Swish, a simplified version of the Swish activation function, is faster to compute and more mobile-friendly. We replace the activation function in detectors from ReLU to H-Swish. The performance improves significantly while keeping the inference time unchanged.

Different from linear step learning rate decay, cosine learning rate decay is exponentially decaying the learning rate. Cosine learning rate drops smoothly, which benefits the training process, especially when the batch size is large.

Too much data augmentation always increases the regularization effect and makes the training more difficult to converge for lightweight models. So in this work, we only use random flip, random crop and multi-scale resize for data augmentation in training.

For training, we use stochastic gradient descent (SGD) with momentum of 0.9 and weight decay of 4e-5. The cosine decay learning rate scheduling strategy is adopted with initial learning rate of 0.1. The batch size is 80x8 by default on 8x32G V100 GPU devices. We train 300 epochs, which costs 2 to 3 days. All experiments are trained on COCO-2017[40] training set with 80 classes and 118k images, and are evaluated on COCO-2017 validation set with 5000 images using the standard COCO AP metric of a single scale. Exponential Moving Average (EMA) heavily utilizes recent information and maintains long-term influence intuitively. Lightweight models are more likely to trap in local optima and are harder to converge. Therefore, we introduce a mechanism that works like regularization, named Cycle-EMA, to reset the content of the history, governed by a forget step.

For the architecture search task, the settings of all hyper-parameters and datasets for supernet training are the same as the original model, detailed in the following section. We use L2 norm gradient clipping to avoid exploding gradients. Another difference is that we train eight candidates for each step since our search space is large.

The results of all our ablation experiments are shown in Table 2. All experimental results are on COCO-2017 validation set.

CSP-PAN. We first get our base model similar to NanoDet, the backbone adopts ShuffleNetV2-1x, the neck adopts PAN without convolution, the loss adopts standard GFL[7] loss, and the label assignment strategy adopts ATSS[26]. All activation functions use LeakyRelu. The resulting mAP (0.5:0.95) is 25.3. Further, we adopt the CSP-PAN structure. The feature map scale is 3. The mAP (0.5:0.95) is increased to 28.1. Finally, we add a feature map scale to the top of CSP-PAN. Just like the final structure of our CSP-PAN, the number of parameters increases by less than 50K. The mAP (0.5:0.95) is further improved to 29.1. Results are shown in Table 2.

Loss. We compare the effects of Varifocal Loss (VFL) and Quality Focal Loss (QFL) under the same configuration in the previous section. The two are close, and the effect of Varifocal Loss is only slightly better than that of Quality Focal Loss. Replacing QFL with VFL, the mAP (0.5:0.95) is improved to 29.2 from 29.1. Results are shown in Table 2.

Label Assignment Strategy. Under the same configuration in the previous section, we replace ATSS with the original SimOTA and our modified SimOTA. We find that the larger the n, the worse the effect. The parameter n is then set to 10. The performance of ATSS is almost the same as the original SimOTA. The mAP (0.5:0.95) of our modified SimOTA achieves 30.0. Results are shown in Table 2.

We further compare the effect of SimOTA when Varifocal Loss and GIoU loss have different weights. We change the $\lambda$ of formula (1) to perform ablation experiments. The results are shown in Table 3. When the weight of GIoU loss is 6, the best result is obtained.

ESNet Backbone. We compare the performance of ESNet-1x and the original ShuffleNetV2-1.5x on ImageNet-1k. Table 4 shows that with less inference time, ESNet achieved higher accuracy.

We also compare the performance of the original model and our searched model, and the results are shown in Table 5. The searched model under latency constraint only decreases by 0.2% mAP with mobile CPU inference time speed up by 41.5% (54.9% using Paddle Lite). We replace the backbone of the detector mentioned above with ESNet-0.75x, the number of parameters is reduced by nearly 200K, mAP (0.5:0.95) is finally to 29.7. Results are shown in Table 2.

H-Swish Activation Function. Finally, we replace the LeakyRelu with H-Swish for all activation functions, mAP (0.5:0.95) is finally increased to 30.6. Results are shown in Table 2.

From Table 1, we can see that our models far exceed all YOLO models in accuracy and speed. These achievements are mainly owing to the following improvements: (1) our neck is lighter than the neck of the YOLO series, so that backbone and head can be assigned more weights. (2) the combination of our Varifocal loss dealing with class imbalance, dynamic and learnable sample assignment, and regression method based on FCOS performs better in lightweight models. With the same amount of parameters, both the mAP and latency of PP-PicoDet-S surpass the YOLOX-Nano and NanoDet. Both the mAP and latency of PP-PicoDet-L exceed the YOLOv5s. Due to the more efficient convolution operator optimization by assembly language, we find that the inference testing performance of our models is even better when using Paddle Lite than using NCNN. In conclusion, our models are ahead of the SOTA models to a large extent.