Unified Chinese License Plate Detection and Recognition with High Efficiency

Due to the importance of LP detection and recognition, researchers built and published a number of LP datasets. ReId [7] is a dataset for license plate recognition with 76k images gathered from surveillance cameras on highway toll gates. Caltech [8] and Zemris [9] collected over 600 images from the road and freeways with high-resolution cameras. Hsu et al. [11] presented a dataset for applications of access control, traffic law enforcement and road patrol. Gonçalves et al. [12] proposed Sense SegPlate Database to evaluate license plate character segmentation problem. Laroca et al. [13] provided a dataset that includes 4,500 fully annotated images from 150 vehicles in real-world scenarios. These datasets strongly support the researches of LP detection and recognition methods.

However, LPs in different countries and regions are usually not the same. The mentioned datasets mostly contain LPs that only include alphanumeric characters. In some countries or regions, such as Chinese mainland regions, Japan and South Korea, the LPs contain some special characters. Therefore, it is still significant to build new datasets for these LPs. Among them, mainland Chinese LP detection and recognition are one of the important tasks which require a large amount of data. To meet the demand, Chinese LP datasets were proposed. Zhou et al. [4] collected 220 LP images where the LPs were of little affine distortion. Yuan et al. [14] presented a dataset that contains vehicle images captured from various scenes under diverse conditions. Zhang et al. [15] proposed a dataset that contains 1,200 LP images from all 31 provinces in mainland China. CCPD [10] is a large and comprehensive dataset with about 290k images that contain plates with various angles, distances, and illuminations. Though they made important contributions to the progress of detection and recognition methods, there are still not enough large and representative datasets.

Owing to the successes of text detection and recognition, LP processing is also well developed. There are methods with end-to-end frameworks, which achieved excellent performance. Zhang et al. [18] integrated LP detection, tracking, and recognition into a unified framework via deep learning. Silva et al. [19] proposed to identify the vehicle and the LP region using two passes on the same CNN and then to recognize the characters using a second CNN. Kessentini et al. [20] presented a two-stage network to achieve the detection and recognition of LPs. In [21], a light CNN was proposed for detection and recognition, which achieved real-time efficiency.

As for Chinese LPs, there are also excellent end-to-end frameworks. Laroca et al. [22] proposed a unified approach for LP detection and layout classification to improve the recognition results. Qin et al. [23] proposed a unified method that can recognize both single-line and double-line LPs in an end-to-end way without line segmentation and character segmentation. However, Chinese LP processing is still challenging due to the large number of categories of Chinese characters. Meanwhile, LP processing under unconstrained scenarios also faces many problems. Therefore, it is still valuable to propose a new end-to-end framework for Chinese LP detection and recognition.

Because CRPD is presented as a supplement for existing datasets, special attention is paid to the diversity of data. The images are mainly captured on electronic monitoring systems, including vehicles that are running, turning, parked, or far away which may cause blur and rotation. The scene includes day and night and different weathers. Quite a part of the images contains more than one LP from different provinces and cities, and there are LPs of special vehicles involved, such as coach cars, police cars, and trailers, whose LPs will contain some special characters. The dataset includes three sub-datasets according to the numbers of LPs: CRPD-single, CRPD-double, and CRPD-multi, as shown in Figure 1. CRPD-single contains images with only one LP, CRPD-double contains images with two LPs, and CRPD-multi contains images with three or more LPs.

There are totally about 25k images for training, 6.25k for validating, and 2.3k for testing. In CRPD-single, there are 20k images for training, 5k for validating, and 1k for testing. In CRPD-double, there are 4k images for training, 1k for validating, and 1k for testing. In CRPD-multi, there are 1k images for training, 0.25k for validating, and 0.3k for testing.

The annotations consist of three parts. The first is LP content which includes numbers, Chinese and English characters. There are some LPs that are too small or seriously blurred whose content is unidentifiable, and they are also annotated while the unrecognizable characters are replaced with a special one. The second is the coordinate of four vertices of the LPs. The last is the LP type, including blue (small cars), yellow and single line (front of large cars), yellow and double lines (back of large cars), and white (police cars).

CRPD provides more than 30k LP images with annotations. Though it is not the largest dataset in current frequently used Chinese datasets, some characteristics of CRPD will be helpful for training a robust model. Existing datasets mostly contain single and focused LP in one image, as shown in Figure 2. In some cases, such as electronic toll payment or parking managing systems, the data is highly effective. But when dealing with data for traffic monitoring systems, suspect car tracking, or vehicle flow measuring, images with more LPs will be required. CRPD aims to fill up this deficiency, so we paid special attention to the number of LPs, as shown in Figure 3.

Also, CRPD has some other advantages for building a robust model. To illustrate them, we compare with CCPD [10], EasyPR [24], ChineseLP [4] and CLPD [15] in some aspects, which are shown in Tables 1, 2 and 3.

The first is the number of LPs. Images in CCPD [10] and CLPD [15] contain one LP, which can better indicate the detection Precision of a network. But as noted above, this may restrict the scenarios where it can be used. In comparison, EasyPR [24], ChineseLP [4] and our CRPD have better compatibility in this aspect.

The second is the status of vehicles. CCPD [10] contains images captured from parking lots, and they are more interested in LPs in various circumstances with different illumination, rotation, and blur. Therefore, the vehicle status is not focused. Our CRPD concentrates on the capability to deal with a variety of vehicles, so there is better coverage on vehicle status.

The last is the vehicle types. Though there are not a number of special vehicles on the road, the detection and recognition of them are still of great importance. Thus, we paid attention to the LPs of special vehicles to ensure the capability of the network trained by CRPD to deal with these LPs. Some images of them are shown in Figure 4. Altogether, our CRPD covers most a variety of common scenes. It is a strong supplement to Chinese LP datasets either used as training or evaluating data.

However, there are also some limitations of CRPD. First, the location of each LP character is not annotated, which restricts the applications of object detection and data augmentation. Meanwhile, it is also difficult to detect each character because it is very small in the perspective of electronic monitoring systems. Second, the LPs are all on the vehicles. The actual LPs can be held in hand or placed on the ground in some circumstances, and the detection and recognition of these LPs are also useful.

The first stage is the LP detection step. As noted above, there are some obstacles for LP detection networks to reach a balance between accuracy and efficiency. Considering the trade-off, we utilize our previous STELA [16], a totally real-time detector, as the basis of our detection branch. The detection network is implemented on RetinaNet [25] and utilizes Feature Pyramid Network (FPN) [26] to construct a rich, multi-scale feature pyramid from a single resolution input image. It consists of three portions: anchor classification, rotated bounding box regression, and anchor refining.

The recognition step is the second stage of the LP processing tasks. Because most of the networks for recognition achieved high efficiency, we simply utilize modules based on CRNN [28] to achieve the recognition. There are three parts in the modules: the convolutional layers, the recurrent layers, and the transcription layer.

In consideration that the backbone of our network has already extracted critical features of the input images, to avoid redundant computation, we utilize the processed feature maps as the input of the recognition branch. In order to effectively deal with rotated plates, RRoIAlign [17] that can crop the feature map with a rotated box, is applied to crop the maps according to the groundtruth boxes of LPs, and the cropped size is $8\times 25$. While training, we consider that the feature maps may not be processed well when the predicted boxes are not accurate, so the feature maps will also be cropped with both groundtruth boxes and predicted boxes with a score higher than 0.9.

And because the input is feature maps, we remove the first three convolutions of the original convolutional layers of CRNN [28] to avoid the over-fitting problems. And in order to reach better accuracy, we refer to our another previous work [29], and replace two convolution layers with deformable convolution layers and add four residual blocks in this branch. The deformable convolution layers have an adaptive receptive field that can better cover the text area, as shown in Figure 8.

The architecture of the convolutional layers is shown in Figure 5. The recurrent layers are based on bidirectional LSTM [30], and the transcription component is based on CTCLoss [31]. The total loss function is a weighted loss

where $\lambda_{det}$ and $\lambda_{rec}$ are constants that indicate the strength of the detection and recognition modules. And in our training, the value of $L_{rec}$ is one-magnitude-order larger than $L_{det}$, so to keep a balance, we set them to 1 and 0.1, respectively.

The training and testing datasets are from CCPD [10], EasyPR [24], and our CRPD. The input images are resized to $640\times 640$ with three channels. In the training stage, the optimizer of the network is Adam [18], the batch size is set to 32, and the learning rate is 1e-4. The network is trained for 35000 iterations which consume about 10 hours. The proposed method is implemented by PyTorch [32]. The experiments are carried on a platform with Intel Xeon(R) E5-2630 v3 CPU and a single NVIDIA TITAN RTX GPU.

To demonstrate the effectiveness of the methods, we utilize the protocols described in [10] to evaluate the models. The detection will be considered as a match if it overlaps a ground truth bounding box by more than 60% and the words match exactly. Then Recall, Precision, and F-score are calculated with

where $TP$ represents the number of positive objects that are predicted as positive and the words match, $FP$ represents the number of negative objects that are predicted as positive, and $FN$ represents the number of negative objects that are predicted as negative.

To demonstrate that the unified architecture brings some improvements and our scheme reaches the best performance, we evaluate the proposed components. Firstly, we test the effectiveness of our unified network. As comparisons, cascaded STELA [16] and CRNN [28] models are utilized. The two networks are trained respectively, and the CRNN [28] model will process the original images cropped in the light of the boxes predicted by STELA [16]. All the models are trained on CRPD and evaluated on CCPD [10], EasyPR [24], and CRPD. From Table 4, we see that our network achieves a few improvements. A better result is reached on CRPD because the training and testing data have the same distribution. The LPs in CCPD [10] have a larger size, so the model trained on our CRPD, which contains LPs with small sizes, is not able to reach the best performance.

Then as we utilize the predicted boxes of the detection modules to crop the feature maps in the training stage, we evaluate the effectiveness of them with different scores. In Table 5, it is obvious that utilizing predicted boxes will bring an improvement on the Recall because more incomplete LPs will be detected, with a little descend on Precision. Considering the recognition branch will also become more robust with this mechanism, it will make the network able to better deal with inaccurate boxes. But because Chinese LPs usually have seven characters, a box with a score lower than 0.86 may cut a whole character off. Thus, there must be a deterioration of the performance when using boxes with scores only higher than 0.85. And the number of boxes with scores higher than 0.95 is less, which causes a smaller batch size for recognition modules, so we choose 0.9 as the threshold finally.

The way to crop the feature maps may also bring different results of the recognition modules. We compare RoIPool [27], RoIAlign [33], and RRoIAlign [17], and the results are shown in Table 6. There is no obvious difference between the first two methods, and we think the reason is that LPs are not small objects, so some slight errors will not influence much. RRoIAlign [17] can better handle the work because the recognition branch will have great progress with consideration of the rotation degrees.

In the recognition branch, we crop the feature maps yield by different layers of the FPN [26] components to explain why we utilize those from the third layer. Though it seems to utilize the feature maps from the third layer is optimal in Table 7, but we consider it is because most of the LPs in the images of CRPD have sizes which match the anchors in the third layer. In the fifth or deeper layers, the anchors are mappings of some huge boxes in the original images, but most of the LPs have a small size.

Finally, to demonstrate that the deformable layers and residual blocks in the recognition branch are effective, ablations are involved, as shown in Table 8. In our metrics, only when the content matches exactly, the result will be regarded as correct. Therefore, deformable convolution brings an improvement in recognition, and the Recall and Precision are both improved.

To demonstrate that our method has reached a satisfactory performance, we make some comparisons with other methods. We involve EasyPR [24], SSD [34], YOLO-v3 [35], YOLO-v4 [36], Scaled YOLO-v4 [37] and Faster-RCNN [38]. A CRNN [28] model is appended to achieve recognition. We utilize cascaded STELA [16] and CRNN [28] as our baseline in experiments. From Table 9, it can be observed that EasyPR [24] cannot treat LPs in CRPD well because it depends on manually designed features. SSD [34], YOLOv3 [35], YOLO-v4 [36], Scaled YOLO-v4 [37] and Faster-RCNN [38] are not specially designed for LPs or text, but the results are still competitive. Our method reaches the best on all the sub-datasets, which proves the effectiveness.

Because Xu et al. [10] evaluated their method on different circumstances, for fair comparisons, experiments of our method on these datasets are also utilized. And as the other methods are trained on CCPD [10], we also utilize it as the training data of our method in this comparison. Because images in CCPD [10] only contain one LP in one image, only Precision is involved when the Recall is not considered. Following the experiments of Xu et al. [10], Cascade classifier [39], SSD300 [34], YOLO9000 [40], Faster-RCNN [38] are involved as the detector with Holistic-CNN [7] as the recognition model. And end-to-end methods TE2E [41] and RPNet [10] are utilized for comparisons. We also involve YOLO-v4 [36] and Scaled YOLO-v4 [37] with CRNN [28] to report more results. Our baseline which consists of STELA [16] and CRNN [28] are also involved. From Table 10, it can be observed that our method has reached the best in most circumstances, and the efficiency is also competitive.

Finally, some detection and recognition results on CRPD are shown in Figure 9 and 10. In the illustrations, we see that most LPs that are recognized incorrectly are also hard to distinguish by humans. Our proposed method is still effective in most cases, and the performance is competitive.