Accuracy Improvement of Object Detection in VVC Coded Video Using YOLO-v7 Features
^††thanks: These research results were obtained from the commissioned research (No.05101) by National Institute of Information and Communications Technology (NICT), Japan.

Video compression technology is necessary for sending and receiving video within limited communication resources. Research and development of technologies to encode video for human vision has been conducted for a long time, and many video coding standards have been created. Among them, VVC is the latest video coding standard, the first version of which was completed in July 2020 [3]. Similar to previous coding methods, it combines intraframe and interframe prediction and operates based on hand-crafted algorithms.

On the other hand, research on video coding using neural networks has also been active in recent years. For example, there are video compression models using RNN [6], image compression models using GAN [7], and interframe prediction models [8, 9]. CNN-based video coding methods for both intraframe and interframe prediction are also proposed [10]. Furthermore, the latest models show comparable coding performance to that of VVC [11].

Many studies have been conducted to improve the quality of the coded video by post-processing [12, 13, 14, 15] using neural networks. In VVC and HEVC, block-based motion compensation and transformation is performed. This method causes block noise in coded images, which deteriorate the quality of these images. To reduce the block noise, a deblocking filter is employed. However, removing the noise is not always possible. Therefore, a model for removing coding noise using neural networks has been studied.

As the accuracy of image recognition improves, there are increasing opportunities for machines to perform video analysis [16]. For this reason, coding techniques for image recognition are attracting attention, and the movement towards standardization is accelerating. The information in video required for machines is considered to be different from the one for humans to view images [1]. Considering the difference, it is important to capture the characteristics of the information in video, which is necessary for image recognition. Therefore, video information should be extracted depending on the purpose of the video.

Many studies have been conducted to extract the video information that is necessary for object detection, which is one of the popular image recognition tasks. One video coding method extracts the information necessary for YOLO9000 [17]. In this model, the video is input to YOLO9000 before coding, and the obtained features are used when the video is encoded [18]. The features represent the part of the image that YOLO9000 places an attention on. The encoding model allocates more bits to this part of the image to decode images that are useful for object detection. The other method integrates video coding model and R-CNN [19]. In this coding model, CNN is trained using the detection results of R-CNN [20]. Both have higher object detection accuracy and higher video compression ratio than existing video coding methods for human vision. Models that compress video information for multi-task are also emerging. MSFC [21] is an image coding model for object detection and segmentation. The neural network used for image compression is trained using losses computed from object detection and segmentation results. Although it is not as accurate as JPEG [22] in terms of image recognition accuracy, it significantly outperforms JPEG in terms of encoding efficiency.

In addition, research is ongoing to apply video coding methods for human vision, such as VVC, to coding methods for image recognition. In some experiments, VVC is used to encode features extracted from videos using neural networks, and these experiments show that VVC can be used effectively in VCM as well.

Our approach is to improve the accuracy of object detection by processing the encoded video with a neural network. The encoded video for human vision is converted to the video for object detection by post-processing. VVC is used as the video coding method, and YOLO-v7 is utilized as the object detection model. The proposed video processing method is based on neural network, and its structure follows the generator of ESRGAN [23]. The proposed model is trained using video features extracted from pre-trained YOLO-v7. Processing VVC coded video using the proposed method can achieve high object detection accuracy at low bitrates.

Some neural network structures for post-processing of encoded video are based on ResNet [24], and some are based on U-Net [25]. These model structures are versatile and have been employed in many models that perform image recognition tasks. ResNet is a model based on residual blocks and is used in the generator of SRGAN [26]. The network proposed in this paper is based on the residual in residual dense block (RRDB) used in the generator of ESRGAN. Same as SRGAN, ESRGAN is a model for image super-resolution. The proposed model of the post-processing is shown in Fig. 1. The use of RRDB allows to reproduce more detailed patterns in the generated image by neural network than that of residual block. The characteristic of RRDB is also useful in processing of encoded video. This is because it helps reconstruct the details of the image. Our model consists of three RRDBs, two convolutional layers, and two activation functions.

Results of image recognition tasks are generally used to train neural networks which compress the video for image recognition. For example, in the studies by S. Wang et al. [20] and Z. Zhang et al. [21] the results of object detection by R-CNN are used to train the image compression models to improve the accuracy of object detection by R-CNN. In our study, YOLO-v7, the latest version of YOLO, is used as the object detection model. In order to improve the object detection accuracy of YOLO-v7 in encoded videos, it is effective to process these videos using the features of the pre-trained YOLO-v7. We process the encoded videos with a CNN-based neural network trained with YOLO-v7 features. In this training, we extract the three kind of features of YOLO-v7 from the backbone of the trained model. For the loss calculation, we use the mean squared error (MSE) between the features of raw video and that of output video of our proposed model. The training process of the proposed model is shown in Fig. 2. The loss function used for training is extracted as

where $yolo$ indicates the feature extractor of YOLO-v7. $I_{raw}$ and $I_{output}$ indicate the raw video and the output video.

In this section, we present a training process of proposed neural network for processing the encoded video. The datasets used for training are SJTU [27], UVG [28], and MCML-4K-UHD [29]. All these datasets contain some 4K (3840x2160) raw video sequences. We select 30 sequences from these datasets and encode these by using VTM10.0 [30]. The configration of the frame reference method is “random access”, and the quantization parameters (QP) are 27, 32, 37, 42, and 47. These 150 types of encoded video created in this way are used as input video for the proposed neural network. The loss function is as shown in (1), and features obtained from the trained model of YOLO-v7 are used. The optimization function is Adam and the learning rate is 1e-5. GPU used for training neural network is GeForce GTX 1080ti.

We perform object detection using YOLO-v7 on the VVC encoded video and the output video of the proposed neural network. In order to measure the object detection accuracy, we need video sequences with object annotations. Therefore, we use SFU-HW-Objects-v1 dataset [31] for evaluation. This dataset is one of the few datasets in which object annotations are assigned to raw video. It contains object annotations for 18 raw video sequences. These sequences are classified into five classes according to the image size and the characteristics of the video. Since the image size of the video sequences used in training is 4K, we use sequences of Class A, B and C, which have a larger image size compared to other classes. The details of our test sequences are shown in Table I. These sequenses are also encoded using VTM 10.0. The configration of the frame reference method is “random access”. Two sequences of Class A are encoded with five QP (27, 32, 37, 42, 47), and other sequences are encoded with three QP (37, 42, 47). We adopt the proposed post-processing method to the encoded video, and the object detection accuracy before and after the proposed processing is compared. The object detection model is pre-trained YOLO-v7, and the confidence threshold is set to 0.25. Average Precision (AP) and F1-score are used as evaluation metrics. When caluculating the value of AP, Intersection over Union (IoU) threshold is always set to 0.5.

The object detection results in PeopleOnStreet sequence are shown in Fig. 3. Since “person” accounted for 97% of the objects in PeopleOnStreet sequence, the detection results of person is also shown in Fig. 3. All three graphs show the relationship between detection accuracy and bitrate. Graph (a) represents mean Average Precision (mAP), graph (b) AP of the person detection, and graph (c) F1-score of the person detection. In this sequence, the proposed method slightly increased the value of mAP, and significantly increased the accuracy of person detection. The value of AP of person detection was improved by about 5 percentage points regardless of QP, and F1-score was improved by about 0.02 to 0.03. The transformation of the frame image of PeopleOnStreet sequence by the proposed method are shown in Fig. 5. Image (b) is the frame image of VVC encoded video, and (c) is that of the output video of the proposed neural network. This figure shows that the proposed post-processing slightly changes the color tone of the frame image.

The object detection results in Traffic sequence are shown in Fig. 4. Since the percentage of objects occupied by “car” in Traffic sequence is 99%, the detection results of “car” is also shown in Fig. 4. All these graphs also show the relationship between detection accuracy and bitrate. Graph (a) represents the value of mAP. Graph (b) and (c) represent the value of AP and F1-score of the car detection, respectively. In Traffic sequence, the value of mAP was improved by proposed method, and AP of car detection was improved by 2 to 3 percentage points and F1-score by about 0.01 to 0.02. The transformation of the frame image of Traffic sequence by the proposed method are shown in Fig. 5. In the case of Traffic sequence, as in the case of PeopleOnStreet sequence, we can confirm the change in the color tone of the frame image due to the proposed image processing.

Furthermore, the object detection accuracy in three sequences in Class B and four sequences in Class C are shown in Fig. 6. The graph (a) shows the relationship between mAP and bitrate for three sequences of Class B, and the graph (b) shows the relationship between mAP and bitrate for four sequences of Class C. In both cases, the object detection accuracy is improved for all QP.

Table II summarizes these results. For the sequences listed in Table I, the object detection accuracy (mAP) when encoded with VVC and when the proposed post-processing is performed on them are shown. The degree of improvement in mAP values are also shown in this table as a gap. For the two sequences of class A, the values of mAP are not so improved by the proposed method compared to the AP values of person and car. The shortage of the training dataset is one cause of this results. The sequences of training datasets include many persons and cars. However, some objects, such as umbrella, sport ball and chair, are not included, even though it is included in test sequences. Nevertheless, Table II shows that mAP improved for all QP of all classes. The reason for this improvement of mAP value is the enhancement in AP values of person, car, and some other objects. From these results, our proposed method is effective for improving the object detection accuracy in VVC coded video.