NVC-1B: A Large Neural Video Coding Model

We increase the number of intermediate feature channels and insert more residual blocks to scale up the model size of the motion encoder and decoder. We build three models ($M_{1}$, $M_{2}$, $M_{3}$) with 52M, 71M, and 96M parameters, respectively. The comparison results as illustrated in Fig. 2(a) indicate that a slight increase in the model size of the motion encoder and decoder results in performance gains. However, continuously increasing the model size of the motion encoder-decoder will reduce the performance gain. For example, when the model size of the motion encoder-decoder is increased to 52M, 15.4% performance gain can be achieved for the HEVC Class C dataset. However, if we further increase its model size to 96M parameters, the performance gain drops to 1.5%. The results indicate that the motion encoder-decoder is not always the larger the better. More analysis can be found in Section V-C3. In addition, we observe that although larger motion encoder-decoders bring performance gain, the training processes become unstable and training crashes occur more frequently.

Based on the abovementioned $M_{1}$, $M_{2}$, $M_{3}$ models, we first increase the channel number of the motion latent representation $\hat{m}_{t}$ and associated hyperprior $\hat{z}_{t}^{m}$. Then, we scale up the model size of motion hyper-encoder, hyper-decoder, and quadtree partition-based spatial context models [32, 39] by increasing the number of intermediate feature channels. We build three models ($M_{4}$, $M_{5}$, $M_{6}$) with 128M, 177M, and 226M parameters, respectively. The comparison results as illustrated in Fig. 2(b) indicate that the increase in the model size of the motion entropy model can bring a little compression performance gain. For example, based on the $M_{1}$ model with 52M parameters, the $M_{4}$ model improves the performance gain from 15.4% to 16.0% for the HEVC Class C dataset. However, although $M_{4}$, $M_{5}$, $M_{6}$ models can further improve performance, similar to $M_{1}$, $M_{2}$, $M_{3}$ models, training crashes occur more frequently.

Based on our small baseline model with 21M parameters, we increase the number of intermediate feature channels and insert more residual blocks to scale up the model size of the contextual encoder and decoder. We build three models ($M_{7}$, $M_{8}$, $M_{9}$) with 50M, 77M, and 90M parameters, respectively. As presented in Fig. 3(a), increasing the model size of the context encoder and decoder can effectively improve compression performance. Different from the motion encoder and decoder, further increasing the model size of the contextual encoder and decoder can result in sustained and stable performance gains. For example, when the model size of the contextual encoder and decoder is increased to 50M, 10.6% performance gain can be achieved for the HEVC Class C dataset. When the model size of the contextual encoder and decoder is further increased to 90M, the performance gain can reach 14.8% for the HEVC Class C dataset. The results show that a larger contextual encoder-decoder can improve the transform capability. More analysis can be found in Section V-C1.

Based on $M_{7}$, $M_{8}$, $M_{9}$ models, we continue to scale up the model size of their contextual entropy models. We increase the channel number of the contextual latent representation $\hat{y}_{t}$ and associated hyperprior $\hat{z}_{t}^{y}$. In addition, we increase the channel number of intermediate features of the contextual entropy model, including the contextual hyper-encoder, hyper-decoder, and quadtree partition-based spatial context models [32, 39]. We build three models ($M_{10}$, $M_{11}$, $M_{12}$) with 92M, 199M, and 212M parameters, respectively. As presented in Fig. 3(b), increasing the model size of the context encoder and decoder can bring stable compression performance improvement. For example, when the model size of $M_{9}$ model is increased from 90M to 212M by scaling up its contextual entropy model, an additional 0.8% performance gain can be achieved for the HEVC Class C dataset. The results show that a larger contextual entropy model can improve the entropy modeling of the contextual latent representation.

Temporal context mining is an intermediate link between the motion encoder-decoder and contextual encoder-decoder. To explore the influence of its model size, we increase the number of intermediate feature channels $N$ and insert more residual blocks to scale up its model size. Based on our small baseline model with 21M parameters, we build three models ($M_{13}$, $M_{14}$, $M_{15}$) with 26M, 47M, and 75M parameters, respectively. As shown in Fig. 4(a), scaling up the model size of the temporal context mining module can improve compression performance stably. For example, when the model size is increased from 26M to 75M, the performance gain can be increased from 5.1% to 12.0% for the HEVC Class D dataset. To explore whether the gain of scaling up the temporal context mining module can be superimposed with the gain of scaling up the contextual encoder-decoder and contextual entropy model, based on $M_{10}$, $M_{11}$, $M_{12}$, we build another three models ($M_{16}$, $M_{17}$, $M_{18}$) with 144M, 252M, and 265M parameters, respectively. As shown in Fig. 4(b), based on the model with a larger contextual encoder-decoder and contextual entropy model, scaling up the model size of the temporal context mining module can bring additional performance gain. For example, based on $M_{12}$ with 212M parameters, increasing its temporal context mining module to build the model $M_{18}$ with 265M parameters can make the performance gain increase from 14.2% to 21.2% for the HEVC Class D dataset. The results show that a larger temporal context mining module can help make full use of the temporal correlation. More analysis can be found in Section V-C2.

In terms of the mixed CNN-Transformer architecture, we build three models $M_{19}$, $M_{20}$, $M_{21}$. For model $M_{19}$, we replace the residual blocks between the convolutional layers of the contextual encoder and decoder with SwinTransformer layers [75] to build a contextual encoder-decoder with mixed CNN-Transformer architecture. We scale up its model size to 42M to compare it with its CNN-architecture counterpart—$M_{7}$ model (50M). For model $M_{20}$, based on $M_{19}$, we continue to insert SwinTransformer layers into the CNN-based entropy model to build a contextual entropy model with mixed CNN-Transformer architecture. We scale up its model size to 240M to compare it with its CNN architecture counterpart—$M_{12}$ model (212M). For model $M_{21}$, based on $M_{20}$, we insert a SwinTransformer layer after each residual block of the temporal context mining module. We scale up its model size to 267M to compare it with its CNN architecture counterpart—$M_{18}$ model (265M). As illustrated in Fig. 5(a), for model $M_{19}$ and model $M_{20}$, inserting the Transformer layers into the original CNN architecture can bring performance gain over the small baseline model for its global feature extraction ability. This phenomenon is consistent with previous work [76, 77, 78] on image coding based on Transformer. However, when compared with their CNN-architecture counterparts of similar model sizes, CNN-architecture counterparts can obtain higher video coding performance gain. For example, comparing $M_{20}$ and $M_{12}$, which both have larger contextual encoder-decoder and larger contextual entropy model, the performance gain of $M_{12}$ is 15.6% but that of $M_{20}$ is only 9.4% for the HEVC Class C dataset. For model $M_{21}$, inserting SwinTransformer layers into the temporal context mining module brings performance loss. The compression performance gain drops from 9.4% achieved by $M_{20}$ to 2.9%. The results indicate that the global feature extraction ability of Transformer layers may not be suitable for learning multi-scale temporal contexts.

In terms of the Transformer architecture, we build three models $M_{22}$, $M_{23}$, $M_{24}$. For model $M_{22}$, we replace all the convolutional layers and residual blocks of the contextual encoder and decoder with SwinTransformer layers [75] to build a contextual encoder-decoder with Transformer architecture. We scale up its model size to 44M to compare it with its CNN-architecture counterpart—$M_{7}$ model (50M). For model $M_{23}$, based on $M_{22}$, we further replace the convolutional layers and depth blocks in the contextual entropy model with SwinTransformer layers to build a contextual entropy model with Transformer architecture. We scale up its model size to 208M to compare it with its CNN architecture counterpart—$M_{12}$ model (212M). For model $M_{24}$, based on $M_{23}$, we replace all the convolutional layers and residual blocks of the temporal context mining module with SwinTransformer layers. We scale up its model size to 232M to compare it with its CNN architecture counterpart—$M_{18}$ model (265M). As illustrated in Fig. 6, for model $M_{22}$, the Transformer-based contextual encoder-decoder can bring a little performance gain over the small baseline model. However, when compared with its CNN architecture counterpart $M_{7}$ with similar model sizes, its performance gain is much smaller. For model $M_{23}$ and model $M_{24}$, the Transform-only-based contextual entropy model and temporal context mining module make the model crash. Compared with the small baseline model, there is a PSNR loss of more than 4dB, which makes it difficult to calculate the BD-rate values. The results indicate that the convolutional layer is necessary for contextual entropy model and temporal context mining module.

For training, we use 7-frame videos of the Vimeo-90k [80] dataset. Following most existing neural video coding models [34, 30, 35, 32], we randomly crop the original videos into 256$\times$256 patches for data augmentation. For testing, we use HEVC dataset [81], UVG dataset [82], and MCL-JCV dataset [83] in RGB and YUV420 format. These datasets contain videos with different contents, motion patterns, and resolutions, which are commonly used to evaluate the performance of neural video coding models. When testing RGB videos, we convert the videos in YUV420 format to RGB format using FFmpeg.

Following [32, 39], we set 4 base $\lambda$ values (85, 170, 380, 840) to control the rate-distortion trade-off. For hierarchical quality, we set the periodically varying weight $w_{t}$ before $\lambda$ as (0.5, 1.2, 0.5, 0.9). We implement our NVC-1B model with PyTorch. AdamW [84] is used as the optimizer and batch size is set to 32. Before the multi-frame cascaded fine-tuning stage, we train the NVC-1B model on 32 NVIDIA Ampere Tesla A40 (48G memory) GPUs for 85 days. In the multi-frame cascaded fine-tuning stage, we train the NVC-1B model on 4 NVIDIA A800 PCIe (80G memory) GPUs for 28 days. To alleviate the CUDA memory pressure induced by multi-frame cascaded training, Forward Recomputation Backpropagation (FRB)¹¹1https://qywu.github.io/2019/05/22/explore-gradient-checkpointing.html is used.

As with most previous neural video coding models, we focus on the low-delay coding scenario in this paper. Following [30, 32, 39, 34], we test 96 frames for each video sequence and set the intra-period to 32. For traditional video codecs, we choose HM-16.20 [85], VTM-13.2 [49], and ECM-5.0 [79] as our benchmarks. HM-16.20 is the official reference software of H.265/HEVC. VTM-13.2 is the official reference software of H.266/VVC. ECM is the prototype of the next-generation traditional codec. We use encoder_lowdelay_main (_rext), encoder_lowdelay_vtm, , and encoder_lowdelay_ecm configurations for HM-16.20, VTM-13.2, and ECM-13.0, respectively. The detailed commands for HM-16.20, VTM-13.2, and ECM-13.0 are shown as follows.

• •

  -c $\{\emph{config file name}\}$ --InputFile=$\{\emph{input file name}\}$ --InputChromaFormat=$\{\emph{input chroma format}\}$ --FrameRate=$\{\emph{frame rate}\}$ --DecodingRefreshType=2 --InputBitDepth=8 --FramesToBeEncoded=96 --SourceWidth=$\{\emph{width}\}$ --SourceHeight=$\{\emph{height}\}$ --IntraPeriod=32 --QP=$\{\emph{qp}\}$ --Level=6.2 --BitstreamFile=$\{\emph{bitstream file name}\}$

For neural video coding models, we choose CANF-VC [36], DCVC [34], DCVC-TCM [30], DCVC-HEM [35], DCVC-DC [35], DCVC-FM [40], and our small baseline model—DCVC-SDD [32] as our benchmarks.

When testing RGB videos, we use RGB-PSNR as the distortion evaluation metric. When testing YUV420 videos, following DCVC-DC [35], we use the compound YUV PSNR as the distortion evaluation metric. The weight of YUV components is set to 6:1:1 [86]. Bits per pixel (bpp) is used as the bitrate evaluation metric. BD-rate [87] is used to compare the compression performance of difference models, where negative numbers indicate bitrate saving and positive numbers indicate bitrate increasing.

When testing RGB videos, we illustrate the rate-distortion curves on the HEVC, UVG, and MCL-JCV RGB video datasets in Fig. 7. The curves show that our proposed large neural video coding model—NVC-1B has significantly outperformed its small baseline model—DCVC-SDD. We list the detailed BD-rate comparison results in Table. III. The results show that our NVC-1B achieves an average –25.1% BD-rate reduction over VTM-13.2 across all test datasets, which is much better than that of DCVC-SDD (–12.7%). It even surpasses the state-of-the-art neural video coding models—DCVC-DC (–18.3%) and DCVC-FM (–14.3%).

When testing YUV420 videos, we illustrate the rate-distortion curves on the YUV420 videos in Fig. 8 and list the corresponding BD-rate comparison results in Table. IV. The results show that based on our small baseline model—DCVC-SDD, the large model improves the average compression performance from –8.4% to –20.5%, which even outperforms ECM (–19.0%).

However, we find that the performance gain on the HEVC Class E dataset is smaller than that of other datasets. This is mainly because the number of frames used for multi-frame cascaded fine-tuning is small (6 frames) under the limitation of GPU CUDA memory. If we have GPUs with larger CUDA memory, we can use more frames (DCVC-DC uses 7 frames and DCVC-FM uses 32 frames) to fine-tune our large model to improve its compression performance.

We visualize the reconstructed frames of our proposed NVC-1B and its small baseline model—DCVC-SDD in Fig. 9. By comparing their subjective qualities, we observe that the frames reconstructed by NVC-1B can retain more textures with a lower bitrate. We take the videoSRC14 sequence of the MCL-JCV dataset, the RaceHorses sequence of the HEVC Class D dataset, and the Kimono1 sequence of the HEVC Class B dataset as examples. For the videoSRC14 sequence, our NVC-1B model can use the bitrate of 0.051 bpp (0.057 bpp for DCVC-SDD) to achieve a reconstructed frame of 37.52 dB (37.46 dB for DCVC-SDD). Observing the earring of the dancing woman in videoSRC14, we can find our NVC-1B model can keep sharper edges. For the RaceHorses sequence, our NVC-1B model can use the bitrate of 0.078 bpp (0.090 bpp for DCVC-SDD) to achieve a reconstructed frame of 30.42 dB (30.32 dB for DCVC-SDD). Comparing the horse’s reins, we can see that our NVC-1B model can reconstruct the reins but DCVC-SDD cannot. For the Kimino1 sequence, our NVC-1B model can use the bitrate of 0.026 bpp (0.027 bpp for DCVC-SDD) to achieve a reconstructed frame of 35.19 dB (35.06 dB for DCVC-SDD). Zooming in the belt on the woman, the frame reconstructed by our NVC-1B model can retain more details.

Following the setting of our small baseline model—DCVC-SDD, we include the time for model inference, entropy modeling, entropy coding, and data transfer between CPU and GPU. All the learned video codecs are run on a NVIDIA A800 GPU. As shown in Table V, the encoding and decoding time of our NVC-1B model for 1080p video are 4.44s and 3.54s per frame, respectively, which are five times longer than that of DCVC-SDD. However, the encoding time of our NVC-1B model is significantly lower than that of traditional video codecs. With the development of lightweight techniques [88, 89] for large models, the encoding and decoding time can be further optimized.

To explore why our proposed large video coding model—NVC-1B can bring performance gain, we analyze the transform energy compaction. We calculate the average bitrate ratio of each channel of the contextual latent representation $\hat{y}_{t}$ over the HEVC Class C and D datasets. Specifically, the contextual bitrate ratio (CR) of channel $c$ for each video sequence is first calculated as (7).

Then, we calculate the average contextual bitrate ratio over all sequences. We sort the average contextual bitrate ratio of each channel from largest to smallest to analyze the transform energy compaction. As illustrated in Fig. 10, for contextual bitrate ratio, we present the top 100 channels with obvious bitrate overhead. Comparing the channel bitrate distribution, we find that the contextual latent representations generated by our proposed large video coding model have higher transform energy compaction than our small baseline model—DCVC-SDD. More bitrates are concentrated in fewer channels.

In addition to the contextual latent representations, we also calculate the channel bitrate ratio of motion latent representations $\hat{m}_{t}$. Similarly, the motion bitrate ratio (MR) of channel $c$ for each video sequence is first calculated as (8).

Then, we calculate the average motion bitrate ratio over all sequences. As presented in Fig. 10, by ranking the average motion bitrate ratio of each channel, we find that although we allocate most of the parameters to contextual parts, the transform energy compaction can be improved for motion compression by end-to-end training. The motion channel bitrate distribution of our large model is also more concentrated. The analysis indicates that our large video coding model can obtain a higher compression performance by increasing the transform energy compaction. More signal energy is concentrated in fewer channels, which is beneficial for entropy coding.

We further analyze the temporal context mining accuracy to explore why our proposed large video coding model—NVC-1B can bring performance gain. We visualize the temporal context $C_{t}^{0}$ predicted by NVC-1B and its small baseline model—DCVC-SDD in Fig. 11. By zooming in the temporal contexts, we find that the context predicted by NVC-1B has smoother object edges, whereas that predicted by DCVC-SDD has obvious prediction artifacts. For example, the edges of the horse’s tail and saddle predicted by DCVC-SDD have much noise, which seriously reduces the temporal prediction accuracy. The analysis indicates that our NVC-1B model can obtain a higher compression performance by improving the temporal context mining accuracy.

In Section IV-A1, we observe a strange phenomenon that if we slightly increase the model size of the motion encoder and decoder it can lead to performance gains. However, the performance gain will decrease if we continuously increase their model size. To explore the reason for this strange phenomenon, we visualize the warp frames predicted by the small baseline model—DCVC-SDD, $M_{1}$, and $M_{3}$ models mentioned in Section IV-A1. As illustrated in Fig. 12, we find that, at a similar bitrate, the $M_{1}$ model with 52M parameters can obtain a more accurate warp frame than DCVC-SDD with 21M parameters. The arm and bubbles predicted by the $M_{1}$ model have fewer artifacts. However, the accuracy of the warp frame predicted by $M_{3}$ decreases dramatically. Its prediction artifacts of arms and bubbles increase significantly. The analysis indicates that the model size of the motion encoder-decoder is not always the bigger the better. An inappropriate model size will reduce the accuracy of temporal prediction.