PNVC: Towards Practical INR-based Video Compression

In the proposed PNVC framework, each video sequence $\{\mathbf{\bm{x}}_{t}\}^{T}_{t=1},\ \mathbf{\bm{x}}_{t}\in\mathbb{R}^{3\times H\times W}$, is segmented into Groups of Pictures (GOPs)²²2We follow the GoP definition in H.266/Versatile Video Coding Test Model (VTM), where the length of a GoP equals the number of consecutive bi-directional predicted (B) frames plus 1. of length $N$ that are independently encoded, where $T$, $H$, and $W$ represent the length, height, and width of $\{\mathbf{\bm{x}}_{t}\}^{T}_{t=1}$, respectively. Within each GOP, frames are either intra-coded as I frames or inter-coded as P/B frames to exploit spatial and/or temporal redundancy within the video.

Each I frame is encoded into $L$ grids of latent tokens $\{\mathbf{\bm{y}}_{t}^{l}\}^{L}_{l=1}$ by an encoder $E$. For each level $l$, $\mathbf{\bm{y}}_{t}^{l}$ consists of rescale and shift parameters $\mathbf{\bm{\gamma}}^{l}_{t}$ and $\mathbf{\bm{\beta}}^{l}_{t}$, concatenated channel-wise, with a resolution of $\left(\frac{H}{2^{l}},\frac{W}{2^{l}}\right)$. A learnable quantizer $Q$ then quantizes these latent grids, producing $\{\hat{\mathbf{\bm{y}}}_{t}^{l}\}^{L}_{l=1}=\{\hat{\mathbf{\bm{\gamma}}}^{l}_{t}\|\hat{\mathbf{\bm{\beta}}}^{l}_{t}\}^{L}_{l=1}$ based on the corresponding quantization parameters $\{\mathbf{\bm{q}}^{l}_{\mathbf{\bm{y}};t}\}^{L}_{l=1}$, where $\|$ denotes the channel-wise concatenation.

An entropy model $g_{\mathbf{\bm{\psi}}}$ is adopted to estimate the probability mass function (PMF) of the hierarchical latents semi-autoregressively along the spatial, channel, and hierarchical axes. The parameter of the entropy model, $\mathbf{\bm{\psi}}$, is updated as $\mathbf{\bm{\psi}}^{l}_{t}\rightarrow\mathbf{\bm{\psi}}+\Delta\tilde{\mathbf{\bm{\psi}}}^{l}_{t}$ at each level $l$. Here, $\mathbf{\bm{\psi}}$ represents the pretrained parameters, and $\Delta\tilde{\mathbf{\bm{\psi}}}^{l}_{t}$ are dequantized parameter updates obtained through online overfitting. The bitstream includes $\{\hat{\mathbf{\bm{y}}}^{l}_{t},\Delta\hat{\mathbf{\bm{\psi}}}^{l}_{t},\tilde{\mathbf{\bm{q}}}^{l}_{\mathbf{\bm{y}};t},\tilde{q}^{l}_{\mathbf{\bm{\psi}};t}\}^{L}_{l=1}$, where $\{\tilde{\mathbf{\bm{q}}}^{l}_{\mathbf{\bm{y}};t}\}^{L}_{l=1}$ and $\{\tilde{q}^{l}_{\mathbf{\bm{\psi}};t}\}^{L}_{l=1}$ are the inverse quantization parameters corresponding to latents and weight updates, respectively. These components are overfitted to the input video frame, entropy encoded, and combined into the bitstream.

At the decoder, the entropy model is updated by $\{\tilde{\mathbf{\bm{\psi}}}^{l}_{t}\}^{L}_{l=1}$ decoded from the bitstream. It is used to help entropy decode the hierarchical latent grids that are then dequantized by $Q^{-1}$ using $\{\tilde{\mathbf{\bm{q}}}^{l}_{\mathbf{\bm{y}};t}\}^{L}_{l=1}$ to produce $\{\tilde{\mathbf{\bm{y}}}^{l}_{t}\}^{L}_{l=1}$. These dequantized latents undergo positional embedding based on an improved scale-aware hierarchical decomposition scheme from (Kwan et al. 2024a), allowing content-specific variations to be queried by spatial-temporal coordinates. A patch-based representation is used, with each element of the smallest $\tilde{\mathbf{\bm{y}}}^{l}_{t}$ corresponding to a patch of the original frame $\mathbf{\bm{x}}_{t}$. The INR-based decoder $D$ defines a patch-wise neural field as a function of 3D coordinates. It uses $L$ stacked ModMixer blocks (proposed in this work) to progressively and conditionally map $t_{S}$, which is a learned bias for I-frame and the reference patch for P-/B- frames, to patches of the reconstructed frame $\tilde{\mathbf{\bm{x}}}_{t}$, with the intermediate activations of each block modulated by the corresponding latent grid.

Each P or B frame is coded following a workflow similar to that of I frames, but with an additional motion encoder $E_{m}$, as shown in Figure 2. With up to two previously reconstructed frames, $\tilde{\mathbf{\bm{x}}}_{\text{ref}_{1};t}$ and $\tilde{\mathbf{\bm{x}}}_{\text{ref}_{2};t}$, in the decoded frame buffer (they can be I or P/B frames) and the current frame $\mathbf{\bm{x}}_{t}$, the multi-resolution latent grids $\{\mathbf{\bm{y}}_{t}^{l}\}^{L}_{l=1}$ are generated by $E$ but with layer-wise conditioning on the motion information extracted from $E_{\text{m}}$. Here, the entropy model $g_{\mathbf{\bm{\psi}}}$ additionally takes into account the decoded reference latent tokens $\tilde{\mathbf{\bm{y}}}^{l}_{\text{ref}_{1};t},\tilde{\mathbf{\bm{y}}}^{l}_{\text{ref}_{2};t}$ and the reconstructed latent tokens from higher-level layers, $\tilde{\mathbf{\bm{y}}}^{>l}_{t}$, as input to further exploit spatio-temporal and hierarchical redundancy.

The ELIC  (He et al. 2022) model is adopted here as the image encoder $E$ to map I/P/B frames to the latent $\{\mathbf{\bm{y}}^{l}_{t}\}^{L}_{l=1}$. For P/B frames, $E$ is reconditioned by concatenating the hierarchical optical flow features, extracted by the motion encoder $E_{m}$, adapted from SpyNet (Ranjan and Black 2017), to incorporate the estimated motion information. The illustration of its network structure is provided in the Supplementary. Both $E$ and $E_{m}$ are overfitted to generate content-adaptive hierarchical latent representations. The overfitting of the next frame is initialized from the network updated for the current frame to further expedite the encoding process.

A hierarchical quality structure (Li, Li, and Lu 2024) is adopted in the quantization module, where the allowable bitrates are adaptively reweighted for each frame based on its distance from reference frames and specific video dynamics. This parameter is referred to as the quality parameter $q_{\text{glob};t}$, generated by a ConvLSTM module based on each token’s estimated impact on the GOP-level RD trade-offs. This module is fixed after pretraining to prevent the frame-wise overfitting process from destroying the acquired hierarchical quality structure. For quantization, another set of finer-grained channel-wise quality parameters $\{\mathbf{\bm{q}}^{l}_{\mathbf{\bm{y}};t}\}^{L}_{l=1}$ are used. For inverse quantization, the corresponding $\{\tilde{\mathbf{\bm{q}}}^{l}_{\mathbf{\bm{y}};t}\}^{L}_{l=1}$ are retrieved and encoded into the bitstream as side information. The quantization and inverse quantization steps are exemplified using $\mathbf{\bm{y}}^{l}_{t}$ as:

	$\displaystyle\hat{\mathbf{\bm{y}}}^{l}_{t}[c][i][j]$	$\displaystyle=\lfloor\mathbf{\bm{q}}^{l}_{\mathbf{\bm{y}};t}[c]\cdot\mathbf{\bm{y}}^{l}_{t}\rceil,$		(1a)
	$\displaystyle\tilde{\mathbf{\bm{y}}}^{l}_{t}[c][i][j]$	$\displaystyle=\tilde{\mathbf{\bm{q}}}^{l}_{\mathbf{\bm{y}};t}[c]\cdot\hat{\mathbf{\bm{y}}}^{l}_{t},$		(1b)

where $c$, $i$, and $j$ denote the channel and spatial indices, respectively. Here, we avoid division in both cases to avoid numerical instabilities. The scalar quantization parameters $\{q_{\mathbf{\bm{\psi}}}\}^{L}_{l=1}$ are updated following a similar procedure.

The discretized hierarchical latents $\{\hat{\mathbf{\bm{y}}}^{l}_{t}\}^{L}_{l=1}$ are entropy coded based on arithmetic coding. Here, PMFs are estimated by a Gaussian distribution $\mathcal{N}(\cdot)$, where the location and scale parameters $\left(\mathbf{\bm{\mu}}_{t}[c][i][j],\mathbf{\bm{\sigma}}_{t}[c][i][j]\right)$ of each element $\hat{\mathbf{\bm{y}}}^{l}_{t}[c][i][j]$ are semi-autoregressively predicted by the entropy network $g_{\mathbf{\bm{\psi}}}$ based on the spatial, temporal, and hierarchical contexts of the element. We use the quadtree-based factorization (Li, Li, and Lu 2023) to encode $\hat{\mathbf{\bm{y}}}^{l}_{t}$, but with two modifications. First, we respectively split the channels of $\hat{\mathbf{\bm{\gamma}}}^{l}_{t}$ and $\hat{\mathbf{\bm{\beta}}}^{l}_{t}$ (they are entropy coded in parallel) into four uneven groups (He et al. 2022), with sizes proportional to 1, 1, 2 and 4, i.e. we double the number of symbols decoded per decoding step due to more contexts available. Moreover, we replace all concatenation operations to aggregate information with element-wise modulation, which we empirically found to be far more efficient in information aggregation, and deploy a new ModMixer module (detailed in the next subsection) to construct and optimize the entropy model. The architecture illustration of the entropy model is provided in the Supplementary. For entropy model’s updates $\{\Delta\hat{\mathbf{\bm{\psi}}}_{t}^{l}\}_{l=1}^{L}$, a fully factorized nonparametric density model $\mathbf{\bm{\pi}}$ (Ballé et al. 2018) is deployed following (Gomes, Azevedo, and Schroers 2023; Zhang et al. 2024).

To enhance model performance without introducing extra inference cost, we devise a novel ModMixer module as a basic building block used in our entropy model $g_{\mathbf{\bm{\psi}}}$ and decoder $D$. This approach is inspired by reparameterization methods (Ding et al. 2021, 2022; Shi, Zhou, and Gu 2024) that leverage the interconversion between linear architectures to trade training-time complexity for inference-time efficiency. Without loss of generality, we define an arbitrary linear layer (convolutional or fully-connected), $\mathbf{\bm{W}}\mathbf{\bm{h}}+\mathbf{\bm{b}}$, to be contracted algebraically from this general formulation:

$$\sum_{m=1}^{M}\left(\overline{\mathbf{\bm{\Lambda}}}_{m;\mathbf{\bm{\gamma_{W}}}}\mathbf{\bm{W}}_{m}+\overline{\mathbf{\bm{\Lambda}}}_{m;\mathbf{\bm{\beta_{W}}}}\right)\mathbf{\bm{h}}+\left(\overline{\mathbf{\bm{\Lambda}}}_{m;\mathbf{\bm{\gamma_{b}}}}\mathbf{\bm{b}}_{m}+\overline{\mathbf{\bm{\Lambda}}}_{m;\mathbf{\bm{\beta_{\mathbf{\bm{b}}}}}}\right),$$

(2)

where $\mathbf{\bm{W}}_{m}\in\mathbb{R}^{c_{\text{out}}\times c_{\text{in}}}$ and $\mathbf{\bm{b}}_{m}\in\mathbb{R}^{c_{\text{out}}}$ denote the trainable, parallel basis of $\mathbf{\bm{W}}$ and $\mathbf{\bm{b}}$, respectively, and the coefficients $\{\overline{\mathbf{\bm{\Lambda}}}_{m;\mathbf{\bm{\gamma_{W}}}},\overline{\mathbf{\bm{\Lambda}}}_{m;\mathbf{\bm{\beta_{W}}}},\overline{\mathbf{\bm{\Lambda}}}_{m;\mathbf{\bm{\gamma_{b}}}},\overline{\mathbf{\bm{\Lambda}}}_{m;\mathbf{\bm{\beta_{b}}}}\}\in\mathbb{R}^{c_{\text{out}}}$ denote the channel-wise affine mixand values. Here, each basis may be further expanded serially into (taking $\mathbf{\bm{W}}_{m}$ as an example) $\mathbf{\bm{W}}_{m}=\mathbf{\bm{W}}_{m;N}\mathbf{\bm{W}}_{m;N-1}\cdots\mathbf{\bm{W}}_{m;1}$. Compared to standard reparameterization methods, Equation (2) establishes a more generalized visual tuning scheme that better suits the context of instance-adaptive video compression. With the affine transformation applied to both weight and bias terms, it is a superset encompassing both weight and feature space tuning, where the mixands could be flexibly re-casted to feature modulation (as in decoder), weight update (as in entropy model), or any other forms of visual tuning accordingly.

In our approach, to leverage the above reparameterization idea and improve fully connected layers (FC), typically adopted in INR-based methods (Sitzmann et al. 2020; Dupont et al. 2021), a new basic building block, ModMixer, is designed as illustrated in Figure 3. Each FC layer is reparameterized both serially and in parallel through stacks of linear layers with varying depths per branch (one per channel group) to capture the hierarchical representations efficiently. Further, a self-attention-style token-mixer is attached before the FC layer to induce inference-time spatial mixing. During pretraining, the token mixer is progressively degenerated and absorbed into the Instance Normalization (Huang and Belongie 2017) and proceeding FC layer (Lin et al. 2024), by initializing a mask $\mathbf{\bm{M}}$ as $\mathbf{\bm{1}}$ that is gradually decayed to $\mathbf{\bm{0}}$: $\mathbf{\bm{M}}\odot\text{TokenMixer}\left(\mathbf{\bm{h}}\right)+\left(1-\mathbf{\bm{M}}\right)\odot\mathbf{\bm{h}}+\mathbf{\bm{h}}$.

The mask decaying strategy is inspired by (Wang et al. 2023; Peng et al. 2024), which leverage the more expressive teacher model to distill the smaller student. During overfitting, we optimize the residual updates, w.r.t the pretrained mixands and basis, which are then consolidated into either weight updates $\Delta\mathbf{\bm{\psi}}^{l}_{t}$ for the entropy model or absorbed into the hierarchical latent grids $\mathbf{\bm{y}}_{t}^{l}$ for the decoder (see Supplementary for full derivations and implementation details).

In Figure 2, the decoder $D(\cdot)$ acting on the grid-based positional encodings incrementally restores spatial information while sequentially composing high-frequency details over low-frequency elements. We follow the hierarchical encoding method proposed by the high-performance HiNeRV (Kwan et al. 2024a), which can be conceptualized as a positional numeral system with power-of-2 bases. This method expresses each coordinate as an ordered set of digits, each of which corresponds to a hierarchy that recursively encodes residuals of the coarser-grained representational level. Specifically, this hierarchical coordinate system decomposes a global coordinate pos into multiple levels of finer detail with base $B_{l}$. At each level $l$, the local coordinate for interpolation is computed as:

$$\texttt{pos}^{l}=\left\lfloor\frac{\texttt{pos}}{\prod_{k=L}^{l-1}B_{k}}\right\rfloor\mathrm{\ mod\ }B_{l}.$$

(3)

We make a simple improvement to the original strategy to cope with the increase in patch size at inference time compared to that seen by the model during pretraining. The likely substantial increase in resolution, denoted by $k$, could result in more fine-tuning endeavours. Maintaining the original grid spacing would result in an increase in overall grid numbers, potentially complicating hierarchical pattern capturing and increasing memory/computational demands To address this, we propose to instead mix and rescale the bases $\{B_{l}\}^{L}_{l=1}$ non-linearly according to the ratio ($k$). The local coordinate at level $l$ is re-calculated as:

$$\texttt{pos}^{l}=\left\lfloor\frac{\texttt{pos}}{\prod^{l-1}_{k=L}w_{k}B_{k}}\right\rfloor\mathrm{\ mod\ }(w_{l}B_{l}),$$

(4)

subject to $w_{1}w_{2}\cdots w_{L}=k$ and $w_{1}\geq w_{2}\geq\cdots\geq w_{L}\geq 1$. Solving this constraint yields a set of $\{w_{l}\}^{L}_{l=1}$ evenly distributed on a log-scale, as will be detailed in Supplementary. The formulation by Equation (4) essentially distributes the interpolation pressure across different hierarchies. Low frequencies, which are typically less sensitive to resolution changes, can handle more scaling without significant loss of information (sparser grid partition at lower resolution). High frequencies, on the other hand, are preserved more carefully from being scaled less. Based on Equation (4), the $l$-th layer of decoder is given by:

	$$\displaystyle\ddot{\mathbf{\bm{\gamma}}}^{l}_{t},\ddot{\mathbf{\bm{\beta}}}^{l}_{t}=\text{Interp}_{\text{bilinear}}\left(\tilde{\mathbf{\bm{\gamma}}}^{l}_{t},\texttt{pos}^{l}\right),\text{Interp}_{\text{blinear}}\left(\tilde{\mathbf{\bm{\beta}}}^{l}_{t},\texttt{pos}^{l}\right),$$
	$$\displaystyle\ddot{\mathbf{\bm{h}}}_{t}^{l-1}=\text{ReLU}\left(\mathbf{\bm{W}}^{l}\left(\left(\ddot{\mathbf{\bm{\gamma}}}^{l}_{t}\text{IN}\left(\mathbf{\bm{h}}^{l}_{t}\right)+\ddot{\mathbf{\bm{\beta}}}^{l}_{t}\right)+\mathbf{\bm{h}}^{l}_{t}\right)+\mathbf{\bm{b}}^{l}\right),$$
	$$\displaystyle\mathbf{\bm{h}}_{t}^{l-1}=\text{Upsample}_{\text{bilinear}}(\ddot{\mathbf{\bm{h}}}^{l-1}_{t};S^{l-1}_{\text{default}}=2).$$

The full model is first pretrained offline, based on static training materials, and then overfitted during inference to adapt to the input video sequence to be compressed. Considering the pre-trained parameters, the “meta-initialization” shared between the encoder and the decoder, the iteratively refined parameter updates, relative to this meta-initialization, are quantized alongside associated side information and then entropy coded into the bitstream. We follow (Lu et al. 2020; Li, Li, and Lu 2023) to aggregate the loss over multiple frames to reduce error propagation and establish the hierarchical quality structure. The pretraining involves minimizing the following rate-distortion loss within each GoP:

$$\frac{1}{N}\sum^{N}_{t=1}\left(\sum^{L}_{l=1}\left(R(\hat{\mathbf{\bm{y}}}^{l}_{t})+R(\hat{\mathbf{\bm{q}}}^{l}_{\text{ch};t})\right)+q_{\text{glob};t}\cdot\lambda\cdot d(\mathbf{\bm{x}}_{t},\hat{\mathbf{\bm{x}}}_{t})\right),$$

(5)

where $d(\cdot,\cdot)$ denotes the distortion, $R(\cdot)$ stands for the rate, and $t\in{1,\cdots,N}$ denotes the displaying order of each frame in a GOP. Following (Kim et al. 2024; Leguay et al. 2024), the quantization of hierarchical latents and weight update is approximated by progressively annealed soft-rounding with additive noises when calculating $R(\cdot)$, and by the straight-through estimator (STE) (Minnen and Singh 2020) when rounding them to optimize the distortion metric. Here MSE is used as the distortion metric targeting the best PSNR performance, while additional models are trained by fine-turning MSE models using $200\cdot(1-\text{MS-SSIM}(\mathbf{\bm{x}}_{t},\hat{\mathbf{\bm{x}}}_{t}))$ (Mentzer et al. 2022) as the distortion metric to yield MS-SSIM-based baselines. During frame-wise overfitting, encoding entails searching for the optimal values of the hierarchical latents, network parameters, and quantization parameters:

$$\sum^{L}_{l=1}\left(R(\hat{\mathbf{\bm{y}}}^{l}_{t})+R(\Delta\hat{\mathbf{\bm{\psi}}}^{l}_{t})+R(\hat{\mathbf{\bm{q}}}^{l}_{\text{ch};t})\right)+q_{\text{glob};t}\cdot\lambda\cdot d(\mathbf{\bm{x}}_{t},\hat{\mathbf{\bm{x}}}_{t}).$$

(6)

Detailed hyperparameter configurations could be found in the Supplementary.

The BD-rate performance is summarized in Table 1, where it can be seen that PNVC (LD and RA modes) outperforms the anchor HM 18.0 (LD) by 34.35% and 39.98% in BD-rate in terms of PSNR on the UVG and MCL-JCV dataset, respectively. PNVC also significantly outperforms state-of-the-art INR-based codecs and is also competitive against the state-of-the-art conventional and neural video codecs. Furthermore, it should be noted that PNVC embodies latency constraints (31 frames for the RA mode and 0 for the LD mode, the same as HM and VTM), whereas other INR models do not. These demonstrate the strong rate-distortion performance of the proposed model. Among all codecs tested, the next-generation standard VTM (RA) remains to be the best performing baseline.

We present qualitative visual comparisons between frames reconstructed by HiNeRV (the best INR-based video codec) and the proposed PNVC in Figure 5. The visual results show that our method is able to yield visually pleasing reconstructions, accurately capturing the fine texture details and fast/complex motions present in the frame. Please see Supplementary for more thorough comparisons.

The complexity figures of PNVC and other benchmark codecs, including encoding and decoding speeds (FPS), full model size (number of parameters) and decoding complexity (MACs/pixel), are summarized in Table 1. These are measured by averaging over a GOP of 1080P videos, including a full roll-out of the entropy coding process, on a PC with an NVIDIA 3090 GPU and an Intel Core i7-12700 CPU. It can be observed that our model achieves a higher decoding speed, comparable to the conventional and INR baselines and considerably faster than all the neural video codecs. The PNVC decoder is also similar in size to INR models and much smaller than neural codecs. The well-roundness of our proposed model is illustrated by the Radar plot, as shown in Figure 1, in which decoding speed, encoding latency, BD-rate in PSNR and MS-SSIM of the selected, top-performing baselines are visualized. It is evident that the proposed PNVC yields the most-balanced and also the best overall rate-distortion-complexity trade-off. It is also noted that the encoding speed of PNVC is still relatively low (so as other INR-based codecs), when compared to autoencoder-based neural compression models (and VTM), which limits real-time deployment (e.g., video conferencing). However, the benefits of Low Delay and Random Access compatibility, low decoding complexity, and competitive rate quality performance still offer great potential to support widely used streaming scenarios, which does not require real-time encoding ability.

To validate the contribution of each design component in the proposed PNVC model, we conducted ablations using the following model variants. The corresponding BD-rate results are summarized in Table 2.