Idempotence and Perceptual Image Compression

Idempotence of image codec refers to the codec’s stability to re-compression. More specifically, denote the original image as $X$, the encoder as $f(.)$, the code as $Y=f(X)$, the decoder as $g(.)$ and the reconstruction as $\hat{X}=g(Y)$. We say that the codec is idempotent if

i.e., the codec is idempotent if the re-compression of reconstruction $\hat{X}$ produces the same result.

In this paper, we use Blau & Michaeli (2018)’s definition of perceptual quality. More specifically, we say that the reconstructed image $\hat{X}$ has perfect perceptual quality if

where $p_{X}$ is the source image distribution and $p_{\hat{X}}$ is the reconstruction image distribution. In this paper, we slightly abuse the word ”perception” for this divergence-based perception, and use the word ”human perception” for human’s perception instead.

We first show that ideal conditional generative codec satisfies idempotence, i.e, $\hat{X}\sim P_{X|Y}\Rightarrow f(\hat{X})=Y$. Given a specific value $y$ with non-trivial probability $p_{Y}(Y=y)\neq 0$, we define the inverse image of $y$ as $f^{-1}[y]=\{x|f(x)=y\}$. By definition, all the elements $x\in f^{-1}[y]$ encode into $y$. Then if we can show $\hat{X}\in f^{-1}[y]$, we can show that this perceptual codec is idempotent.

In previous section, we have shown that perceptual quality brings idempotence. It is obvious that the simple converse is not true. A counter-example is JPEG2000 (Taubman et al., 2002), which is idempotent by design but optimized for MSE. However, we show that with unconditional generative model, idempotence does bring perceptual quality.

Besides, those results can be extended to image restorations (See Appendix. A).

Theorem  2 implies a new paradigm to achieve perceptual image compression by sampling from a pre-trained unconditional generative model with the idempotence constraint. More specifically, we can rewrite the left hand side of Eq. 9 in Theorem 2 as

Then the problem has exactly the same form as a broad family of works named ‘generative model inversion’ for image super-resolution and other restoration tasks (Menon et al., 2020; Daras et al., 2021; Wang et al., 2022; Chung et al., 2022a). From their perspective, $f(.)$ is the down-sample operator, $Y$ is the down-sampled image and $||f(\hat{X})-Y||^{2}$ is the ‘consistency’ penalization that secures the super-resolved image $\hat{X}$ corresponds to the input down-sampled image. The $\hat{X}\sim p_{X}$ is the ‘realness’ term, and it ensures that $\hat{X}$ lies on the natural image manifold (Zhu et al., 2016). And solving Eq. 11 is the same as finding a sample that satisfies the consistency, which is the inverse problem of sampling. Therefore, they name their approach ‘inversion of generative model’. For us, the $f(.)$ operator is the encoder, and $Y$ is the bitstream. And so long as our encoder is differentiable, we can adopt their inversion approach for image super-resolution to inverse the codec.

To better understand our proposed approach, we describe the detailed procedure of communicating an image from sender to receiver and implementation details. We start with the pre-trained model that is required for the sender and receiver:

• •

  The sender and receiver share a MSE optimized codec with encoder $f_{0}(.)$, decoder $g_{0}(.)$. Despite we can use any base codec, MSE-optimized codec leads to tightest MSE bound.

• •

  The receiver has a unconditional generative model $q_{X}$ approximating the source $p_{X}$.

And the full procedure of encoding and decoding an image with our codec is as follows:

• •

  The sender samples an image from the source $X\sim p_{X}$.

• •

  The sender encodes the image into bitstream $Y=f_{0}(X)$, with the encoder of pre-trained MSE codec. And $Y$ is transmitted from sender to receiver.

• •

  Upon receiving $Y$, the receiver inverses a generative model with idempotence constraint:

  $$\displaystyle\min||f_{0}(\hat{X})-Y||^{2}\textrm{, s.t. }\hat{X}\sim q_{X}.$$

  (12)

For practical implementation, we can not directly use the binary bitstream $Y$ for idempotence constraint. This is because most of generative model inversion methods (Menon et al., 2020; Daras et al., 2021; Chung et al., 2022a) use the gradient $\nabla_{\hat{X}}||f_{0}(\hat{X})-Y||^{2}$. On the other hand, for most NIC approaches, the entropy coding with arithmetic coding (Rissanen & Langdon, 1979) is not differentiable. We propose two alternative constraints to this problem:

• •

  y-domain constraint: We do not constrain the actual bitstream $Y$. Instead, we constrain the quantized symbols before arithmetic coding. Further, we use straight-through-estimator to pass the gradient through quantization. And we call this constraint y-domain constraint.

• •

  x-domain constraint: On the other hand, we can also decode the re-compressed image and constrain the difference between the MSE reconstructed image of source $X$ and the MSE reconstructed image of sample. More specifically, instead of solving Eq. 12, we solve:

  $$\displaystyle\min||g_{0}(f_{0}(\hat{X}))-g_{0}(Y)||^{2}\textrm{, s.t. }\hat{X}\sim q_{X},$$

  (13)

  Similarly, the quantization is relaxed by STE to allow gradient to pass. And we call this constraint x-domain constraint.

The y-domain and x-domain constraint correspond to two idempotence definitions in Eq. 1. And when $g_{0}(.)$ is MSE optimal, those two constraints are equivalent (See Theorem 3 of Appendix. A). And beyond STE, other gradient estimators such as additive noise, multi-sample noise and stochastic gumbel annealing can also be adopted (Ballé et al., 2017; Xu et al., 2022; Yang et al., 2020).

Compared with previous works using conditional generative model (Mentzer et al., 2020; Muckley et al., 2023), our proposed approach does not require specific conditional generative model for different codec and bitrate. It only requires one unconditional generative model, and it can be directly applied on even pre-trained MSE codec. Furthermore, it is theoretically equivalent to conditional generative codec, which means that it conveniently shares the same theoretical properties in terms of rate-distortion-perception trade-off (Blau & Michaeli, 2019).

Compared with previous proof of concept using unconditional generative model (Ho et al., 2020; Theis et al., 2022), our proposed approach does not require bits-back coding (Townsend et al., 2018) or sample communication (Li & El Gamal, 2018), which might have rate or complexity overhead. Further, it is implemented as a practical codec. Moreover, our approach uses exactly the same bitstream $Y$ as MSE codec. This means that the receiver can choose to reconstruct a MSE optimized image or a perceptual image and even achieve perception-distortion trade-off (See Appendix. B.3).

Metrics We use Fréchet Inception Distance (FID) to measure perceptual quality. This is because we use Blau & Michaeli (2018)’s definition of perceptual quality, which is the divergence between two image distributions. And FID is the most common choice for such purpose. We use MSE and Peak-Signal-Noise-Ratio (PSNR) to measure distortion, which are the default choice for image codec. We use bpp (bits per pixel) to measure bitrate. To compare codec with different bitrate, we adopt Bjontegaard (BD) metrics (Bjontegaard, 2001): BD-FID and BD-PSNR. Those BD-metrics can be seen as the average FID and PSNR difference between codec over their bitrate range.

Datasets Following previous works in unconditional image generation (Karras et al., 2019; Ho et al., 2020), we train our unconditional generative models on FFHQ (Karras et al., 2019) and ImageNet (Deng et al., 2009) dataset. As (Chung et al., 2022a), we split the first 1000 images of FFHQ as test set and the rest for training. And we use first 1000 images of ImageNet validation split as test set and use the ImageNet training split for training. To test the generalization ability of our method on other datasets, we also use first 1000 images of COCO (Lin et al., 2014) validation split and CLIC 2020 (Toderici et al., 2020) test split as additional test set. As previous works (Karras et al., 2019; Ho et al., 2020) in unconditional generative model, we central crop image by their short edge and rescale them to $256\times 256$.

Previous State-of-the-art Perceptual Codec We compare our approach against previous state-of-the-art perceptual codec: HiFiC (Mentzer et al., 2020), Po-ELIC (He et al., 2022b), CDC (Yang & Mandt, 2023) and ILLM (Muckley et al., 2023). HiFiC is the first codec achieving perceptual near lossless compression with very low bitrate. Po-ELIC is the winner of CVPR CLIC 2022, a major competition for perceptual codec. CDC is a very recent perceptual codec using conditional diffusion model. ILLM is the latest state-of-the-art perceptual codec. We use the official model to test HiFiC, CDC and ILLM. We contact the authors of Po-ELIC to obtain the test results on our datasets. For fairness of comparison, we also test a re-trained version of HiFiC (Mentzer et al., 2020) and ILLM (Muckley et al., 2023) on FFHQ and ImageNet dataset. So that their training dataset becomes the same as ours. We acknowledge that there are other very competitive perceptual codec (Iwai et al., 2021; Ma et al., 2021; Agustsson et al., 2022; Goose et al., 2023; Hoogeboom et al., 2023), while we have not included them for comparison as they are either unpublished yet or do not provide pre-trained model for testing.

Unconditional Generative Model As we have discussed, our approach requires an unconditional generative model for inversion. The most adopted models for inverse problem are StyleGAN family (Karras et al., 2019) and DDPM (Ho et al., 2020). Multiple approaches are proposed to invert StyleGAN and DDPM. And they are shown to be effective for image super-resolution and other restoration tasks (Menon et al., 2020; Daras et al., 2021; Roich et al., 2022; Daras et al., 2022; Wang et al., 2022; Chung et al., 2022b; a). For StyleGAN inversion, we choose two mostly adopted approach: PULSE (Menon et al., 2020) and ILO (Daras et al., 2021). For DDPM inversion, we choose MCG (Chung et al., 2022b) and DPS (Chung et al., 2022a) as they are the most popular methods for non-linear inversion.

To find the most suitable unconditional generative model, we compare StyleGAN2 + PULSE, StyleGAN2 + ILO, DDPM + MCG and DDPM + DPS. The implementation details are presented in Appendix. B.1. As Tab. 1 and Fig. 3 show, compared with DDPM + DPS, other three methods either have too high FID or have MSE significantly larger than $2\times$MSE of ELIC. Further, they look visually less desirable and less similar to the source. Therefore, we use DDPM + DPS inversion in later experiments.

Diversity of Reconstruction Another feature of our approach is the reconstruction diversity. Though it is theoretically beneficial to adopt stochastic decoder for conditional generative codec (Freirich et al., 2021), most of previous works (Mentzer et al., 2020; He et al., 2022b; Muckley et al., 2023) adopt deterministic decoder and lost reconstruction diversity. On the other hand, our approach preserves this diversity. In Fig. 5, we show the pixel-level standard deviation $\sigma$ of our reconstruction on the first image of Fig. 4 with sample size 16. Further, in Fig. 5, 16 of Appendix B.3, we present alternative reconstructions, which differ a lot in detail but all have good visual quality.

Idempotence Consider we have a MSE codec. The first time compression is $\hat{X}^{(1)}=g_{0}(f_{0}(X))$, and re-compression is $\hat{X}^{(2)}=g_{0}(f_{0}(\hat{X}^{(1)}))$. We evaluate its idempotence by MSE between first time compression and re-compression $||\hat{X}^{(1)}-\hat{X}^{(2)}||^{2}$. According to our theoretical results, we can also use our approach to improve idempotence of base codec. Specifically, after first time compression, we can decode a perceptual reconstruction $\hat{X}^{(1)}_{p}$ by Eq. 12, and use this perceptual reconstruction for re-compression $\hat{X}^{(2)^{\prime}}=g_{0}(f_{0}(\hat{X}^{(1)}_{p}))$. By Theorem 1, we should have $||\hat{X}^{(1)}-\hat{X}^{(2)^{\prime}}||^{2}=0$, i.e., this augmented codec is strictly idempotent. In practice, as shown in Tab. 3, the augmented codec’s re-compression MSE is smaller than the base codec. This indicates that our proposed approach can acts as an idempotence improvement module for MSE codec.

Complexity Compared with conditional generative codec (e.g., HiFiC (Mentzer et al., 2020) and ILLM (Muckley et al., 2023)), our approach has lower training but higher testing complexity. Compared with inversion based image super-resolution (e.g. MCG (Chung et al., 2022b), DPS (Chung et al., 2022a)), our approach’s complexity is similar. Tab. 4 shows that our approach greatly reduces training time. Specifically, for conditional generative codec, a conditional generative model is required for each rate. If the codec supports $K$ rates, $K$ generative models should be trained. For HiFiC and ILLM, each model takes approximately 1 week. And the codec needs $K$ weeks of training. While for us, this extra training time is $0$ as we can utilize pre-trained unconditional generative model. While as the inversion-based image super-resolution, our method requires gradient ascent during testing, which increases the testing time from $\sim 0.1$s to $\sim 60$s.

Idempotence is a consideration in practical image codec (Joshi et al., 2000). In traditional codecs like JPEG (Wallace, 1991) and JPEG2000 (Taubman et al., 2002), the encoding and decoding transforms can be easily made invertible, thus idempotence can be painlessly achieved in these codec. In NIC, however, it requires much more efforts to ensure the idempotence (Kim et al., 2020). This is because neural network based transforms are widely adopted in NIC, and making these transforms invertible either hinders the RD performance (Helminger et al., 2021) or complicates the coding process (Cai et al., 2022). To the best of our knowledge, we are the first to build theoretical connection between idempotence and perceptual quality.

The majority of perceptual image codec adopt conditional generative models and successfully achieve near lossless perceptual quality with very low bitrate (Rippel & Bourdev, 2017; Tschannen et al., 2018; Mentzer et al., 2020; Agustsson et al., 2022; Yang & Mandt, 2023; Muckley et al., 2023; Hoogeboom et al., 2023). Blau & Michaeli (2019) show that conditional generative codec achieves perfect perceptual quality with at most twice of optimal MSE. Yan et al. (2021) further prove that this approach is the optimal among deterministic encoders. The unconditional generative model-based codec is explored by Ho et al. (2020); Theis et al. (2022). Though no actual codec is implemented, they reveal the potential of unconditional generative model in image compression. To the best of our knowledge, our approach is the first actual codec using unconditional generative model. It does not require training new models and is equivalent to conditional generative model-based perceptual codec.

After the submission deadline, we become aware of an alternative of Theorem. 1 and Theorem. 2, which is presented in a concurrent paper by Ohayon et al. (2023).