Deep learning based Image Compression for Microscopy Images: An Empirical Study

Image compression is the process of reducing the size of digital images while retaining the useful information for reconstruction. This is achieved by removing redundancies in the image data, resulting in a compressed version of the original image that requires less storage space and can be transmitted more efficiently. In many fields of research, including microscopy, high-resolution images are often acquired and processed, leading to significant challenges in terms of storage and computational resources. In particular, researchers in the microscopy image analysis field are often faced with infrastructure limitations, such as limited storage capacity or network bandwidth. Image compression can help mitigate such challenges, allowing researchers to store and transmit images efficiently without compromising their quality and validity. Lossless image compression refers to the compression techniques preserving every bit of information in data and make error-free reconstruction, ideal for applications where data integrity is paramount. However, the limited capability in size reduction, e.g., 2 $\sim$ 3 compression ratio walker2023comparison is far from sufficient for easing the data explosion crisis. In this work, we focuses on lossy compression methods, where some information lost may occur but can yield significantly higher compression ratio. Despite lossy compression techniques (both classic and deep learning based) are widely employed in the computer vision field, their feasibility and impact in the field of biological microscopy images remain largely underexplored.

In this paper, we proposed a two-phase evaluation pipeline, compression algorithms comparison and downstream tasks analysis in the context of microscopy images. In order to fully explore the impact of lossy image compression on downstream image analysis tasks, we employed a set of label-free models, a.k.a., in-silico labelling christiansen2018silico . Label-free model denotes a deep learning approach capable of directly predicting fluorescent images from transmitted light bright-field images Ounkomol2018Label-freeMicroscopy . Considering the large amount bright-field images are being used in regular biological studies, it is of great importance that such data compression techniques can be utilized without compromising the prediction quality.

Through intensive experiments we demonstrated that deep learning based compression methods can outperform the classic algorithms in terms of compression ratio and post-compression reconstruction quality, and their impact on the downstream label-free task, indicating their huge potentials in the bioimaging field. Meanwhile, we made an preliminary attempt to build 3D compression models and reported the current limitation and possible future directions. Overall, we want to raise the awareness of the importance and potentials of deep learning based compression techniques, and hopefully help a strategical planning of future data infrastructure for bioimaging.

Specifically, the main contribution of the paper is:

1. 1.

   Benchmark common classic and deep learning-based image compression techniques in the context of 2D grayscale bright-field microscopy images.

2. 2.

   Empirically investigate the impact of data compression to the downstream label-free tasks.

3. 3.

   Expand the scope of the current compression analysis for 3D microscopy images.

The remaining of this paper is organized as follows: section 2 will introduce classic and deep learning-based image compression techniques, followed by the method descriptions in section 3 and experimental settings in section 4. Results and discussions will be presented in section 5 with conclusions in section 6.

The classic data compression techniques are well studied in the last few decades, with the development of JPEG Wallace1992jpeg , a popular lossy compression algorithm since 1992, and its successors JPEG 2000 Marcellin2002AnJPEG-2000 , JPEG XR Dufaux2009TheNutshell , etc. In recent years some more powerful algorithms, such as Limited Error Raster Compression (LERC) are proposed HowAPI . Generally, the compression process approximately involves the following steps: color transform (with optional downsampling), Domain transform (e.g., Discrete Cosine Transform (DCT) ahmed1974discrete in JPEG), quantization and further lossless coding (e.g. run-length encoding (RLE) or Huffmann coding huffman1952method ).

Recently, deep learning based image compression gains its popularity thanks to the significantly improved compression performance. Roughly speaking, a deep learning based compression model consists of two sub-networks: a neural encoder $f$ that compresses the image data and a neural decoder $g$ that reconstructs the original image from the compressed representation. Besides, the latent representation will be further losslessly compressed by some entropy coding techniques (e.g. arithmetic coding Rissanen1979Arithmetic ) as seen in fig. 1. Specially, the latent vector will be firstly discretized into $\mathbf{z}$: $\mathbf{z}=\llbracket f(\mathbf{X})\rrbracket,\mathbf{z}\in\mathbb{Z}^{n}$. Afterwards, $\mathbf{z}$ will be encoded/decoded by the entropy coder ($f_{e}/g_{e}$) and decompressed by the neural decoder $g$: $\hat{\mathbf{X}}=g(g_{e}(f_{e}(\mathbf{z})))$. The objective is to minimize the loss function containing Rate-Distortion trade-off cover1999elements ; shannon1959coding :

	$$\displaystyle\mathcal{R}:=\mathbb{E}\left[-\log_{2}P(\mathbf{z})\right]$$		(1)
	$$\displaystyle\mathcal{D}:=\mathbb{E}[\rho(\mathbf{X},\hat{\mathbf{X}})]$$		(2)
	$$\displaystyle\mathcal{L}:=\mathcal{R}+\lambda\cdot\mathcal{D}$$		(3)

where $\mathcal{R}$ corresponds to the rate loss term, which highlights the compression ability of the system. $P$ is the entropy model that provides prior probability to the entropy coding, and $-\log_{2}P(\mathbf{\cdot})$ denotes the information entropy and can approximately estimate the optimal compression ability of the entropy encoder $f_{e}$, defined by the Shannon theory shannon1959coding ; shannon1948mathematical . $\mathcal{D}$ is the distortion term, which can control the reconstruction quality. $\rho$ is the norm or perceptual metric, e.g. MSE, MS-SSIM wang2003multiscale , etc. The trade-off between these two terms is achieved by the scale hyper-parameter $\lambda$.

Since the lossless entropy coding entails the accurate modeling of the prior probability of the quantized latent representation $P(\mathbf{z})$, Ballé et al. Balle2018VariationalHyperprior justified that there exists statistical dependencies in the latent representation using current fully factorized entropy model, which will lead to suboptimal performance and not adaptive to all images. To further improve the entropy model, Ballé et al. proposes a hyperprior approach Balle2018VariationalHyperprior , where a hyper latent $\mathbf{h}$ (also called side information) is generated by the auxillary neural encoder $f_{a}$ from the latent space $\mathbf{y}$: $\mathbf{h}=f_{a}(\mathbf{y})$, then the scale parameter of the entropy model can be estimated by the output of the auxillary decoder $g_{a}$: $\phi=g_{a}(E_{a}(\mathbf{h}))$, so that the entropy model can be adaptively adjusted by the input image $\mathbf{x}$, with the bit-rate further enhanced. Minnen et al. Minnen2018JointCompression extended the work to get the more reliable entropy model by jointly combining the data from the above mentioned hyperprior and the proposed autoregressive Context Model.

Besides the improvement in the entropy model, lots of effort are also put into the enhancement of the network architecture. Ballé et al. balle2015density replaced the normal RELU activation to the proposed Generalized Division Normalization (GDN) module in order to better capture the image statistics. Johnston et al. johnston2019computationally optimized the GDN module in a computational efficient manner without sacrificing the accuracy. Cheng et al. Cheng2020LearnedModules introduced the skip connection and attention mechanism. The transformer-based auto-encoder was also reported for data compression in recent years zhu2021transformer .

The evaluation pipeline was proposed in this study to benchmark the performance of the compression model in the bioimage field and estimate their influence to the downstream label-free generation task. Illustrated in fig. 2, the whole pipeline contains two parts: compression part: $x\xrightarrow{g\circ f}\hat{x}$ and downstream label-free part: $(x/\hat{x})\xrightarrow{f_{l}}(y/\hat{y})$, where the former is designed to measure the Rate-Distortion performance of the compression algorithms and the latter aims to quantify their influence to the downstream task.

During the compression part, the raw image $x$ will be transformed to the reconstructed image $\hat{x}$ through the compression algorithm $g\circ f$:

$$\hat{x}=(g\circ f)(x)=g(f(x))$$

(4)

where $f$ represents the compression process and $g$ denotes the decompression process. Note that the compression methods could be both classic strategies (e.g. jpeg) and deep-learning based algorithms. The performance of the algorithm can be evaluated through Rate-Distortion performance, as explained in eqs. 1, 2 and 3.

In the downstream label-free part, the prediction will be made by the model $f_{l}$ using both the raw image $x$ and the reconstructed image $\hat{x}$:

$$y=f_{l}(x),\hat{y}=f_{l}(\hat{x})$$

(5)

The evaluation to measure the compression influence to the downstream tasks is made by:

	$$\displaystyle V=[y,\hat{y},y_{t}]\quad,\quad\rho=[\rho_{i}]_{i=1}^{4}$$		(6)
	$$\displaystyle S=\left\{(V_{i},V_{j})\mid i,j\in\{1,2,3\},i\neq j\right\}$$		(7)
	$$\displaystyle L=\left[\rho(V_{i},V_{j})\mid(V_{i},V_{j})\in S\right]$$		(8)

where the evaluation metric L is the collection of different metrics $\rho_{i}$ on different image pairs $S_{i}$. V is the collection of the raw prediction $y$, prediction made by the reconstructed image $\hat{y}$ and the ground truth $y_{t}$. S is formed by pairwise combinations of elements from V. $\rho$ represents the metric we used to measure the relation between image pairs. In this study we totally utilized four metrics: Mean Squared Error (MSE), Structural Similarity Index Measure (SSIM), Peak Signal-to-Noise Ratio (PSNR) and Pearson Correlation.

To conclude, through the above proposed two-phase evaluation pipeline, the compression performance of the compression algorithm will be fully estimated and their impact to the downstream task will also be well investigated.

In this section, we will present and analyze the performance of the image compression algorithms and their impact on the downstream label-free task, using the proposed two-phase evaluation pipeline.

In this research, we proposed a two-phase evaluation pipeline, in order to benchmark the rate-distortion performance of different data compression techniques in the context of grayscale microscopic brightfield images and fully explored the influence of such compression to the downstream label-free task. We found that AI-based image compression methods can significantly outperform classic compression methods and have minor influence on the following label-free model prediction. Despite some limitations, we hope our work can raise the awareness of the application of deep learning-based image compression in the bioimaging field and provide insights into the way of integration with other AI-based image analysis tasks.