BigFUSE: Global Context-Aware Image Fusion in Dual-View Light-Sheet Fluorescence Microscopy with Image Formation Prior

BigFUSE first rethinks dual-view LSFM fusion from a Bayesian perspective, that is, the conditional probability of fused volume in terms of ”in-focusness”, is given on not only a pair of image inputs, but also prior knowledge that these two images are illiminated from opposing orientations respectively:

where ${\rm{Y}}\in{{\mathbb{R}}^{M\times N}}$ is our predicted fusion with minimal light scattering effect. ${{\rm{X}}^{a}}$ and ${{\rm{X}}^{b}}$ are two image views illuminated by light source $a$ and $b$, respectively. We choose ${{\rm{Y}}}\in\{{{\rm{X}}^{a}},{{\rm{X}}^{b}}\}$ depending on their competitive image clarity at each pixel. Priors $\cal P$ denote our empirical favor of ${{\rm{X}}^{a}}$ against ${{\rm{X}}^{b}}$ at each pixel if photons travel through fewer scattering tissues from source a than b, and vice versa. Due to the non-positive light scattering effect along illumination path, there is only one focus-defocus change per column for dual-view LSFM in Fig. 1a. Thus, fusion is equivalent to estimating a focus-defocus boundary $\omega$ defined as a function associating focus-defocus changes to column indexes:

which can be further reformulated by logarithm:

where ${\omega_{i}}$ denotes the focus-defocus changeover at i-th column, ${{\rm{X}}_{:,i}}$ is the i-th column of ${\rm{X}}$. Next, estimating $\omega$ is decomposed into: (i) define the column-wise image clarity, i.e., log-likelihood ${\rm{log}}({p(({\rm{X}}_{:,i}^{a},{\rm{X}}_{:,i}^{b})|{\omega_{i}})})$; (ii) consider the belief on a spatially smooth focus-defocus boundary, namely log-prior ${\rm{log}}(p(\omega))$.

In LSFM, log-likelihood ${\rm{log}}({p(({\rm{X}}_{:,i}^{a},{\rm{X}}_{:,i}^{b})|{\omega_{i}})})$ can be interpreted as the probability of observing $({\rm{X}}_{:,i}^{a},{\rm{X}}_{:,i}^{b})$ given the hypothesis that the focus-defocus change is determined as $\omega_{i}$ in the i-th column:

where $\oplus$ is a concatenation, $c(\bullet)$ is the columne-wise image clarity to be defined.

With log-likelihood ${\rm{log}}({p(({\rm{X}}_{:,i}^{a},{\rm{X}}_{:,i}^{b})|{\omega_{i}})})$ considering the focus-defocus consistency along illuminations using image formation prior in LSFM, BigFUSE then ensures consistency across columns in $p(\omega)$. Specifically, the smoothness of $\omega$ is characterized as a window-based polynomial fitness using linear least squares:

where ${\hat{\upsilon}_{i}}={c_{i}}{{\rm{\Omega}}_{i}}=[{c_{i,0}},{c_{i,1}},\ldots,{c_{i,Q}}][\boldsymbol{\omega}_{i-s}^{\rm{T}},\boldsymbol{\omega}_{i-s+1}^{\rm{T}},\ldots,\boldsymbol{\omega}_{i+s}^{\rm{T}}]$, ${\boldsymbol{\omega}_{i}}=[\omega_{i}^{Q},\omega_{i}^{Q-1},\ldots,1]$, ${c_{i}}$ is the parameters to be estimated, the sliding window is with a size of $2s+1$.

Finally, in order to estimate the $\omega$ together with the fitting parameter $c$, BigFUSE reformulates the the posterior distribution in Eq. (2) as follows:

where $\lambda$ is the trade-off parameter. Here, BigFUSE alternates the estimations of $\omega$, and $c$, and iterates until the method converges. Specifically, given $c^{(n)}$ for the n-th iteration, ${{\omega}_{i}^{(n+1)}}$ is estimated by maximizing (E-step):

which can be solved by iterating over $\{i|1\leq i\leq M\}$. BigFUSE then updates $c_{i}^{(n+1)}$ based on least squares estimation:

Additionally, ${\rm{A}}^{n+1}$ is updated based on Eq. (10) subject to $\omega^{(n+1)}$ (M-step). BigFUSE proposes to initialize $\omega$ based on ${\rm{F}}^{a}$ and ${\rm{F}}^{b}$:

where $|{\rm{\Phi}}|$ denotes the total number of elements in ${\rm{\Phi}}$.

We compare BigFUSE to four baseline methods: (i) DWT [16]: a multi-resolution image fusion method using discrete wavelet transform (DWT); (ii) NSCT [18]: another multi-scale image fuser but in the NSCT domain; (iii) dSIFT [9]: a dense SIFT-based focus estimator in the image space; (iv) BF [19]: a focus-defocus boundary detection method that considers region consistency in focus; and two BigFUSE variations: (v) $\mathcal{S}(\bullet)$: built by disabling smooth constraint; (vi) $\mathcal{P}(\bullet)$: formulated by replacing the weighted summation of pixel-level clarity measures for overall characterization, by a simple average.

To access the blind image fusion performance, we adopt three fusion quality metrics, $Q_{mi}$ [4], $Q_{g}$ [12] and $Q_{s}$ [13]. Specifically, $Q_{mi}$, $Q_{g}$ and $Q_{s}$ use mutual information, gradient or image quality index to quantify how well the information or features of the inputs are transferred to the result, respectively. In the simulation studies where ground truth is available, mean square error (EMSE) and structural similarity index (SSIM) are used for quantification.

We first evaluate BigFUSE in fusing dual-view LSFM blurred by simulation. Here, we blur a mouse brain sample collected in [2] with spatially varying Gaussian filter for thirty times, which is chemically well-cleared and thus provides an almost optimal ground truth for simulation, perform image fusion, and compare the results to the ground truth. BigFUSE achieves the best EMSE and SSIM, statistically surpassing other approaches (Table. 1, $p<0.05$ using Wilcoxon signed-rank test). Only BigFUSE and BF realize information fusing without damaging original images with top two EMSE. In comparison, DWT, NSCT and dSIFT could distort the original signal when fusing (green box in Fig. 2).

BigFUSE is then evaluated against baseline methods on real dual-view LSFM. A large sample volume, zebrafish embryo ($282\times 2048\times 2048$ for each view), is imaged using a Flamingo Light Sheet Microscope. BigFUSE takes roughly nine minutes to process this zebrafish embryo, using a T4 GPU with 25 GB system RAM and 15 GB GPU RAM. In Fig. 4, inconsistent boundary is detected by BF, while methods like DWT and NSCT generate structures that do not exist in either input. Moreover, only BigFUSE can exclude ghost artifact from the result (red box), as BigFUSE is the only pipeline that considers image formation prior. Additionally, we demonstrate the impact of bigFUSE on a specific downstream task in Fig. 5, i.e., segmentation by Cellpose. Only the fusion result provided by bigFUSE allows reasonable predicted cell pose, given that ghosting artifacts are excluded dramatically. This explains why the $Q_{s}$ in Fig. 3 is suboptimal, since BigFUSE do not allow for the transmission of structural ghosts to the output.