Generation and Recombination for Multifocus Image Fusion with Free Number of Inputs

Feature reconstruction-based methods usually fuse the extracted multifocus image features and then generate the fused images based on the fused features. In such methods, the representation of image features is a crucial factor that influences the fusion image. Currently, the commonly used methods for image feature representation include multi-scale decomposition (MSD) [6, 1, 7, 8, 9], sparse representation (SR)[10, 11, 12, 13, 14, 15, 16, 17], and deep learning (DL)[18, 4, 19, 20, 21, 22, 23, 24]. MSD-based methods often employ multiscale image analysis method such as Wavelet Transform[6, 1], Nonsubsampled Contourlet Transform[7, 2], and Curvelet Transform[8] to represent the the source images. However, these methods employ fixed basis to represent the input image, so their sparse representation ability is weak. SR-based methods have been widely used in multifocus image fusion since they can learn a set of basis from the training samples to sparsely represent the image. Specifically, Yang et al.[14] proposed a SR-based method for restoration and fusion of noisy multifocus images. Zhang et al.[15] developed an analysis-synthesis dictionary pair learning method for fusion of multi-source images, including multifocus images. Li et al.[16] devised an edge-preserving method for fusion of multi-source noisy images. This method can perform fusion and denoising of multifocus noisy images in one framework.

In DL-based methods, the feature reconstruction-based approach generally includes three steps: feature extraction, feature fusion, and fusion image reconstruction. Since the fusion result is reconstructed from fused features, there may be some errors between the reconstructed and the ideal results. This demonstrates that the result obtained by the method based on feature reconstruction is not ideal. To alleviate this issue, Zhao et al.[18] proposed a CNN-based multi-level supervised network for multifocus image fusion, which utilized multi-level supervision to refine the extracted features. Zang et al.[19] introduced spatial and channel attention to enhance the salient information of the source images, effectively avoiding the loss of details. To address the challenge raised by the absence of ground truth, Jung et al.[20] proposed an unsupervised CNN framework for fusion of multi-source images, where an unsupervised loss function was deigned for unsupervised training.

In an unsupervised learning mode, to prevent the loss of source image details, Mustafa et al.[25] added dense connections into feature extraction network at different levels, improving the visual quality of the fused images. Hu et al.[22] proposed a mask prior network and a deep image prior network to achieve multifocus image fusion with zero-shot learning, thereby solving the problem caused by inconsistency between synthetic images and real images. Although the feature reconstruction-based method can generate images that look natural, there is still a certain gap between the generated and the ideal results due to the information loss during the feature extraction and image reconstruction.

Pixel recombination-based methods usually combine the focused pixels of the source image to construct the fused image. Therefore, how to accurately detect the pixels in the focus area is critical to them. To this end, Liu et al.[26] proposed a twin network and trained with a large number of samples generated by gaussian filtering to achieve the detection of focused pixels. Guo et al.[27] employed a fully connected network to explore the relationship between pixels in the entire image, enabling the determination of pixel focus property. Ma et al.[28] introduced an encoder-decoder feature extraction network consisting of SEDense Blocks to extract the features from source images, and then the focused pixel is distinguished by measuring spatial frequency of the features. Xiao et al.[29] proposed a U-Net with global-feature encoding to determine the focus property of pixel. Ma et al.[30] proposed SMFuse for focused pixel detection where the repeated blur map and the guided filter are both used to generate the binary mask. Considering the complementarity of features at different receptive fields, Liu et al.[31] proposed multiscale feature interactive network to improve detection performance of the focused pixel.

The aforementioned methods usually construct fused image using the detected focused pixels. However, it is extremely challenging to determine the focus property of hard pixels located at the smooth regions or the boundaries of focused regions. If the fused image is obtained like above, severe artifacts will be introduced into the fused result. To solve this problem, Li et al.[32] proposed an effective multifocus image fusion method based on non-local means filtering, in which the detection result of focused pixel in the transition regions between focused and defocused areas is smoothed by the non-local means filtering, and then the smoothed result is used to guide the fusion process. Similarly, Liu et al.[5] combined feature reconstruction with focused pixel recombination, and then developed an effective method for fusion of multifocus images. Although above methods are effective, they ignore the negative impact of hard pixels on the fusion quality. Additionally, the most of existing methods often assume that the number of input images is two and the focus property is complementary. As a result, they cannot fuse the multiple images in a parallel mode. Alternating fusion strategy is a commonly used method to solve this problem, but it tends to lose the information of the source images. In contrast, the developed GRFusion can fuse multiple images simultaneously and can solve the problems caused by hard pixels effectively.

As shown in Fig. 2, the proposed method mainly consists of three components: hard pixel detection (HPD), full-focus image generation with multi-directional feature embedding (FFIG-MFE), and fusion result construction guided by hard pixels (FRC-G-HP). HPD is mainly used to determine which pixels are from the focus ambiguity area. This allows us to take specific solution to address the fusion of hard pixels. FFIG-MFE is used to generate an all-focus fusion image based on the fused features of the source images. FRC-G-HP combines the all-focus fusion result and the source images to generate the final fused result under the guidance of detection results of focused pixels. In the following section, we will elaborate on these components carefully.

Due to the ambiguity of the focus properties of hard pixels, it’s not optimal to construct the final fused result by selecting the focused pixels from source images. To reduce the negative effect of hard pixels on fusion result, we devise a full-focus image generation method with multi-directional feature embedding (FFIG-MFE) to construct the fusion result of hard pixels. FFIG-MFE consists of multi-directional edge extraction (MDEE), weight generator and edge feature embedding (EFE). As shown in Fig.2, multi-directional edge extraction in FFIG-MFE can be achieved by the designed multi-directional edge extraction kernels presented in Fig.5. Specifically, the size of each kernel is ${3\times 3}$ with the value of center position being 1. In FFIG-MFE, there are eight kernels corresponding to eight different directions are used to extract the edges from the source images. The multi-directional edges are denoted as:

where $\bm{F}_{i,{k_{n}}}$ denotes the edge features of the source image $\bm{I}_{i}$ in the $n$-th direction.

To fully utilize the features at eight direction to generate a full-focus image, a weight generation mechanism is designed to dynamically produce the weights for integrating multi-direction features. As shown in Fig.6, the weight generator consists of ${1\times 1}$ convolutional layer and a residual branch, which first employs, ${1\times 1}$ convolutional operation to reduce the number of channels to $1/4$ of the original number, and then utilizes a ${3\times 3}$ convolutional operation and a ${3\times 3}$ convolutional operation with dilation rate of $d(d=1,3,5,7)$ to extract features under different respective field. The outputs of ${1\times 1}$ convolutional layer and residual block are concatenated, and then weights $\omega_{i}^{n}(x,y)$ can be achieved after ${1\times 1}$ convolution and softmax. With $\omega_{i}^{n}(x,y)$, the integrated multi-directional edge features can be represented as:

Since $\bm{F}_{edge,i}$ contains the focus property of source image $\bm{I}_{i}$, we use it to highlight the effect of focused features in EFE for generating a full-focus image. As shown in Fig.7, we extract the features from $\bm{F}_{edge,i}$ using ${1\times 1}$ convolution to make the dimension consistent with $\bm{F}_{i}$. Let $\bm{f}_{edge,i}(x,y)$ be the feature vector composed of elements from different channels of $\bm{F}_{edge,i}$ at location $(x,y)$. The result is represented as $\bm{F}_{i}$ after $\bm{I}_{i}$ is processed by convolutional operation, BN and ReLU activate function. We measure the similarity between $\bm{f}_{edge,i}(x,y)$ and all feature vectors $\bm{f}_{i}(x\pm\delta,y\pm\delta)$ by inner product, where $\bm{f}_{i}(x\pm\delta,y\pm\delta)$ denotes the feature vector within $\left[{x\pm\delta,y\pm\delta}\right]$. Then, the weight is obtained after softmax activated:

where $(x^{\prime},y^{\prime})\in\left[{x\pm\delta,y\pm\delta}\right]$. The weight is used to scale all feature vectors $\bm{f}_{i}(x\pm\delta,y\pm\delta)$:

When $(x,y)$ in ${\hat{\bm{f}}}_{i}(x,y)$ traverses all pixels of the entire image, the features for generating a full-focus image are generated. Let

Since $\bm{M}_{h}$ can represent the difficulty of focused pixel detection, we concatenate it with $\hat{\bm{F}}$ and then feed it to the decoder to generate full-focus image $\bm{F}_{g}$:

where $\bm{E}_{d}$ denotes a full-focus image decoder consisting of a residual block and a convolutional layer.

With $\bm{M}_{h}$ and full-focus image $\bm{F}_{g}$, a hard pixels guided fusion result construction is proposed. If the pixel does not belong to the hard pixels, the fused result is constructed directly by selecting the corresponding pixel from source images. Otherwise, the corresponding pixel of $\bm{F}_{g}$ is selected. To this end, the Eq.(19) is developed to update the decision map for fusion result construction:

where ${\tilde{\bm{M}}}_{i}(x,y)=1$ indicates that the pixel located at $(x,y)$ is the focused pixel in the source image $\bm{I}_{i}$. The final fused result guided by ${\bm{M}}_{h}$ can be expressed as:

The loss functions of the method in this paper are divided into two parts: the hard pixels detection loss and the full-focus image generation loss:

Hard pixels detection loss: In HPD, cross entropy loss formulated in Eq.(21) is used to enable the network to realize the detection of pixel focus property:

where ${p_{i}}(x,y)$ denotes the focused probability at position $(x,y)$ for the $i$th source image, and ${g_{i}}(x,y)$ is its corresponding label.

Full-focus image generation loss: To obtain a clear full-focus image, besides the global reconstruction loss, the local reconstruction loss is also used in the hard pixel region:

where $\bm{G}_{f}$ denotes the label of generated full-focus image, $\lambda$ is a hyperparameter that balances the effects of global and local loss.

In training of GRFusion, we construct training set by artificial synthesis. Similar to DRPL[35], 200 all-in-focus images selected from ImageNet Large Scale Visual Recognition Challenge 2012 (ILSVRC2012)[36] are used to synthesize the training samples. To achieve the data augmentation, each raw image is randomly cropped to 9 sub-images whose sizes are ${256\times 256}$. Let a ${256\times 256}$ sub-image be $\bm{I}$, a pair of multifocus images is synthesized by:

where $\bm{M}$ denotes a binary mask randomly generated by the “findContours” function in OpenCV, and ${\bm{\tilde{}}{G}}(\cdot)$ is Gaussian filter with five different blurred versions. In the testing, the datasets used in our experiments are Lytro[37], MFI-WHU[38] and MFFW[39]. The details of the testing datasets are shown in Table I and the source images used to show the fused results are displayed in Fig.8:

The training process includes two stages: the training of HPD and the training of FFIG-MF. FFIG-MF is trained after the training of HPD. In training of HPD, the total epoch is set to 600 with batch size of 24. In training of FFIG-MF, the total epoch is also set to 600 and the batch size is set to 4. The loss functions in Eq.(21) and Eq.(22) are minimized with the Adam Optimizer [40]. The learning rate is initialized to 0.0001 and the exponential decay rate is set to 0.9. The proposed method is implemented on Pytorch framework and the experiments are conducted on a desktop with a single GeForce RTX 3090.

To demonstrate the effectiveness of our GRFusion, both qualitative and quantitative quality assessment are performed. In this process, we compare GRFusion with 9 state-of-the-art methods. These methods can be divided into two categories, i.e., feature reconstruction-based methods including MFF-GAN[38], ZMFF[22], MUFusion[41] and SwinFusion[42], pixel recombination-based methods including DRPL[35], GACN[43], SESF[28], SMFuse[30] and MSFIN[31]. To quantitatively evaluate the fusion results, five commonly used metrics including mutual information(${Q_{MI}}$[44]), edge based similarity metric(${Q_{AB/F}}$[45]), chen-blum metric(${Q_{CB}}$[46]), nonlinear correlation information entropy(${Q_{NCIE}}$[47]) and structural similarity (${Q_{SSIM}}$[48]) are used in our experiments. In these metrics, ${Q_{MI}}$ evaluates how much information of the source images is included in the fusion result. ${Q_{AB/F}}$ measures the degree of edges of source images transferred into the fused result. ${Q_{CB}}$ assesses the amount of information transferred from source images into fused result. ${Q_{NCIE}}$ evaluates the fused image from preservation of detail, structure, color and brightness. ${Q_{SSIM}}$ measures the structural similarity between the fusion result and the source images. The larger the values of these evaluation indicators, the better the quality of the fusion results.

The first experiment is performed on fusion of two source images. In this process, the image pairs from “Lytro” and “MFI-WHU” are used as the test samples, and the fusion results of the source image pairs shown in Fig.8 are displayed in Fig.9 and Fig.10. As observed, GRFusion performs observable advantages when compared to the other 9 methods. First of all, GRFusion is deigned by combining feature reconstruction-based method and focused pixel recombination-based method. Therefore, it can maximize the retention of focused information in the source image, avoiding the loss of detailed information and the introduction of artifacts. Meanwhile, thanks to FRC-G-HP, the degradation of fusion results caused by focus detection errors can be effectively alleviated. Consequently, the developed method can produce the results with better visual quality as shown in Fig.9 and Fig.10.

Specifically, from the boxed area we can see that the residual information in the results generated by the proposed method is less than that of the compared methods. This indicates that GRFusion can transfer more focused information of the source images into the fusion results. Moreover, we can also find that MFF-GAN, MUFusion and SwinFusion methods suffer from chromatic aberrations, and ZMFF tends to loss the information in focused regions. The results obtained by the focused pixel recombination-based methods, i.e., DRPL, GACN, SESF, SMFuse and MSFIN are prone to blurring at the boundary of the focus region. This can be seen from the residual information in the boxed areas. To evaluate the fused results of each method more objectively, we further list quantitative comparison results in Table II. As we can see, the results generated by the proposed method are optimal in most matrices. It further demonstrates the effectiveness of the proposed method.

Unlike most of the existing methods, GRFusion is able to achieve the parallel fusion of multiple images and shows superior fusion performance. To validate this argument, the sequence sample set in Lytro and MFFW with more than two multifocus source images are tested. Specifically, there are 3, 4 and 6 source images in ‘Lytro-triple series-03”, “MFFW3-03”, “MFFW3-06” and “MFFW3-02”. The fusion results generated by different methods are shown in Fig. 11. From the boxed areas shown in the first and the second columns, it can be seen that GRFusion can not only achieve fusion processing of three source images, but also more effectively preserve the focus information, resulting in a fusion image with higher visual quality. From the remaining results illustrated in Fig. 11, we can draw similar conclusions. This is because the method proposed in this paper can perform parallel fusion processing on multiple source images, avoiding the introduction of multiple feature extraction used in alternating fusion. Therefore, the proposed method can effectively alleviate the information loss caused by insufficient feature extraction. To objectively evaluate the quality of fusion results, the evaluation results of five metrics on the fusion images shown in Fig. 11 are displayed in Fig. 12. The objective evaluation results further prove that the proposed method also has advantages when fusing more than two source images.

In our method, there is one critical hyperparameter, i.e., $\lambda$, in loss function. To search an appropriate value for $\lambda$, we investigate the impact of $\lambda$ when it changes within $[0,1]$. As shown in Fig.17, when $\lambda=0$ and $\lambda=0.01$, the fusion result suffers from slight chromatic, and when $\lambda=0.1$, $\lambda=0.5$, $\lambda=1$, there is residual information presenting in the boxed areas. This indicates that not all information of the focused regions has been transferred into the fusion result. When $\lambda=0.05$, the fusion result shows higher quality since there is less residual information remaining in the boxed region. From the visual quality of the fusion results, we can conclude that $\lambda=0.05$ is the best choice. The quantitative evaluation of fusion results when $\lambda$ takes different values is shown in Fig.16, which further demonstrates superiority of $\lambda=0.05$ over other settings.