A Deep Decomposition Network
for Image Processing:
A Case Study of Visible and Infrared Image Fusion

The wavelet transform has been successfully applied to many image processing tasks. The most common wavelet transform technique for image fusion is the Discrete Wavelet Transform (DWT) [19, 20]. DWT is a signal processing tool that decomposes signals into high and low frequency signals. Generally speaking, low-frequency information contains the main characteristics of the signal, whereas high-frequency information conveys the details. In the field of image processing, 2-D DWT is usually used to decompose images. The wavelet decomposition of the image is given as follows:

where $\phi(\cdot)$ is a low-pass filter, and $\psi(\cdot)$ is a high-pass filter. The input signal $M_{(}x,y)$ is an image combining the signals of all bands. Along the $x$ and $y$ directions, high-pass and low-pass filtering are performed respectively. As shown in Fig.1, the low-frequency image is a coarse approximation of the content of the three high-frequency images which contain the vertical detail, diagonal detail, and horizontal detail, respectively.

In addition, the Laplacian operator is a simple differential operator having the rotation invariance properties. The Laplacian transform of a two-dimensional image function is the isotropic second derivative, defined as:

For digital image processing the equation is approximated by its discrete form as:

The Laplacian operator can also be expressed in the form of a convolution template, using it as a filtering kernel:

where $G_{1}$ and $G_{2}$ are the template and the extended template of the discrete Laplacian operator. The second differential characteristic of this template is used to determine the position of the edge. The filters are often used in image edge detection and image sharpening, as shown in Fig.2.

It should be noted that the traditional edge filtering operator is just a high-frequency filter. While highlighting the edges, the filters also amplify the noise.

Li and Wu et al. proposed a method [10] to decompose images using low-rank representation [21] (LatLRR). In essence, their method can be described by the following optimization problem:

where $\mu$ is a hyper-parameter, $\|\cdot\|_{*}$ is the nuclear norm, and $\|\cdot\|_{1}$ is the $l_{1}$ norm. $X$ is observed data matrix. $Z$ is the low-rank coefficients matrix. $L$ is a projection matrix. $E$ is a sparse noisy matrix. The authors use this method to decompose the image into detail image $I_{d}$ and base image $I_{b}$. We can see from Fig .3 that $I_{d}$ is a high-frequency image, and $I_{b}$ is a low-frequency image.

As shown in Fig .3, the low-frequency image $I_{b}$ is further decomposed to obtain a cascade of high-frequency images $I_{d1}$, $I_{d2}$ and $I_{d3}$.

Similarly, this method decomposes the infrared image and the visible light image to obtain high-frequency images and low-frequency images. After fusing the information in the decomposed frequency, the fused image $I_{f}$ is obtained by means of reconstruction.

In 2017, Liu et al. proposed a neural network-based method [11]. The authors divide the image into many small patches. Then CNN is used to determine whether each small patch is blurry or clear. The network builds a decision activation map to indicate which pixels of the original image are clear and which are focused. The authors showed that a well-trained network can accomplish multi-focus fusion tasks very well, but their method is not suitable for other image fusion applications.

In order to enable the network to fuse visible light images and infrared images, Li and Wu proposed a deep neural network (DenseFuse) [13] based on an auto-encoder. First they train a powerful encoder and decoder to extract the features of the original image, and then reconstruct it without losing information. The infrared image and the visible light image are encoded. The two sets of coding features are then fused. Finally, the fused features are input into the decoder to obtain the fused image. These methods use the encoder to decompose the image into several latent features. Then these features are fused and reconstructed to obtain the fused image.

In the past few years, Generative Adversarial Networks (GANs) have also been applied to image fusion. The pioneering work based on this approach is FusionGan [22]. The generator inputs infrared and visible light images and outputs a fused image. In order to improve the quality of the generated image, the authors designed an application-specific loss function.

These deep learning-based methods have the ability to map images into a high-dimensional feature space to process and fuse features. CNN can effectively extract the different dimensions images features. However, the CNN-based image fusion methods needs to a well-designed network structure, and can only be applied to specific image fusion tasks. In view of the evident effectiveness of image decomposition and neural network based feature extraction, we propose a novel method which combines multi-layer image decomposition and feature extraction using a unified neural network framework for infrared and visible light image fusion.

Our goal is to construct and train a deep neural network so that it decomposes the source image into several high-frequency signals and one low-frequency signal for subsequent fusion operations. The structure of the network is shown in Fig. 4, and the network settings are detailed in Table I.

In Fig. 4 and Table I, $I_{ori}$ is the original input image, and $I_{re}$ is the reconstructed image. The backbone of the network is constituted by four feature extraction convolutional blocks ($Cin,C1,C2,C3$). The low-frequency feature extraction part is the ${}^{\prime}semantic^{\prime}$ block in the figure. The ${}^{\prime}semantic^{\prime}$ block shown in Fig. 6 includes two down-sampling convolutional layers ($S0,S1$) with a stride of 2 and a common convolutional layer ($S3$) which generates a low-resolution semantic image $I_{s}$. Then $I_{s}$ is up-sampled to the same size as $I_{ori}$ to obtain the low-frequency image $I_{ups}$.

We copy the features of different depth ($C1,C2,C3$) and and then reshuffle their channels with convolutional layers ($R1,R2,R3$). Subsequently, we input them into the ${}^{\prime}detail^{\prime}$ branch of the shared weights to obtain three high-frequency images $I_{g1}$, $I_{g2}$ and $I_{g3}$. The detail branch here is shown in Fig. 5 and Table I, which includes three convolutions (D0, D1, D2). The number of channels is reduced to 1 to obtain a high-frequency image. The reason for adding reshuffle layers ($R1,R2,R3$) here is that the detail block is weight-sharing. The feature maps that extract high-frequency information should follow the same channel distribution. So we add a $1\times 1$ convolutional layer that does not share weights, and reshuffle and sort the feature channels so that the features can adapt to the weight-shared details block.

Finally, the three high-frequency images ($I_{g1}$, $I_{g2}$, $I_{g3}$) and one low-frequency image ($I_{ups}$) are added pixel by pixel to obtain the final reconstructed image $I_{re}$. Note that the high-frequency image and the low-frequency image are complementary. In fact, when the network learns to generate images, the high-frequency image can be viewed as the residual data of the low-frequency image. Bearing this in mind, we reflect this reality in terms of a residual branch (${}^{\prime}C-res^{\prime}$ block). We skip-connect the result of $cin$ directly to the front of the $semantic$ block, adding it to the result of $c3$, and input it to the following layers. In this way, $C1,C2$ and $C3$ process the residual data between the source image and the semantic image intuitively. In order to make the skip-connected data more closely match the deep features of $C3$, we performed three convolutions in ${}^{\prime}C-res^{\prime}$ block to enhance the semantics of the skip-connected features.

As shown in the activation function in Table I, to consider the properties of low-frequency and high-frequency images jointly, we choose LeakyRelu [23] as the activation function of the convolution layers in the backbone network and the high frequency components ($Cin,C1,C2,C3,detail$), and the Relu function is used as the activation function of the convolution layers of the residual branch (${}^{\prime}C-res^{\prime}$) and the $semantic$ block ($S0,S1,S2$). The output of Relu has a certain degree of sparseness, which allows our low-frequency features to filter out irrelevant information and retain more blurred but semantic information. As a final consideration, in order to constrain the pixel value of the resulting image to a controllable range, we use the $Tanh$ activation function at the last layer of the $detail$ and $semantic$ blocks.

In general, the convolution block performs a convolution operation ($cin$) to obtain a set of feature maps containing various features. After the three identical convolution operations($c1,c2,c3$), three sets of shallow features are extracted. Applying two downsampling ($semantic$) operations, a deep feature is obtained. We argue that shallow features contain more low-level information such as texture and detailed features. We reshuffle the channels and feed these three sets of shallow features into the high-frequency branch $(detail$) to obtain three high-frequency images. As deep features, in contrast, contain more semantic and global information, we convolve and upsample them to obtain our low-frequency images. We use the residual branch ($C-res$) to explicitly combine the high-frequency low-frequency features. Lastly, we add these feature images pixel by pixel to get a reconstructed image.

In the training phase, the loss function (${Loss}_{total}$) of our network consists of three components. They are the gradient loss (${L}_{detail}$) of the high-frequency image, the distribution loss (${L}_{semantic}$) of the low-frequency image, and the content reconstruction loss (${L}_{reconstruction}$) of the reconstructed image. The formula of the loss function is defined as follows:

$\alpha$ and $\beta$ are hyper-parameters balancing the three losses.

As shown in Fig. 7, ${L}_{detail}$ calculates the mean square error loss between the high-frequency feature map ($I_{g1}$, $I_{g2}$, $I_{g3}$) and the gradient the input image and accumulates them. The loss, $L_{detail}$, is formally defined as

where $I_{ori}$ is the input source image and $I_{g\text{-}i}$ is the $ith$ high-frequency image. The $MSE(X,Y)$ is the mean square error between $X$ and $Y$. The gradient image of the original image is obtained by using the Laplacian gradient operator $Gradient()$. The Laplacian operator performs a mathematical convolution operation according to Equ.4.

In Equ.6, ${L}_{semantic}$ is a data distribution loss. It combines the loss of the high-frequency image, and the loss of the reconstructed image. We define reconstruction loss and detail loss as strong loss and semantic loss as weak loss. The reconstruction loss constrains the final generated image, and the detail loss constrains the high frequency features of the image. In addition, the final feature results consists of high-frequency features and low-frequency features. The mean square error loss of the reconstruct loss and detail loss is a strong constraint loss at the pixel level. If the low-frequency semantic loss is also a strong loss, then it is the sum of three strong losses. Each loss has a pixel-level strong constraint on the image data, which will lead to conflicts and contradictions. Therefore, we use the mean square error function to constrain the loss of high frequency information and reconstruction. Then, the downsampled image is used as the beacon of low frequency image data distribution, and its data distribution is guided by the adversarial loss function .

where $I_{s}$ is the low-frequency semantic image generated by the network, $I_{down}$ is the low-frequency blurred image obtained by downsampling the source image twice, and $L_{adv}$ is the adversarial loss.

where $n\in\mathbb{N}_{N}$, and $N$ represents the number of images. Here we use the loss function defined in LSGAN [24].

In Equ. 6, $L_{reconstruction}$ is the image content reconstruction loss of the reconstructed image. It consists of two elements. One is the pixel-level reconstruction loss ${L}_{pix}$, and the other is the structural similarity loss ${L}_{ssim}$ as follows:

where $\gamma$ is a hyper-parameter that balances the two losses. $L_{pixel}$ and $L_{ssim}$ are calculated as follows:

As shown in the Fig. 7, the total loss function $L_{total}$ is given as follows:

$\lambda_{1},\lambda_{2},\lambda_{3}$ are the hyper-parameters used for balancing the three losses.

Once the system is trained, each test image is first decomposed, as shown in Fig. 9. The fusion strategy (”FS” in Fig.9) fuses the corresponding feature images. These are then reconstructed to obtain the final fused image.

In particular, in Fig.9, $I_{ir}$ and $I_{vi}$ represent infrared and visible light image respectively. The two images are fed into the decomposition network to obtain two sets of feature images. One group of feature images comes from the visible light images, including three visible light high-frequency images ($I_{vi\text{-}g1}$, $I_{vi\text{-}g2}$, $I_{vi-g3}$) and one visible light low-frequency image ($I_{vi\text{-}ups}$). Another group of feature images comes from infrared images, comprising three infrared high-frequency images ($I_{ir\text{-}g1}$, $I_{ir\text{-}g2}$, $I_{ir\text{-}g3}$) and one infrared low frequency image ($I_{ir\text{-}ups}$). For the corresponding four groups of feature images, our fusion strategy offers a variety of options to obtain the final fused image $I_{f}$ as discussed in the following subsection.

We design a fusion strategy to get a fused image. As shown in the Fig.8, we first use the decomposition network to decompose the visible light image $I_{vi}$ and the infrared image $I_{ir}$ to obtain two sets of high and low frequency feature images. The corresponding high-frequency and low-frequency feature images (such as $I_{vi\text{-}g1}$ and $I_{ir\text{-}g1}$) are fused using different specific fusion strategies to obtain fused high-frequency feature images and low-frequency feature images ($I_{f\text{-}g1}$, $I_{f\text{-}g2}$, $I_{f\text{-}g3}$, $I_{f\text{-}ups}$). Finally, the fusion feature image is added pixel by pixel to obtain the fused image $I_{f}$, which is the same as reconstructing an image in the training phase.

We used two fusion strategies for both high-frequency image fusion, and low-frequency image fusion, namely, pixel-wise averaging (avg) and pixel wise max pooling (max) as shown in Fig.10.

The formulas for high frequency feature fusion $I_{f\text{-}g}$ and low frequency feature fusion $I_{f\text{-}ups}$ are given as follows:

where $i$ represents the high-frequency image index, and $\mathbb{N}$ denotes the number of pixels in the image. $I_{vi\text{-}gi}^{n}$ and $I_{ir\text{-}gi}^{n}$ represent a pixel in the corresponding three groups of high-frequency images, and $I_{vi\text{-}ups}^{n}$ and $I_{ir\text{-}ups}^{n}$ are the pixels in the low-frequency image. We calculate and fuse the corresponding pixels to produce the fused high frequency image $I_{f\text{-}gi}^{n}$ and low frequency image $I_{f\text{-}ups}^{n}$. Finally, the three fusion features are added to obtain the final fused image $I_{f}$ as follows:

The choice of hyper-parameters is guided by the requirement to maintain the contributions to the final loss values of the same order of magnitude. Accordingly, in formula 12, we set $\lambda_{1}$ = 0.1, $\lambda_{2}$ = 100, $\lambda_{3}$ = 10 by cross validation.

Our goal is to train a powerful decomposition network that can decompose images into high-frequency and low-frequency components. In this sense, our input images in the training phase are not limited to multi-modal image data. We can also use MS-COCO[25] and Imagenet[26] or other images to achieve this goal. In our experiment, we use MS-COCO as the training set to design our decomposition network. We select about 80,000 images as input images. These images are converted to gray scale images which are then resized to 256$\times$256.

For the infrared image and visible light image fusion task, we use the TNO dataset [27] and the RoadScene dataset [28]. For the RoadScene dataset, we convert the images to gray scale to keep the the visible light image channels consistent with infrared image. For the multi-focus image fusion task, we use the Lytro dataset [29]. The Lytro image are split according to the RGB channels to obtain three pairs of images. The fusion result is merged according to the RGB to obtain a fused image. For the medical image fusion task, we use the Harvard dataset[30]. For the multi-focus image fusion task, we use the dataset in [31].

We input batchsize of 64 images to the network every iteration. And, we select Adam[32] optimiser with an adaptive learning rate decay method[33] as the learning rate scheduler. We set the initial learning rate to 1e-3, the attenuation factor to 0.5, the stopping criterion to 5 iterations, and the minimum learning rate threshold to 1e-8. We set the maximum number of epoches to 1000.

In the test phase, because our network is fully convolutional, we input infrared images and visible light images without preprocessing operations.

The experiment is conducted on the two NVIDIA TITAN Xp GPUs and 128GB of CPU memory. We decompose 1000 images with 256$\times$256 resolution one by one and calculate the average calculation time. It takes about 2ms to decompose each image.

As shown in Fig 11(a), it is difficult for the network to learn to extract the semantic low-frequency content, if we do not impose constraints on the low-frequency image $I_{s}$. More crucially, low-frequency images tend to lose important semantic information.

As shown in Fig 11(b), without the ${L}_{semantic}$ adversarial loss, the high-frequency images will contain some information that should not appear, such as the brightness, contrast, colors distribution and other semantic information, which are not high-frequency information.

In order to allow the $semantic$ block to learn the useful low-frequency information, we guide it using weak supervision loss. As in Equ. 9, we regard the down-sampled image $I_{down}$ as an approximate solution of the low-frequency image, so that the low-frequency image $I_{s}$ generated by the network follows the distribution of low-frequency images.

As shown in 11(c and d), the high-frequency images do not learn wrong low-frequency information, and the low-frequency images contain almost all low-frequency semantic information.

Although the loss function of our high-frequency image is calculated with reference to a gradient map produced by the Laplacian operator, there is some discrepancy between them. In Fig 12, we present the high-frequency images decomposed by the proposed decomposition network and the Laplacian operator. It can be seen that the Laplacian gradient images only extract part of the high-frequency information content, and the image has a lot of noise.

The high-frequency image decomposed by our decomposition network not only retains almost all high-frequency information content, but also extracts the object contour and other detailed information. Thus our high-frequency images contain a certain degree of semantic information for recognition.

We select ten classic and state of the art fusion methods to compare with our proposed method, including Curvelet Transform (CVT)[34], dualtree complex wavelet transform (DTCWT)[35], Laplacian Pyramid (LP)[36], Ratio of Low-pass Pyramid (RP)[37], DenseFuse[13], the GAN-based fusion network (FusionGAN)[22], a general end-to-end fusion network(IFCNN)[14], ResNetFusion [38], NestFuse[16], HybridMSD [39], PMGI [40], FusionDN[28], U2Fusion[41], MDLatLRR[10], Guided Filtering Fusion(GFF)[42], MEF-GAN[43], Guided Filter focus region detection(GFDF)[44], Convolutional Sparse Representation(CSR)[45], Cross Bilateral Filter fusion method(CBF)[46], Discrete Cosine Harmonic Wavelet Transform(DCHWT)[47] and Multi-resolution Singular Value Decomposition (MSVD)[48]. We use the publicly available codes of these methods and the parameters recommended by their inventors to obtain the fused images.

As there are currently no generally agreed evaluation indicators to measure the quality of the fused image, we will use several objective measures for a comprehensive comparison, as well as a subjective evaluation.