MFDNet: Multi-Frequency Deflare Network for Efficient Nighttime Flare Removal

During the process of capturing photographs, various factors, including unfavorable weather conditions, optics-induced diffraction, and relative motion between the camera and object, can lead to the deterioration of image quality. This degradation results in the loss of important information in the captured image, necessitating the restoration of the image to its original quality. Typical image restoration tasks include but are not limited to image deblurring chen2023deep , image denoising zhang2023compressive , image dehazing chougule2023agd , rain removal ragini2023detformer , reflection removal 9381898 , shadow removal 9408597 , and more. Recently, some state-of-the-art image restoration methods zamir2022restormer ; zamir2021multi ; wang2022uformer ; chen2021hinet ; chen2021pre ; liang2021swinir ; Li2021OnET ; tu2022maxim ; 8485303 ; 9173764 ; luo2023hir have appeared to handle different image restoration tasks.

For the nighttime flare removal task, we propose a lightweight Multi-Frequency Deflare Network (MFDNet) based on the Laplacian Pyramid. As shown in Figure 2, our MFDNet consists of two primary modules: the Low-Frequency Flare Perception Module (LFFPM) and the Hierarchical Fusion Reconstruction Module (HFRM). Next, we first describe the overall pipeline of MFDNet, and then we detail the LFFPM in Section 3.2 and the HFRM in Section 3.3.

The Laplacian Pyramid (LP) burt1987laplacian is a frequency-band image decomposition technique derived from the Gaussian Pyramid (GP). The main idea of the LP method is to decompose an image linearly into high-frequency and low-frequency bands. And based on the LP, the image reconstruction process can be implemented with precision and reversibility. Specifically, the LP is obtained by calculating the difference between adjacent layers in the Gaussian Pyramid. According to burt1987laplacian ; liang2021high , the lumination information of the image is more related to the low-frequency band and the high-frequency component contains more detailed content information such as textures. Inspired by the above properties of LP, we decouple the image’s illumination and content information, remove the flare artifacts in the low-frequency part of the image, and subsequently fuse the low-frequency flare-free image with the detailed high-frequency information to restore the final flare-free image. The entire process is depicted in Figure 2.

Specifically, given a nighttime flare-corrupted image $I\in R^{H\times W\times 3}$, MFDNet first decomposes it into a Laplacian Pyramid, generating a set of different frequency parts $L=[l_{1},l_{2},\cdots,l_{n-1}]$ and the lowest frequency image $I_{n}$, where $H\times W$ denotes the spatial dimension, and $n$ is the number of LP’s decomposed levels. The components of $L$ have progressively reducing resolutions from $H\times W$ to $\frac{H}{2^{n-1}}\times\frac{W}{2^{n-1}}$, and $I_{n}$ has $\frac{H}{2^{n}}\times\frac{W}{2^{n}}$ pixels. After getting $I_{n}$, we input it into the Low-Frequency Flare Perception Module (LFFPM) for flare removal and get a low-frequency flare-free image ${\hat{I}}_{n}$. $I_{n}$ and ${\hat{I}}_{n}$ are then provided to the Hierarchical Fusion Reconstruction Module (HFRM), which fuses each layer of $L$ with ${\hat{I}}_{n}$ incrementally to create $\hat{L}=[\hat{l}_{1},\hat{l}_{2},\cdots,\hat{l}_{n-1}]$ for reconstruction. $L$ and $\hat{L}$ share a one-to-one mirror relationship, so the final flare-free image $\hat{I}$ can be reconstructed by ${\hat{I}}_{n}$ and $\hat{L}$. Because our method only needs to remove flares in the low-frequency components, it significantly reduces computational complexity.

The Low-Frequency Flare Perception Module (LFFPM) mainly includes the following parts, the Feature Extraction Transformer Block (FETB) and the Feature Refinement Transformer Block (FRTB) for extracting and refining features, and a U-shaped network with skip connections, composed of the Feature Encoder Convolution Block (FECB) and the Feature Decoder Convolution Block (FDCB). Transformers have been demonstrated to be more effective than CNNs in modeling long-range dependencies. As depicted in Figure 2, we use the Feature Extraction Transformer Block (FETB) in the Low-Frequency Flare Perception Module (LFFPM) to extract global features from the input image $I_{n}$. Moreover, at the end of LFFPM, we use the Feature Refinement Transformer Block (FRTB) to generate the enhanced features. After multiple FRTBs and FRTBs, we concatenate the output of each block to obtain longer-distance features. Inspired by wang2023ultra , FETB and FRTB use axis-based self-attention(ASA) to reduce computational complexity and gated feed-forward network (GFFN) to capture more critical features. Traditional self-attention’s computational complexity is quadratic with the input resolution. ASA computes self-attention sequentially on the height and width axes across the channel dimension, resulting in linear complexity. GFFN applies GELU and elementwise product to eliminate less relevant features in two parallel paths, then combines the relevant features through element-wise summation. As shown in Figure 3, the computation of FETB and FRTB are represented as:

where $\mathbf{F}$ is the input of FETB and FRTB. $\mathbf{F}^{\prime}$ and $\mathbf{\hat{F}}$ are the outputs of ASA and GFFN, respectively. $\mathrm{LN}$ represents the layer normalization ba2016layer .

In order to alleviate the limitation of Transformers in capturing local dependencies, the Low-Frequency Flare Perception Module (LFFPM) uses a convolution-based U-shape encoder-decoder structure consisting of Feature Encoder Convolution Block (FECB) and Feature Decoder Convolution Block (FDCB) to enhance detailed feature representation, as shown in the Figure 2. Additionally, to ensure that both the global and local features are adequately fused, the Low-Frequency Flare Perception Module (LFFPM) automatically learns the weighted summation of the global long-distance features extracted by FETB and the output of FDCB. Inspired by chen2022simple , FECB and FDCB are designed as simple and efficient nonlinear activation networks. As shown in Figure 4, FECB and FDCB mainly include the following parts: layer normalization ba2016layer , convolution, Gate, and Channel Attention hu2018squeeze . Gate divides the feature map into two parts in the channel dimension and multiplies them. The formula of Gate is:

where $\mathbf{X}$ and $\mathbf{Y}$ are feature maps.

The Channel Attention(CA) mechanism can capture global information efficiently. The formula of Channel Attention is:

where $\mathbf{X}$ denotes the feature map, $W$ is fully-connected layers, and $\mathrm{Pool}$ represents the global average pooling. $\ast$ is channel-wise product operation.

As shown in Figure 2, after obtaining the low-frequency flare-free image ${\hat{I}}_{n}$ through the Low-Frequency Flare Perception Module (LFFPM), the Hierarchical Fusion Reconstruction Module (HFRM) fuses ${\hat{I}}_{n}$ with the high-frequency parts $L=[l_{1},l_{2},\cdots,l_{n-1}]$ layer by layer to obtain $\hat{L}=[\hat{l}_{1},\hat{l}_{2},\cdots,\hat{l}_{n-1}]$ for reconstruction.

We concatenate the upsampled $I_{n}$ and ${\hat{I}}_{n}$ with $l_{n-1}$, and then input them into a simple network to learn a mask $M_{n-1}$ to guide the fusion process. The specific fusion process is as follows:

where $\otimes$ represents the pixel-wise multiplication. To refine the fused features further, we perform a dilated convolution operation on ${{l}^{\prime}}_{n-1}$, then use the Feature Aggregation Block (FAB) to re-weight the significance of features.

Through the above steps, the low-frequency flare-free image ${\hat{I}}_{n}$ achieves the feature fusion operation with one layer in the high-frequency parts $L=[l_{1},l_{2},\cdots,l_{n-1}]$. By analogy, we need to upsample $\mathbf{M}_{n-1}$ through linear interpolation and perform the same fusion operation with the next layer in the high-frequency parts $L=[l_{1},l_{2},\cdots,l_{n-1}]$. Behind the Feature Aggregation Block (FAB) in the final layer, we use the Spatial Pooling Pyramid (SPP) he2015spatial to facilitate remixing multi-context features. This way, we get $\hat{L}=[\hat{l}_{1},\hat{l}_{2},\cdots,\hat{l}_{n-1}]$ to reconstruct the final flare-free image.

As shown in Figure 5, the Feature Aggregation Block (FAB) structure is simple. Inspired by cun2020towards , we use a squeeze-and-excitation block hu2018squeeze in FAB to learn the weights of different channel features through some linear layers and pooling layers, which can automatically preserve essential features. Additionally, convolution is used to squeeze the features and match the original channels.

We train our model with different losses, including the mean square error (MSE) loss $L_{M}$, structural similarity loss $L_{S}$ wang2004image , and perceptual loss zhang2018unreasonable .

Given the final output image $I_{out}$ and ground truth image $I_{gt}$, the perceptual loss $L_{P}$ is defined as:

where $F$ is a pre-trained AlexNet krizhevsky2012imagenet feature extractor. We compute the $l_{2}$ distance between $F_{l}(I_{out})$ and $F_{l}(I_{gt})$ for layer $l$.

In summary, our total loss function can be expressed as:

where we empirically set $\lambda_{m}=1$, $\lambda_{s}=0.3$ and $\lambda_{p}=0.7$ respectively.

We compare the performance of MFDNet with several state-of-the-art methods for nighttime flare removal in Flare7K real-world and synthetic datasets. These methods include a nighttime dehazing method zhang2020nighttime , a nighttime visual enhancement method sharma2021nighttime , two U-Net-based flare removal methods wu2021train ; dai2022flare7k , and HINet chen2021hinet , MPRNet zamir2021multi , Restormer zamir2022restormer , Uformer wang2022uformer .

We conduct ablation studies to evaluate the contributions of the different components of our proposed method. Specifically, we measure the contributions of the Feature Extraction Transformer Block (FETB) and the Feature Refinement Transformer Block (FRTB), the Feature Encoder Convolution Block (FECB) and the Feature Decoder Convolution Block (FDCB), and the Feature Aggregation Block (FAB). The results from Table 3 and Figure 10 show that these components of our MFDNet are all effective for flare removal.

The Transformer block for global feature extraction and refinement can improve the deflare performance, because global information is crucial in accurately identifying the flare. Additionally, the convolution-based encoder-decoder structure provides substantial capabilities to remove flare artifacts and restore more image details. As can be seen from Table 3 and Figure 10, after removing the FECB and FEDB, the deflare effect of the model drops significantly, indicating that in the task of flare removal, it is very important to capture the detailed information of the input flare-corrupted image. Moreover, the FAB helps to integrate information from different frequency bands to reconstruct the final flare-free image.

However, the proposed method exhibits some limitations that need to be addressed. Firstly, if the light source is diminutive and faint, accompanied by glare flare, the contrast between the light source and adjacent flare areas may not be distinctly pronounced. Under such circumstances, our approach might not fully restore the light source. To address this issue, we might manually determine an optimal luminance threshold for saturation during restoration as shown in Figure 11. In future work, we will endeavor to resolve this challenge by employing adaptive techniques to recover the light source. Secondly, our method may fail to completely remove flare shimmers with very high brightness and extensive shapes, probably because they have obvious texture features in the high-frequency band. We intend to resolve these shortcomings in future studies.