Towards High-quality HDR Deghosting with Conditional Diffusion Models

We provide a brief review of “variance-preserving” diffusion models from [28, 29], which gradually transform a Gaussian noise distribution $x_{T}$ into a complex data distribution $x_{0}$ using a latent variable model consisting of two processes: the diffusion process and the reverse process.

Given a dataset of input-output pairs, each pair consists of three LDR images $y$ and an HDR image $x$ (see Fig. 2). We aim to learn a parametric approximation to $p(x|y)$ through a stochastic iterative refinement process (Sec. III-A) that maps three LDR images $y$ to a target HDR image $x\in{\mathbb{R}^{d}}$. We propose a simple but effective scheme, which learns a conditional reverse process without modifying the diffusion process Eq. (1). As shown in Fig. 2, we integrate the feature condition generator branch $g_{\theta}$ into the noise predictor $f_{\theta}$, which guides the recovery of specific HDR images by adding LDR information at each reverse time step Eq. (4). Letting $g_{\theta}$ denotes feature condition generator, the new objective becomes: $\mathcal{L}_{noise}(\theta)=$

where $g_{\theta}$ does not require an extra loss because the gradient from the loss flows through $f_{\theta}$ into $g_{\theta}$. We include a pseudocode for the modified training procedure in Alg. 1.

To encourage the model to fully utilize the learned patch statistics and focus on local contextual information, we carefully design a patch-based sampling method called Sliding-window Noise Estimation. The typical patch-based approach infers the decomposed image patches and then optimally merges the results, ignoring the intermediate sampling step, and may contain artifacts from independently restored results.

Unlike overlapping the final reconstruction patch after sampling in DDPM, we use the sliding window method (see Fig. 4) to estimate the smoothed whole-image noise in each reverse step. Specifically, we first decompose the input $x_{t}{\in}\mathbb{R}^{H\times W\times C}$ of arbitrary size by a grid-like arrangement, where each grid cell contains $r\times r$ pixels. Then, we create a sliding window $P_{d}$ moving over this grid with the step of each grid cell, where the window size is $p{\times}p$ pixels $(p{>}r)$ and $d$ represents the current number of slides. In practice (Alg. 2), $P_{d}$ is a binary mask matrix of the same dimensionality as $x_{t}$, and $Slide(\cdot)$ denotes extract patch from current location $d$. We define two matrices $\Theta{\in}\mathbb{R}^{H\times W\times C}$ and $M{\in}\mathbb{R}^{H\times W\times C}$ and all elements are 0, where $\Theta$ records the cumulative noise estimation that each pixel receives through the sliding window, and $M$ records the number of times it has received an estimation. Next, we can calculate the mean noise estimate $\tilde{\Theta}$ of the whole image after $P_{d}$ passes through the entire image and use the smoothed noise estimation for sampling in each reverse time step. Thanks to these operations, our method can effectively mitigate damage to the learned local posterior, ensuring higher fidelity between locally adjacent blocks and avoiding semantic confusion problems in HDR images.

In HDR imaging tasks, the color information of the image is equally important to the perceptual quality of the generated content. Therefore, we propose a novel yet simple approach to providing image space constraints for diffusion models, avoiding the color distortion caused by the luminance difference of LDR images. Since the diffusion models aim to fit the probability distribution of noise added during the diffusion process in training, rather than the pixel-level image distribution, traditional methods [30] require T-step iterative sampling to obtain the final sampled image result. However, this approach is inefficient and computationally expensive. Thus, we attempt to transform the predicted probability distribution into an image distribution using the input noise image $x_{t}$ and constrain it in image space using $\mathcal{L}_{1}$ or $\mathcal{L}_{2}$ loss functions. This is mathematically feasible, as we have derived the formula by reverse engineering Eq. (2):

where $x_{0,t}$ represents the converted image result at $t$ step. Thus, the image space loss function $\mathcal{L}_{image}$ can be written as:

It is worth noting that $x_{t}$ is the input of the network, and $\bar{\alpha}_{t}$ is a hyperparameter pre-set in the network. When calculating gradients, it can be treated as a constant, so it does not affect the propagation of gradient flow. The final loss $\mathcal{L}_{our}$ can be formulated as:

Considering the typical diffusion model inference requires relatively large steps (e.g.., 1000) to achieve optimal image synthesis quality, [49] accelerated this process by “skipping” steps with appropriate update rules. Since our method only revised the estimation of $\epsilon_{\theta}$, it does not affect the ability to perform implicit sampling, where the sampling method is a non-Markovian forward process:

and implicit sampling using a noise predictor can be executed by:

It is worth noting that this sampling method is independent of the training process. When incorporating Eq. (2) into $\tilde{\mu}_{t}(x_{t},x_{0})$, the sampling process will be consistent with the original DPM by setting $\sigma_{t}^{2}=\tilde{\beta}_{t}$. Therefore, we can choose a sub-sequence $\{1,\cdots,T^{\prime}\}$ of $\{1,\cdots,T\}$ used in training without modifying the training settings, which allows us to generate high-quality HDR images in fewer iterative steps (Alg. 2 line 12).

Training models on the tonemapped domain is often essential for existing deep learning-based methods [19, 20, 23, 24, 26, 27, 60, 59, 41]. Since traditional DDPM-based methods only learn a probability distribution of the added noise in each step, we devised a ddpm-specific approach that applies the $\mu$-law function to transform the HDR image before adding noise, in order to indirectly achieve a similar effect.

Training. During the training phase, the input LDR-HDR image pairs $(X,Y)$ in the training set are used to train our model with the total diffusion step $T$ (see Algorithm. 1). Since the HDR images are usually displayed after tone mapping, training the network on the tone-mapped images is more effective than training directly in the HDR domain [19]. Given an HDR image $x$ in the HDR domain, we maps HDR pixel values to a more uniform distribution using $\mu~{}law$:

where $\mathcal{T}(\cdot)$ denotes the tone mapping operate, $\mu$ is a parameter defining the amount of compression. In this work, we always keep $x$ in the range [0, 1], $\mu=5000$, and train the network with $x_{0}$ in the tone-mapping case (Alg. 1 line 5). Before feeding the LDR images to the Attention Network, we first map the input LDR images $Y_{i}$ to the HDR domain relying on gamma correction to generate a corresponding set of $H_{i}$. As suggested in [19], we concatenate images $Y_{i}$ and $H_{i}$ along the channel dimension to obtain the 6-channel tensors $y_{i}=[Y_{i},H_{i}],~{}i=1,2,3$ as the input of the network.

Inference. A T-step inference process takes three LDR images (the 6-channel tensors) as input, as illustrated in Algorithm 2. Unlike the training phase, we encode implicitly aligned LDR information $\tilde{x}$ by the Attention Network $g_{\theta}$ only once, which speeds up the inference. We random sample a latent variable $x_{T}\in\mathcal{N}(0,\mathbf{I})$ and convert it to a particular tone-mapped HDR image by a stochastic iterative refinement process using Sliding-window Noise Estimation. Since the training phase is based on a tone-mapping case, we utilize $R(\cdot)$ to restore the final output, which can be expressed as:

where $R(\cdot)$ is the reverse process of $\mathcal{T}(\cdot)$. We will discuss the motivation for optimizing the network in the tonemapped domain in Sec. V-A.

To better illustrate the gradual effects of our method, we present intermediate results from the denoising process. During training, we adopt $T=1000$, resulting in 1000 intermediate results with adjacent ones having visually similar contents. We perform inference via the proposed SWNE method based on DDIM and present the intermediate results. As shown in Fig. 5 (a), we demonstrate the intermediate results at each step inferred by SWNE when $T=5$, which reveals how our method progressively denoises generating a high-quality HDR image from an input noise. Considering that our method predicts $x_{0}$ directly from the estimated noise at each denoising step and optimize it with an image space loss, we additionally present the results of directly predicting $x_{0}$ from each step’s intermediate result, as shown in Fig. 5 (b). It can be observed that the quality of the predicted $x_{0}$ improves gradually as $t$ approaches 0, with less noise and finer details.

Furthermore, through the intermediate results, we conduct additional results to verify the effectiveness of DDPM for ghosting reduction in HDR imaging. We present the results of directly predicting $x_{0}$ from each intermediate step when $T=5$ using DDIM and SWNE. As depicted in Fig. 6, the earlier results ($x_{3},x_{4}$) have noticeable noise and ghosting artifacts. For example, ghosting artifacts can be observed in the arm region in Fig. 6 (a) and window sill in Fig. 6 (b), similar to AHDR [20]. With the denoising proceeds (from $x_{4}$ to $x_{0}$), while noise is reduced and details enhanced, the ghosting artifacts are progressively alleviated. We consider that this phenomenon can be attributed to the DDPM-based model having favorable content generation capabilities, thus it can remove ghosting and generate more perceptually pleasing HDR reconstructions compared to regression-based models.

We evaluate the proposed method and compare it with state-of-the-art methods. Specifically, we compare the proposed method with a patch-based method [57], the flow-based approach with DNN merger [19], six CNN-based methods: DeepHDR [58], AHDR [20], HDR-GAN [26], NHDRR [41], ADNet [59] and APNT [60], and two ViT-based methods [23, 24]. The same training dataset and setting are used for deep learning methods.

We perform ablation learning on different components to validate their effectiveness.

In addition, we use DDIM and set $T=5$ to present the intermediate results of directly predicting $x_{0}$ at each denoising step. As shown in Fig. 13, $x_{4}$ to $x_{0}$ denote the intermediate results at different steps, with reduced noise and increased details as denoising proceeds. Fig. 13 (a) shows results using $512\times 512$ patch-based SWNE inference, while Fig. 13 (b) uses full-image inference without SWNE. It can be observed that in the early phases of the denoising step (e.g.,$x_{4}$), the variants have very similar outputs, but as denoising continues, full-image inference utilizes erroneous context to reconstruct saturated regions, leading to unrealistic results inconsistent with human perception. While leveraging broader context can help reconstruct regions lacking reliable local information to some extent, the semantics of distant locations often differ from the local patch. In contrast, the SWNE approach (Fig. 13 (a)) encourages the model to fully utilize the learned patch statistics and focus on local contextual information, thus avoiding semantic confusion and producing more plausible HDR reconstructions.

We conducted a perceptual study with human subjects to further evaluate the performance of the proposed HDR imaging framework. To obtain pairwise preference ratings comparing different HDR imaging methods on the Kalantari [19] dataset, we presented participants with side-by-side $512\times 512$ crops of images generated by each method. Specifically, we decoupled the 15 imaging results from each model into $15\times 6=90$ patches of size 512$\times$512. Since HDR images are usually displayed after tone mapping, we visualized the images after applying MATLAB’s tonemap operation to convert them to the tonemapped images. The images were displayed on a 27-inch LED monitor with $2560\times 1440$ resolution. To ensure suitable participants, we employed the Snellen visual acuity test and a color blindness assessment. A total of 30 graduate student participants (18 females, 12 males) aged 18-30 years met the criteria. To improve rating consistency, we included a training step before the rating task to familiarize participants with the distortion types present in the test. There were 90 independent image groups, each containing crops from the same scene location generated by different methods. Participants were asked to select the image with the better quality from side-by-side crops of size $512\times 512$.

Results in Tab. VII show the average rater preference computed from $30\times 90=2700$ comparisons. Each value represents the percentage of times raters preferred the row over the column. As highlighted, these results indicate that our HDR imaging model outperformed the competing methods.

In previous work, [19] found that when a model produces HDR images in the linear HDR domain, the estimated images typically demonstrate discoloration, noise, and other artifacts. Therefore, they proposed training the model in a more uniform amplitude distribution domain to obtain better visual results. Given the estimated HDR image $I^{\hat{H}}$ and the ground truth HDR image $I^{H}$, this loss term is defined as:

where $\mathcal{T}(\cdot)$ denotes the $\mu$-law function, and $\mu=5000$. In our work, since traditional DDPM-based methods only learn a probability distribution of the added noise in each step, we cannot directly apply the loss function in Eq. 16. Therefore, we devised a ddpm-specific approach that applies the $\mu$-law function to transform the HDR image before adding noise, in order to indirectly achieve a similar effect. In our initial experiments training directly on the linear HDR domain, results exhibited discoloration, noise, and artifacts in dark regions. As seen in Fig. 15, while HDR domain results (Fig. 15 (a)) appear similar, tonemapped images (Fig. 15 (b)) reveal significant artifacts when trained on the linear HDR domain. The $\mu$-law function boosts the pixel values in the dark regions, and thus, optimization in the tonemapped domain gives more emphasis to these darker pixels in comparison with the optimization in the linear domain. It is worth noting that, as discussed in [19], tonemapping does not directly and irreversibly change the fundamental problem, unless the tonemapping operator chosen is non-differentiable or irreversible. For example, Gamma encoding, defined as ${(I^{H})}^{1/\gamma}$ with $\gamma>1$, is perhaps the simplest way of tonemapping in image processing. However, since it is not differentiable around zero, it is not suitable for use in our system.

As our method aims to explore the capabilities of DDPM for HDR imaging, we only adopted the common DDIM [49] algorithm to accelerate model inference in this work. Without any acceleration algorithms, the iterative sampling process of DDPM does require more time to generate the final HDR results compared to prior deep learning approaches. Thus, the standard DDPM paradigm presents challenges for practical deployment on lightweight devices. To address this issue, both academia and industry have initiated research into accelerating diffusion models along two main directions: (1) reducing the number of inference steps [49, 68, 69]; and (2) engineering optimizations [70, 71]. Since our method builds upon the same theoretical framework, these acceleration techniques can be seamlessly applied. With state-of-the-art strategies [71], inference of billion-parameter DDPM models can be achieved on mobile devices within 2 seconds without compromising image quality.

We benchmarked the inference time under different strategies to provide quantitative comparisons (Tab. VIII). Without any acceleration, our DDPM model takes 293 seconds to generate a 1500$\times$1000 HDR image using 1000 sampling steps. By adopting DDIM [49], we can achieve comparable visual results with only 25 steps, reducing the inference time to 7.53 seconds. Using DPM Solver [68] allows further acceleration, producing comparable quality with just 5 steps and 0.59 seconds. While slower than some existing techniques, this demonstrates the potential to significantly decrease running time while maintaining visual fidelity. Designing DDPM-based methods suitable for lightweight devices will be an important direction for our future work.