Multi-view Self-supervised Disentanglement for General Image Denoising

Given two corrupted views of the same image $x^{k}$, $y_{1}^{k}\triangleq\mathcal{T}_{1}(x^{k})$ and $y_{2}^{k}\triangleq\mathcal{T}_{2}(x^{k})$, the encoder $\mathit{G}_{\theta}^{\mathcal{X}}$ mainly perform the scene feature space encoding that can be formulated as:

$$z_{x}^{k,1}\triangleq\mathit{G}_{\theta}^{\mathcal{X}}(y_{1}^{k}),\ z_{x}^{k,2}\triangleq\mathit{G}_{\theta}^{\mathcal{X}}(y_{2}^{k}),$$

(5)

where $z_{x}^{k,1}$ and $z_{x}^{k,2}$ are the estimation of the scene feature corresponding to the inputs $y_{1}^{k}$ and $y_{2}^{k}$.

The process of clean image reconstruction is then completed by the $\mathit{D}_{\psi}^{\mathcal{X}}$:

$$\hat{x}_{1}^{k}\triangleq\mathit{D}_{\psi}^{\mathcal{X}}(z_{x}^{k,1}),\ \hat{x}_{2}^{k}\triangleq\mathit{D}_{\psi}^{\mathcal{X}}(z_{x}^{k,2}).$$

(6)

Similar to the process of estimating scene features, the estimation of distortion features by $\mathit{E}_{\rho}^{\mathcal{N}}$, followed by the reconstruction of noise with $\mathit{F}_{\phi}^{\mathcal{N}}$, can be described as follows:

$$\begin{split}u_{n}^{k,1}\triangleq\mathit{E}_{\rho}^{\mathcal{N}}(y_{1}^{k}),\ u_{n}^{k,2}\triangleq\mathit{E}_{\rho}^{\mathcal{N}}(y_{2}^{k}),\\ \hat{\eta}_{1}^{k}\triangleq\mathit{F}_{\phi}^{\mathcal{N}}(u_{n}^{k,1}),\ \hat{\eta}_{2}^{k}\triangleq\mathit{F}_{\phi}^{\mathcal{N}}(u_{n}^{k,2}).\end{split}$$

(7)

We adhere to the methodology introduced by N2N [19] to use noisy images as supervisory signals. The objective function of the aforementioned process can be simplified to:

$$\begin{split}\underset{\theta,\psi}{\operatorname{argmin}}\,\mathcal{L}^{\mathcal{X}}&\triangleq||\hat{x}_{1}^{k}-y_{2}^{k}||,\\ \underset{\rho,\phi}{\operatorname{argmin}}\,\mathcal{L}^{\mathcal{N}}&\triangleq||(y_{1}^{k}-\hat{\eta}_{1}^{k})-y_{2}^{k}||.\end{split}$$

(8)

The objective of $\hat{x}_{2}^{k}$ and $\hat{\eta}_{2}^{k}$ are the same as above, with only a subscript difference. It should be noted that, although our objective functions are similar to that of N2N, our goal is not simply to find and remove noise, but rather to capture the common features shared across multiple views.

For general latent codes $z_{x}^{k}$ to sufficiently represent scene information in the image space, it is natural to assume that these codes exhibit a certain degree of freedom, allowing them to intersect with the noise space. Consequently, there is no guarantee of complete isolation between $z_{x}^{k}$ and $u_{\eta}^{k}$. To meet the requirements of properties (1) and (3), we use an additional decoder $\mathit{R}_{\delta}^{\mathcal{Y}}$ to reconstruct a corrupted view over a cross-feature combination, e.g. $z_{x}^{k,1}$ from $y_{1}$ and $u_{n}^{k,2}$ from $x_{2}$, which can be represented as:

$$\begin{split}\hat{y}_{1}^{k}\triangleq\mathit{R}_{\delta}^{\mathcal{Y}}(z_{x}^{k,2},u_{n}^{k,1}),\ \hat{y}_{2}^{k}\triangleq\mathit{R}_{\delta}^{\mathcal{Y}}(z_{x}^{k,1},u_{n}^{k,2}).\end{split}$$

(9)

This realisation explicitly requires $z_{x}^{k,i}$ to represent the common part and $u_{n}^{k,j}$ to represent the unique part within the corrupted views. We then optimise $\{\theta,\rho,\delta\}$ from $\{\mathit{G}_{\theta}^{\mathcal{X}},\mathit{E}_{\rho}^{\mathcal{N}},\mathit{R}_{\delta}^{\mathcal{Y}}\}$ using the following objective:

$$\begin{split}\underset{\theta,\rho,\delta}{\operatorname{argmin}}\,\mathcal{L}^{\mathcal{C}}&\triangleq||\hat{y}_{1}^{k}-y_{1}^{k}||+||\hat{y}_{2}^{k}-y_{2}^{k}||.\end{split}$$

(10)

Generally, it is possible for there to be a trivial solution from $u_{n}^{k,i}$ to $y^{k}_{i}$ in Equation (9) such as, when $u_{n}^{k,1}$ is extracted from $y_{1}^{k}$ and used to reconstruct it as well. However, Equation (7) explicitly requires $u_{n}^{k,1}$ to rebuild the noise, which prevents the collapse of $u_{n}^{k,1}$ in expressing $y_{1}^{k}$.

The aforementioned latent constraint might appear to be restricted at first, but in fact, it enables us to capture a large number of degrees of freedom in latent space implementation. For instance, it is possible to have multiple scene features that correspond to a single scene. However, in such cases, the mapping from input to scene features becomes ambiguous. To tackle this issue, we propose the use of the Bernoulli Manifold Mixture (BMM) as a means of leveraging the shared structure within the scene latent.

Given the assumption of property (2), the acquired scene features $z_{x}^{k,1}$ and $z_{x}^{k,2}$ are expected to be identical and interchangeable with one another, as they both refer to the same scene feature. BMM establishes a new explicit connection between the scene features of multi-views, which can be expressed in the equation as:

$$\hat{z}_{x}^{k}\triangleq\text{Mix}_{p}(z_{x}^{k,1},\,z_{x}^{k,2}),$$

(11)

where the $\hat{z}_{x}^{k}$ is an estimation of the true underlying scene feature. Let $b_{f}$ define a sample instance drawn from a Bernoulli distribution with probability $p\in(0,1)$, the function $\text{Mix}_{p}(\cdot)$ described in Equation (11) denotes:

$$\text{Mix}_{p}(\bm{m},\bm{n})\triangleq b_{f}\odot\bm{m}+(1-b_{f})\odot\bm{n}.$$

(12)

By establishing this new connection (Equation 11), we can enhance the interchangeability between $z_{x}^{k,1}$ and $z_{x}^{k,2}$ by optimising the reconstruction performance on $\hat{z}_{x}^{k}$.

In MeD, we denote $\hat{z}_{x}$ = $\mathit{G}_{\theta}^{\mathcal{X}}(\hat{z}_{x}^{k})$, and the objective for implementing Equation (13) can be formulated as follows:

$$\begin{split}\underset{\theta,\rho,\psi}{\operatorname{argmin}}\,\mathcal{L}^{\mathcal{M}}&\triangleq\lambda||\hat{z}_{x}-\text{Mix}_{p}(y^{k}_{1},\,y^{k}_{2})||,\end{split}$$

(16)

where the $\lambda$ is the weight parameter. Here, the target is using a mixed version of $y^{k}_{1}$ and $y^{k}_{2}$. The choice of this is driven by the intuition that the hybrid version would better align with the aforementioned blended features.

Noted that, the nature of feature disentanglement requires no leak from input to output, however, the global residual connection of the original DnCNN [41] cannot satisfy. Thus, we incorporate the Swin-Transformer (Swin-T) [24] instead of the traditionally used DnCNN in our experiments. Nevertheless, as Swin-T is not originally designed for image restoration, we make some modifications to enforce local dependence across the image. Specifically, we add one Convolution Layer each before the patch embedding and after the patch unembedding of Swin-T, as inspired by SwinIR [21]. The resulting modified network backbone is denoted as Swin-Tx.

To ensure a fair comparison, we use the Swin-Tx backbone for all methods in our study, except for DBD. As DBD did not release the code, we follow the instructions presented in the paper and make our best effort to re-implement it. However, we observe that the two-view DBD could not converge efficiently, which is consistent with the findings in the paper. Therefore, we limit our evaluation to the four-view DBD, denoted as DBD₄. Furthermore, we replace the U-Net backbone originally used in the LIR method with Swin-Tx to maintain consistency in our evaluation. This results in an average improvement of approximately 1 dB in PSNR for Gaussian denoising.

In all experiments, all methods were trained using only DIV2K [4] and the same optimisation parameters, except for LIR and $\text{DBD}_{4}$ which used manually selected parameters obtained through experiments. For more training and evaluation details including the choice of parameters, please refer to the supplementary Section B. Code is available at: https://github.com/chqwer2/Multi-view-Self-supervised-Disentanglement-Denoising.

We first investigate the denoising generalisation of the methods using synthetic zero-mean additive white Gaussian noise (AWGN). The experiments are divided into two parts. The first segment employs fixed variance AWGN, whereas the second segment employs varied variance Gaussian for training in a separate manner. Table 1 summarises the quantitative results evaluated on CBSD68 Dataset [26] at variance levels of 15, 25, 50, and 75.

In the context of random variance, it has been observed that MeD exhibits superior performance across all four noise levels compared to other methods, including supervised methods. These findings imply that MeD can benefit more from varying training noise than other methods. More experiments and details on AWGN noise removal can be found in the supplementary Table 9.

In the previous subsection, we demonstrated the remarkable generalisation ability of our model in the case of Gaussian noise. Here, we aim to extend this investigation to other types of unseen noise and evaluate the denoising generalisation ability of our method. Specifically, we consider Poisson noise, Speckle noise, Local Variance Gaussian noise, and Salt-and-Pepper noise. For a more detailed synthetic process, please refer to the supplementary Section F.

First, we demonstrate qualitative comparisons of denoising unseen noise types using models trained only with Gaussian $\sigma=25$ in Figure 3. Next, in order to further verify the denoising generalisation ability of MeD, we employ its scene encoder and decoder as pre-trained models to be compared against other methods. It should be noted that the pre-training and fine-tuning methods employed in this study may differ, as shown in Table 2. The pre-training of all test models was conducted on a Gaussian sigma value of 25, followed by fine-tuning with a Gaussian sigma range of 5 to 50. Since the training schema of N2S, R2R, and DBD₄ differs from MeD, we do not include them in this section. However, evaluations of these methods on unseen noise are still presented in Section 4.4 under different settings.

Here we further investigate the generalisation ability of our method by introducing our general Noise Pool. The Noise Pool comprises the five aforementioned types of noise, each with a diverse range of noise levels. During training, we randomly sample from the noise pool to provide the model with noisy images. This novel approach simulates a realistic scenario where noise is unknown and can originate from various sources to some extent.

Specifically, we evaluated all methods using the random noise pool approach to train and test on combined or single noise types. The results are summarised in Table 3.

In our previous experiments, we demonstrated the exceptional denoising performance of our MeD approach on synthetic noises. However, real-world noise is often more complex and challenging than synthetic noise. In this subsection, we aim to evaluate the generalisation performance of our approach on real-world noise by testing it on the SIDD [1], CC [27] and PolyU [36] datasets. To assess the denoising performance on real-world noise, we use the same pre-trained models as in Section 4.4. The representative qualitative results on the SIDD dataset in the standard RGB colour space are presented in Figure 4.

Our approach achieves remarkable performance on real-world noise without even being trained on more expensive real-world data. For a more complete study, we also conduct experiments on model training with real-world data (more details please refer to the supplementary Table 11), showing superior performance and even better generalisation ability to data out of the training data distribution.

In Figure 4, the presence of noise persists even after applying denoising techniques, yet ours demonstrates the most authentic outcomes compared to others. For instance, while the noise particles remain prominent in the N2C results, they are absent in our results. Overall, the results indicate that the MeD approach is well-suited for real-world denoising tasks, providing a robust and reliable solution for improving image quality in challenging environments.

Although we only showcase two views for the experiments above, our method can be easily expanded to multiple views. To investigate the impact of the numbers of views, here we further conduct a study comparing 2, 3, and 4 views, in Table 5.

The results indicate that increasing the number of views consistently improves the performance across different noise types. For example, when dealing with Speckle noise, the 4-view model achieves a 0.26 dB higher PSNR than the 2-view model. However, it is worth noting that employing $n$ views requires $n!$ cross-computations within each view pair during training, resulting in a significant increase in computational cost (e.g. from 2-view to 4-view leads to a $10\times$ training time increase in our experiment).

Here we investigate the potential of the proposed MeD for other more general image restoration tasks, such as image super-resolution and inpainting. In this study, we generalise the previously defined degradation (noise) to a residual image between a clean image and a corrupted image. Moreover, we expand the definition from Noise Pool to a more general one – Corruption Pool that contains not only noise but also general corruption.

The computations described in this research were performed using the Baskerville Tier 2 HPC service¹¹1https://www.baskerville.ac.uk/. Baskerville was funded by the EPSRC and UKRI through the World Class Labs scheme (EP/T022221/1) and the Digital Research Infrastructure programme (EP/W032244/1) and is operated by Advanced Research Computing at the University of Birmingham.