Speckles-Training-Based Denoising Convolutional Neural Network Ghost Imaging

From the view of mathematics, the process of GI is an optical expansion of the object in a set of spatial basis functions: each pattern represents a basis function, and the bucket detection is finding the corresponding coefficient. The correlation calculation is to reproduce the object by combining the bases and the coefficients together. In short, GI is an optical tool to acquire the spatial spectra of an object in the representation of certain bases. The selection of bases, how well to measure the coefficients and image reconstruction algorithm determine the imaging quality.

A typical GI system projects a set of light patterns (denoted as $\{P_{1}(\mathbf{r}),P_{2}(\mathbf{r}),...P_{M}(\mathbf{r})\}$) onto an object, where $M$ is the total number of the patterns. The illumination light is modulated by the object, and then reach the bucket detector that reads out the corresponding intensities as $\{b_{1},b_{2},...,b_{M}\}$. A bucket detection can be formulated as

which is the $j$-th expansion coefficient on the $j$-th basic fucntion $P_{j}(\mathbf{r})$. The whole detection process can be expressed in matrix form:

or

The input of DnCNN is noisy image, which composed of original image and residual image. The output of DnCNN is residual image. The original image is eliminated in the process of training. The optimization target of DnCNN is not the mean-square error (MSE) between noisy image and original image, but the MSE between predicted residual image and real residual image. The loss function of DnCNN can be expressed as

where $\Re\left({}\right)$ denotes the residual mapping $\Re\left({{y_{i}}}\right)={v_{i}}$. The residual learning formulation is utilized to train a residual mapping. ${y_{i}}$ is noisy image, ${v_{i}}$ is residual image. Then, we have ${x_{i}}={y_{i}}-\Re\left({{y_{i}}}\right)$, ${x_{i}}$ is the original image, and $\Theta$ is the trainable parameters, respectively. According to the principle of residual network, when the residual is 0, identity mapping is established, which is easy to train and optimize. During testing, high quality target image can be obtained quickly by subtracting prediction residual image from noisy image. DnCNN has a good denoising effect on a specific Gaussian noise level.

Inspired by the denoising effect of DnCNN on Gaussian noise, we consider introducing DnCNN into GI, and design training method that conform to the characteristic of GI. In GI, speckles are utilized to illuminate target. Fixed speckles correspond to fixed noise distribution. Consequently, we propose a DnCNN-based GI denoising method using fixed speckles to train and illuminate target in this paper. A sequence of fixed speckles are utilized to illuminate the samplings in training set. Then, we input the noisy image into the networks, and the output of the networks is predicted residual image. In this way, a mapping between speckles and residual is formed. We train a denoising networks that the noise is corresponding to a sequence of fixed speckles. Then, the high-resolution target image is obtained by subtracting the predicted residual image from the noisy image. The purpose of this method is to obtain a better target image when using this sequence of fixed speckles to illuminate any target. The loss function of the proposed method can be expressed as

where $\Re\left({}\right)$ denotes the residual mapping between speckles and residual, ${x_{s}}$ is the original image and ${y_{s}}$ is the noisy image. After training, the sequence of speckles used in the training are utilized to illuminate target, and noisy image input the trained network to predict residual image. Finally, the improved target image is obtained by subtracting the residual image from the noisy image.

The image can be recovered from the correlation between the patterns and the bucket signal:

$\mathbf{P}^{T}\mathbf{P}$ plots the convariance map of the two columns of $\mathbf{P}$. If $\mathbf{P}^{T}\mathbf{P}=I$ (an identity matrix), $G^{(2)}(\mathbf{r})=\mathbf{O}(r)$. This condition is satisfied, when $\mathbf{P}$ is an orthogonal matrix (such as Hadamard). When $P_{j}(\mathbf{r})$ is a random pattern, how much $\mathbf{P}^{T}\mathbf{P}$ is close to $I$ depends on the linear independency of two different coloums of $\mathbf{P}$. Meanwhile, $M$ should be large enough to satisfy the statistic condition. In practics, $M$ can not be set a very large number, otherwise the imaging speed would be highly slowed down. On the other hand, an insufficient large $M$ will worsen the image reconstruction. Let’s factorize the correlation into two terms:

The second term on the right hand side is called residue in GI, and we can find that speckle is directly related to residual. $G^{(2)}$ is termed as a noisy image contaminated by the residual image. Fig. 1 shows the architecture of the proposed method.

The architecture of the DnCNN in our work is shown in Fig. 2.

The network has a total of 17 layers. The first layer of the network is composed of a convolutional layer and a ReLU layer, the 2nd to 16th layers are both composed of convolutional layer, ReLU layer and BN layer, and the last layer is a convolutional layer. The padding of all convolutional layer is 1. Because we use single-channel grayscale images for training and testing, the convolutional layer in 1st layer contains 64 filters with size 1*3*3. The 2nd to 16th layers contain 64 filters with size 64*3*3. To ensure the number of input and output channel is 1, the last convolutional layer contains 1 filter with size 64*3*3. Simultaneously, to make the network input and output have the same size, we did not use the Pooling layer in our network.

We use 390 images in [24] of size 64*64 for training, and set the patch size as 64*64. We train 500 epochs, 1000 epochs, 1500 epochs and 2000 epochs for the DnCNN models, respectively. Before training, we use a sequence of fixed random speckles to sampling every element in training set, and the target image with noise is obtained by GI method. In this way, we prepared two data sets. One is obtained by sampling the data in training set with 2048 speckles, which is defined as sampling rate = 0.5, and the other is obtained by sampling the data in training set with 4096 speckles, which is defined as sampling rate = 1. Then, we use these two data sets to train two DnCNN models. The equipment used in our experiment is a desktop computer, the CPU is Intel-i7-9750H-2.6GHz, and the graphics is NVIDIA-GeForce-GTX-1660Ti.

First, we use the remaining 10 images in [24] (defined as Set10 and shown in Fig. 3) to test the proposed method under two different sampling rates, and compare with BC and CS through PSNR/SSIM.

It can be seen from the data listed in Table I that the proposed method can achieve the best PSNR results at both sampling rates. On average, at sampling rate = 0.5, when epoch reaches 1000, the proposed method is 12.81 dB higher than BC algorithm and 0.74 dB higher than CS algorithm. At sampling rate = 1, when epoch reaches 2000, the proposed method is 15.27 dB higher than BC algorithm and 0.87 dB higher than CS algorithm (The CS algorithm used in this paper is orthogonal-matching-pursuit, OMP). Table I shows the PSNR value of each image in Set10.

In addition, we test the proposed method on the other two larger datasets contain 12 and 68 images under two sampling rates, defined as ExtSet12 and ExtSet68. We test each image in the two datasets and compare the average PSNR/SSIM. We listed the average values calculated from each element in the datasets in Table II. The testing results of the Set12 and Set68 show that the proposed method achieves the best performances in both PSNR and SSIM. Compared to BC and CS algorithm in ExtSet12, the proposed method has a PSNR gain of about 10.17 dB and 0.65 dB at sampling rate = 0.5 and a PSNR gain of about 12.98 dB and 0.67 dB at sampling rate = 1. Similarly, for ExtSet68, the proposed method has 12.06 dB and 0.68 dB improvement at sampling rate = 0.5 and 14.64 dB and 0.75 dB improvement at sampling rate = 1.

Then, we compare the proposed method with the conventional DnCNN on Set10, ExtSet12 and ExtSet68 at sampling rate = 1. The DnCNN in our comparison testing is trained by the 390 images that used in the training network of the proposed method. We consider three noise levels for DnCNN, i.e., $\sigma$ = 15, 25 and 50. Fig. 4 shows the trend chart of PSNR comparison results. We can find that the performance of the proposed method on each data set is better than DnCNN trained with the same training set. Table III shows the specific values. Compared with the PSNR of the proposed method, we can find that the best PSNR performance of DnCNN on Set10, ExtSet12 and ExtSet68 is 24.84 dB, 22.36 dB and 24.05 dB respectively, which is less than the proposed method 0.83 dB, 0.17 dB and 0.62 dB, respectively.

Noteworthy, the two subsections above (subsection B and C) show that the lower PSNR of the network input, the greater the improvement of the proposed method. The higher PSNR of the input, the limited room for the improvement.

Moreover, we build GI experimental platform to verify the proposed method. The schematic diagram is shown in Fig. 5.

A expanded laser beam is incident onto a SLM, which modulates the phase of the light field on each pixies. The SLM is covered by a mask with randomly distributed pinholes. Only the fields within the pinholes can have the phase modulation. The size of each pinhole is 0.2 mm. The number of the pinholes is 24, which are distributed within a circle of 5 mm diameter. Thus, all above devices construct a light source that has 24 sub-fields. By manipulating the phases of the sub-fileds, the interference between them generates a variety of light patterns on a distant plane. However, the number of the sub-fields is insufficient, and the generated patterns are not sufficiently linear independent, which will indue repetitive visual artifacts in the reconstruction image. The proposed method is able to remove such artifacts and retrieve a desirable image.

We train a DnCNN using the speckles sequence in the above experiment and test it on Set10. Similarly, we compare the proposed method with CS and DnCNN under two sampling rates. We carry out different $\sigma$ values in DnCNN and different epoch numbers in the proposed method to compare. The results are shown in Fig. 6.

Fig. 6 (a) shows the comparison results under different conditions. Fig. 6 (b) shows the origin image of the first picture in Set10. Fig. 6 (c) shows the result of CS method (PSNR = 16.25 dB). Fig. 6 (d) shows the result of DnCNN (PSNR = 22.83 dB). Fig. 6 (e) shows the result of the proposed method (PSNR = 27.22 dB). From the experimental results, we can see that the proposed method has better effect. Noteworthy, the performance of the proposed method in physical experiment is better than the simulation data based test. Detailed comparison results on Set10 are shown in Table IV.

To test the denoising ability of the proposed method on the corresponding speckles sequence, we use two sets of different speckles sequences to compare. The results are shown in Fig. 7.

The red curve is the testing result of the DnCNN trained by the speckles sequence corresponding to this curve, and the blue curve is the testing result of another speckles sequence on this network. Fig. 6 shows that no matter which sampling rate, the effect of the proposed will descend when the illumination speckles sequence does not match the training speckles sequence.

In deep learning method, data enhancement can generate new training samples to prevent over fitting. RandomHorizontalFlip and RandomRotation are used as data enhancement strategies in DnCNN. However, we find that using these two data enhancement strategies in the proposed method will affect the performance through experiments. The PSNR decrease to half of that before denoise (shown in Fig. 8). We infer that the noise in the image obtained by the fixed speckles sequence is related to the position, the data enhancement of spatial transformation such as RandomHorizontalFlip and RandomRotation will destroy the noise distribution in the image, resulting in the network can not learn the original noise distribution in the image, so the image quality after denoising will not increase but decrease. The comparison results are shown in Fig. 8.

We carry out several different sets of contrast experiments including enable RandomHorizontalFlip and RandomRotation (H+R), enable RandomHorizontalFlip (H), enable RandomRotation (R) and close RandomHorizontalFlip and RandomRotation (None). Fig. 8 (a) shows the comparison results at sampling rate = 1 between the proposed method and conventional DnCNN that enable RandomHorizontalFlip and RandomRotation. Fig. 8 (b) shows the results of four different collocations, we can find that for training DnCNN with the proposed method, the effect of closing data enhancement is the best. On the other hand, for conventional DnCNN, the effect of enabling RandomHorizontalFlip and RandomRotation is the best.