Bare Demo of IEEEtran.cls
for IEEE Conferences

Quanta Image Sensor (QIS) is envisioned as a candidate solution to to resolve the limited full well capacity issue resulting from shrinking pixel size. Since its introduction in 2005 [fossum200611] a significant number of research have demonstrated the feasibility of QIS and have proposed a few protoypes. In [Ma:17] Ma et al. demonstrated a QIS prototype with read noise below $0.25e^{-}$ rms at room temperature and frame rate beyond 1000 frames per second. This high level of photon sensitivity, low read noise, and high speed have made QIS an ideal sensor for low-light imaging applications.

The subject of this paper is to consider the high dynamic range imaging problem of QIS. One of the major shortcomings of a digital camera is the lack of dynamic range. Like traditional CMOS image sensors (CIS), the limited full well capacity of QIS upper bounds the dynamic range it can support. However, owing to its high speed, QIS is able to leverage photons acquired by the multiple frames to extend the dynamic range.

In this paper, we present a high-dynamic range image reconstruction theory and algorithm for QIS. We offer two contributions. First, we theoretically derive the dynamic range which can be offered by QIS compared to CIS. This provides a foundation of how much dynamic range we can expect from QIS. Second, we propose a novel high-dynamic range reconstruction algorithm for QIS. Our reconstruction algorithm is customized for QIS as it handles the truncated Poisson statistics uniquely present in QIS. We compare the performance of our algorithm with other state-of-the-art high-dynamic range reconstruction methods.

The dynamic range is defined as the ratio of the illumination that just saturates the sensor to the read noise. As we do not consider any read noise in the sensors, the quantity dynamic range does not exist. So, instead we study dynamic range by analyzing the SNR for the sensors. In [fossum2013modeling], Fossum states that the exposure referred signal-to-noise ratio $\text{SNR}_{H}$ is a better metric because of non-linearity in the QIS response, than the regular voltage based SNR. So, in this section, we first derive the expression for $\text{SNR}_{H}$, assuming an imaging model similar to [Chan16] and use it to characterize the dynamic range of the CIS and QIS.

Let $c$ denote the discretized illumination (photons per second) of a particular pixel we are interested in, in the scene we are capturing. Let $\tau$ denote the integration time. Then the number of photons $X$ reaching the sensor in a particular frame is modelled as a Poisson random variable $X\sim\text{Poisson}(\theta),$ where $\theta=\tau c$. Let $B$ denote the output of the sensor of a sensor which can count up to $L$ photons. Then $B=X$, if $X<L$, and $B=L$, if $X\geq L$.

$\text{SNR}_{H}$ is the ratio of the exposure signal to the exposure referred noise. The exposure referred noise is defined as $\sigma_{H}={\sigma_{B}\frac{\partial\theta}{\partial\mu_{B}}}$, where $\sigma_{B}$ is the standard deviation of $B$ and $\mu_{B}$ is the mean and $\theta=\tau c$. So, for sum of T frames taken at a particular exposure, the exposure referred signal-to-noise ratio is

For sum of T frames taken at a particular exposure, the exposure signal is $T\theta$ and the exposure referred noise is $\sqrt{T}\sigma_{H}$. So,

The expressions for $\mu_{B}$ and $\sigma_{B}$, the mean and standard deviation of the output of the sensor $B$ is derived in theorem II and then we find the expression for $\frac{\partial\mu_{B}}{\partial\theta}$.

Let $B$ (the output of the sensor) be a thresholded Poisson random variable defined as $B=X$, if $X<L$, and $B=L$, if $X\geq L$ where $X\sim\text{Poisson}(\theta)$, for some fixed integer $L>0$. Then, the mean $\mu_{B}$ and the variance $\sigma_{B}^{2}$ of the random variable $B$ are

μ_B &= θ(ψ_L-1(θ)) + L(1 - ψ_L(θ))
σ_B^2 = L^2 - ∑_n=0^L-1 ((2n+1) ψ_n+1(θ)) - μ_B^2 where $\psi_{q}(s)=\sum\limits_{k=0}^{q-1}\frac{s^{k}e^{-s}}{k!}$ the incomplete gamma function.

From Theorem II, we can obtain the expression for $\frac{\partial\mu_{B}}{\partial\theta}$ as

Now that we have the expression for $SNR_{H}$, we conduct two experiments to study and compare the dynamic range of the CIS and QIS.

1. The exposure referred signal-to-noise ratio ($\text{SNR}_{H}$) is compared for QIS and CIS in \freffig:SNR_CISvsQIS. The details on how the experiment was run can be found in the figure. Two main observations are made. First, notice that the Dynamic range of one single exposure for QIS in both modes are much larger than the dynamic range of the CIS. Also, notice that single bit QIS has a better range than the 2 bit QIS. This implies that QIS with lesser number of bits is more advantageous in terms of dynamic range. Second, thanks to the extended overexposure latitude, the combined $\text{SNR}_{H}$ of the QIS with multiple exposures is stable over a wide range of illumination, without the dipping effects that is observed in CIS. This means, we can achieve a consistent SNR over all parts of the HDR image in a QIS.

The two observations made here make it obvious that QIS has a clear advantage over CIS in HDR imaging. Not only does QIS have better dynamic range, it also has more consistent SNR over the range of illumination in which it is operating, compared to CIS which has sudden dips in the SNR.

2. In this experiment (\freffig:DR_cpmpare), we show using a synthetic imaging experiment that the QIS has a better dynamic range than the CIS. We again compare 3 different modes - CIS, one bit QIS and three bit QIS. There is no post processing of the data other than just tone mapping for the QIS. Notice that the PSNR for the QIS modes are much higher than the CIS. The figure also points towards regions where the smaller dynamic range of the CIS causes the loss of information in certain bright regions, which the QIS preserves.

While the first experiment showed that QIS is better at HDR imaging theoretically, the second experiment where we simulated both the QIS and CIS imaging, showed visually that QIS has a better dynamic range than the CIS.