Coded Illumination for 3D Lensless Imaging

Mask-based lensless cameras replace the lens with a layer of coded mask and capture linear multiplexed measurements with an image sensor. The mask pattern can be placed parallel to the sensor plane at distance $d$, as illustrated in Figure 1. In general, we can model the measurement recorded at a sensor pixel $(s_{u},s_{v})$ as a linear function of the scene intensity as

$$y(s_{u},s_{v})=\int I(x,y,z)\,\varphi(s_{u},s_{v};x,y,z)\,dxdydz,$$

(1)

where $I(x,y,z)$ denotes the image intensity at 3D location $(x,y,z)$ and $\varphi(s_{u},s_{v};x,y,z)$ denotes the point spread function (PSF) or the sensor response recorded at $(s_{u},s_{v})$ in the sensor plane for a point source at $(x,y,z)$.

The general system in (1) can be simplified depending on the system design and placement and pattern of the mask. In our proposed method, we use a separable model proposed in [1], where we use a rank-1 matrix as the amplitude mask. With the separable mask placed parallel to the image sensor, the PSF of an arbitrary point, $\varphi(s_{u},s_{v};x,y,z)$, will be a rank-1 matrix, and the general model in (1) can be written in a simpler form as a separable system.

Suppose we discretize the continuous scene $I(x,y,z)$ into $D$ depth planes $\mathbf{I_{1},\ldots,I_{D}}$, each with $N\times N$ pixels. The separable system can be represented in the following compact form:

$$\mathbf{Y}=\sum_{k}\mathbf{\Phi_{k}I_{k}\Phi_{k}^{T}}.$$

(2)

$\mathbf{Y}$ represents $M\times M$ sensor measurements and $\Phi_{k}$ represents the system matrix for the $k$-th depth plane.

We use a projector separated by baseline distance $B$ from the camera to illuminate the scene with a sequence of coded illumination patterns (as illustrated in Figure 1. The effect of coded illumination can be modelled as an element-wise product between the illumination pattern and the scene. We divide the field-of-view (FOV) cone of the projector into $N\times N$ angles, which also determines the spatial discretization of the scene. We generate a sequence of illumination patterns and capture the corresponding measurements on the sensor. The measurements captured for $i$-th pattern $\mathbf{P_{i}}$ can be represented as

$$\mathbf{Y}_{i}=\sum_{k}\mathbf{\Phi_{k}(P_{i}\odot I_{k})\Phi_{k}^{T}}.$$

(3)

Note that we assume the same illumination pattern for every depth plane at a time. This is because we use the projector to determine the scene discretization at every depth plane.

To recover the 3D scene as a stack of $D$ planes, $\mathbf{I=\{I_{1},\ldots,I_{D}\}}$, we solve the following regularized least-squares problem:

$$\min_{\mathbf{I_{1},\ldots,I_{D}}}~\sum_{i}\|\mathbf{Y_{i}}-\sum_{k}\mathbf{\Phi_{k}(P_{i}\odot I_{k})\Phi_{k}^{T}}\|_{2}^{2}+\lambda\|\mathbf{D(I)}\|_{2}.$$

(4)

$\mathbf{D(I)}$ represents a finite difference operator that computes local gradients of the 3D volume $\mathbf{I}$ along spatial and depth directions. The $\ell_{2}$ norm of the local differences provides the 3D total variation function that we use as the regularization function. The total variation function constrains the magnitude of the local variation in the reconstruction and is widely used in ill-conditioned image recovery problems [36, 16]. The optimization problem in (4) can be solved using an iterative least-squares solvers; we used the TVreg package [37].

As discussed in previous work on 3D lensless imaging [3, 5, 13, 6], the points at different depth in the scene provide a scaled version of the mask pattern as the sensor response. However, if the object is far from the lensless camera, the depth-dependent differences in the sensor reponse become almost negligible.

Coded illumination in our proposed system provides robust 3D reconstruction for two main reasons: (1) Coded illumination selects a subset of scene points that contribute to each sensor measurement. (2) Spatial separation between camera and projector (i.e., baseline) maps depth variations in scene points into depth-dependent shifts in the sensor response. Since shifted versions of the the mask pattern can be easily resolved compare to the scaled versions, the baseline plays a critical role in quality of 3D reconstruction.

Let us consider the 1D case of our proposed framework, the projector $P$ is placed at a baseline distance $B$ away from the camera $C$, as shown in Figure 1. For an arbitrary point at $(p,z)$ in the coordinate system of $C$, its measurements on camera $C$ can be written as

$$\phi(s;p,z)=\text{mask}\left[(1-\frac{d}{z})s+d\frac{p}{z}\right],$$

(5)

where $s$ denotes the coordinates on the camera sensor and $d$ denotes the sensor-to-mask distance. The coordinates of two arbitrary points ${A_{1}}(p_{1},z_{1})$ and ${A_{2}}(p_{2},z_{2})$ on the same ray of the projector in the coordinate system of $P$ become $(p_{1}+B,z_{1})$, $(p_{2}+B,z_{2})$ in the coordinate system of camera $C$ (because of the the baseline between camera and projector). We can represent the camera response or PSF corresponding to each of these points as

	$\displaystyle\phi(s;p_{1},z_{1})$	$\displaystyle=\text{mask}\left[(1-\frac{d}{z_{1}})s+\frac{dp_{1}}{z_{1}}-\frac{dB}{z_{1}}\right]$
		$\displaystyle=\text{mask}\left[(1-\frac{d}{z_{1}})(s-\frac{dB}{z_{1}-d})+d\frac{p_{1}}{z_{1}}\right]$		(6)

	$\displaystyle\phi(s;p_{2},z_{2})$	$\displaystyle=\text{mask}\left[(1-\frac{d}{z_{2}})s+\frac{dp_{2}}{z_{2}}-\frac{dB}{z_{2}}\right]$
		$\displaystyle=\text{mask}\left[(1-\frac{d}{z_{2}})(s-\frac{dB}{z_{2}-d})+d\frac{p_{2}}{z_{2}}\right].$		(7)

Note that $\frac{p_{1}}{z_{1}}=\frac{p_{2}}{z_{2}}$ because the two points are at different depths on the same ray angle. Therefore, the PSF of points $A_{1}$ and $A_{2}$ differ from each other with a scaling factor $1-\frac{d}{z}$ and a depth-dependent shift $\frac{dB}{z-d}$.

Specifically, when the object is far from the camera, we can often ignore the difference in depth scaling factor $1-\frac{d}{z}$, and the difference of the depth-dependent shift becomes $|\frac{dB}{z_{1}-d}-\frac{dB}{z_{2}-d}|.$ When the baseline is zero, two point light sources on the same ray are the scaled versions of each other, and the scaling factor becomes almost the same when the object distance $(z)$ is large. However, by separating the camera and project by baseline distance $B$, the camera observes a shifted 3D grid; two points on the shifted grid provide an angular difference with respect to the camera. Therefore, the depth resolvability of the system improves. This effect was previously discussed in [12] for more general geometries with multiple cameras.

In general, the projector $P$ and camera $C$ can be separated laterally and axially. Lateral separation provides depth-dependent shifts of the PSF, which we discussed in (6) and (7). Axial separation would provide depth-dependent scaling and shifts of the PSF, which can also be deduced from (6) and (7). For instance, if the camera and projector are separated axially by $\Delta z$, the depth-dependent shifts can be calculated by replacing $z_{1},z_{2}$ with $z_{1}+\Delta z,z_{2}+\Delta z$, respectively. Since these terms appear in the denominator, their influence on the PSF shift will be small compared to lateral baseline $B$.

We test different types of binary illumination patterns for the simulation. The patterns are designed to be binary to keep the model simple and to avoid the effect of non-linearity caused by the Gamma curve of the projector..
Uniform. One pattern that illuminates all the pixels simultaneously;
Random. A sequence of separable binary random matrices. We ensure that the union of all the patterns should illuminate all the pixels (i.e., if we add up all the illumination patterns, they should not have zero entries anywhere).
Shifting dots array. The base illumination pattern consists of dots separated by $k$ pixels along the horizontal and vertical directions. We then generate a total of $k^{2}$ illumination patterns, each of which is a shifted version of the base pattern. The summation of all the patterns will give us a uniform illumination pattern.
Shifting lines. Similar to shifting dots array, the base illumination patterns consist of horizontal lines separated by $k$ pixels along vertical axis and vertical lines separated along horizontal axis. We then generate shifted version of these two base patterns. The summation of all the patterns is a uniform illumination pattern.

We present simulation results with different number and types of illumination patterns in Figure 2. The simulated test scene is taken from NYU depth dataset[38]. The depth of scene is rescaled into the range from $40\text{cm}$ to $60\text{cm}$ and discretized into 50 depth planes to simulate the sensor measurements. The camera setup and the baseline between camera and projector are fixed during the simulation. The shifting lines and shifting dots outperform the uniform pattern in terms of depth RMSE. Also, the depth RMSE drops as we increase the number of illumination patterns.

The baseline between the lensless camera and the projector affects the depth resolvability of the system. Shifting the lensless camera by a distance, the camera observes the scene from a side view and transfers the depth difference of two points into angular difference. We present simulation results in Figure 3 to demonstrate the effect of camera-projector baselines. The number of illumination patterns for all the simulation are the same. We then fix the baseline along axial direction to 0cm and the baseline along lateral direction as $B=\{0,2.5,5,7.5\}\text{cm}$. As shown in Figure 3, the depth RMSE is decreased as we increase the baseline between the camera and the projector. When the baseline is zero, which means the camera and projector are overlapped, we barely distinguish any depth.

One important consideration for our method is that the target object should lie within the intersection of the sensor FOV cone and the projector illumination cone. As we increase the baseline, the intersection of the two cones is pushed farther from the sensor. Therefore, we should determine the maximum baseline based on the object distance, sensor FOV, and projector cone. If we increase the baseline beyond the maximum limit, then the reconstruction quality can decrease.

In existing structured illumination methods [27, 30, 31], a lens-based camera is used to capture the scene from the side view of the projector and depth map can be accurately reconstructed by triangulation. We can model the lens-based camera as an ideal pinhole camera (ignoring photon or sensor noise) for the sake of comparison with our method. We present simulation results comparing a pinhole-based camera with structured illumination in Figure 4. The baseline between the projector and the camera is fixed at 5cm in all the simulations. Results in Figure 4 show that the pinhole mask (that represents an ideal lens-based camera) provides better results compared to the MLS mask. Compared to mask-based lensless camera where the sensor measurements are multiplexed, a lens-based system can offer better conditioning and depth reconstruction. Nevertheless, a lens-based camera imposes additional burden in terms of device thickness, weight, and geometry.

In our proposed method, multiple frames of measurements are captured, which introduce additional limitations such as long capture time and low frame rate. In Figure 5, we present simulation results comparing our method with another multi-shot lensless imaging system called SweepCam [13]. SweepCam captures multiple frames of sensor measurements while translating the mask laterally. The translation of the mask offers a perspective shift in the measurements that depends on the depth of objects in the scene. In our simulations, the SweepCam mask is translated to 48 positions within a range of $2.88\text{mm}\times 2.88\text{mm}$. However, since the translating distance of the mask is limited by the sensor area, the SweepCam method fails to resolve the depths when the scene is farther than 30cm.

We present experimental results of 3D reconstruction with our proposed method for real objects in Figures 7 and 9. We show the results of reconstructed depth planes, estimated all-in-focus images and depth maps using uniform, shifting lines, and shifting dots patterns. For comparison, we captured the original image and depth map for each scene using Intel RealSense D415 depth camera, where the baseline between the lens-based camera and the projector is 55mm.

The results in Figure 7 compare 3D reconstruction with uniform and 48 shifting lines. In the first two rows, the scene is a slanted box, containing continuous depth varying from 40cm to 60cm. In the last two rows, the scene contains a red toy located at 40cm and a green toy lying from 50cm to 60cm. The results in the first three columns in Figure 7 represent three depth planes at 58cm, 62cm, and 66cm. The results show that the correct depth can be easily distinguished in images reconstructed with 48 shifting lines pattern, whereas depth planes reconstructed with the uniform illumination pattern show incorrect depth and intensity. The estimated all-in-focus image and depth maps for 48 shifting lines also appear significantly better than those from the uniform illumination patterns.

The results in Figure 8 compare different number and types of illumination patterns. We observe that the uniform illumination pattern barely recover any depth. The illumination pattern with 16 and 49 shifting dots provide better results than uniform illumination. 16 shifting lines provide slightly better results compared to shifting dots and 48 shifting lines provide significanlty better image and depth map.

In summary, the ill-conditioned system matrices with uniform illumination pattern cause various artifacts in 3D reconstruction. Capturing measurements from coded illumination improves the conditioning of the overall system and the reconstructed images have better spatial and depth resolution. Increasing the number of illumination patterns provides better reconstruction. More illumination patterns would require longer acquisition time as well, which enforces a trade off between the quality of reconstruction and data acquisition time.

We show experimental results for different baselines in Figure 9. We captured the same scene with $5.5\text{cm}$ and $10.5\text{cm}$ baseline and performed 3D reconstruction with the respective measurements. The results in Figure 9 show that 10.5cm baseline offers finer depth resolution (indicated as narrow depth of field) compared to the reconstruction with 5.5cm baseline. The improvement is small, and this effect was observed in the simulation results in Figure 3(b) that show the depth RMSE of the system tapers off as we increase the baseline between the camera and the projector.