Adaptive Gating for Single-Photon 3D Imaging

Post-processing for pile-up compensation. There is extensive prior work on post-processing techniques for compensating the effects of pile-up. Perhaps the best known is Coates’ technique [12], and its generalizations [33, 15, 16, 19, 38, 37]. Coates’ technique uses a probabilistic model for photon arrivals to estimate the true incident scene transient, from which we can estimate depth. Alternatively, Heide et al. [17] use the same probabilistic model to perform maximum likelihood estimation directly for depth. We discuss how these approaches relate to our work in Section 3.

Gating schemes. Gating refers to the process of desynchronizing laser pulse emission and SPAD acquisition, and is commonly used to mitigate the effects of pile-up. The most common gating technique, known as fixed gating, uses a fixed delay between laser pulse emission and the start of SPAD acquisition, suppressing the detection of early-arriving photons. If the gating delay approximately matches the scene depth, fixed gating significantly reduces pile-up; otherwise, fixed gating will either have no significant effect, or may even suppress signal photons if the gating delay is after the scene depth. Gupta et al. [15] introduced a gating technique that uses uniformly-distributed delays spanning the entire depth range of the SPAD. This uniform gating technique helps mitigate pile-up, without requiring approximate knowledge of scene depth. Lastly, Gupta et al. [15] showed that it is possible to achieve uniformly-distributed delays between pulse emission and the start of SPAD acquisition by operating the SPAD without gating, at free-running mode [20, 13, 3, 37, 38]. We discuss fixed gating, uniform gating, and free-running mode in Section 4.

Spatially-adaptive LiDAR. SPAD-based LiDAR typically uses beam steering to raster-scan individual pixel locations. Recent work has introduced several techniques that, instead of performing a full raster scan, adaptively select spatial locations to be scanned, in order to accelerate acquisition [8, 35, 49]. These techniques are complementary to ours: Whereas they adaptively sample spatial scan locations, our technique adaptively samples temporal gates.

Other LiDAR technologies. There are several commercially-available technologies for light detection and ranging (LiDAR), besides SPAD-based LiDAR [39]. A common alternative in autonomous vehicles uses avalanche photodiodes (APDs) [25]. Compared to SPAD-based LiDAR, APD-based LiDAR does not suffer from pile-up, but has reduced sensitivity. Other LiDAR systems use continuous-wave time-of-flight (CWToF) cameras [14, 46, 22]. CWToF-based LiDAR is common in indoor 3D applications [45, 42, 4]. Unlike SPAD-based and APD-based LiDAR, CWToF-based LiDAR is sensitive to global illumination (multi-path interference).

Other SPAD applications. SPADs find use in biophotonics applications [10], including fluorescence-lifetime imaging microscopy [23, 9, 5, 24, 44], super-resolution microscopy [28], time-resolved Raman spectroscopy [28], and time-domain diffuse optical tomography [50, 34]. Other applications include non-line-of-sight imaging [48, 11, 30, 27], and high-dynamic-range imaging [19, 18].

Fixed gating. A fixed gating scheme uses the same gate for all $P$ TCSPC cycles, $g_{p}=g_{\text{fixed}},\,p\in\left\{1,\dots,P\right\}$. So long as this fixed gate is before the scene depth, $g_{\text{fixed}}<d$, it will prevent the detection of early-arriving photons due to ambient light, and thus increase the probability of detection of signal photons. For fixed gating to be effective, $g_{\text{fixed}}$ should be close, and ideally equal to the true depth $d$, as setting s$g_{\text{fixed}}=d$ maximizes the detection probability $p_{d}^{g_{\text{fixed}}}\left(d\right)$ in Equation (5). Unfortunately, this requires knowing the true depth $d$, or at least a reliable estimate thereof; such an estimate is generally available only after several cycles.

Uniform gating. Gupta et al. [15] introduced a uniform gating scheme, which distributes gates uniformly across the entire depth range. If, for simplicity, we assume that the numbers of cycles and temporal bins are equal, $P=\mathrm{T}$, then uniform gating sets $g_{p}=p,\,p\in\left\{1,\dots,P\right\}$. This maximizes the detection probability of each bin for a few cycles, and “averages out” pile-up effects. Compared to fixed gating, uniform gating does not require an estimate of the true depth $d$. Conversely, uniform gating cannot take advantage of increasing information about $d$ as more cycles finish.

Desired behavior for adaptive gating. Before formally describing our adaptive gating scheme, we describe at a high-level the desired behavior for such a scheme. Intuitively, an ideal gating scheme should behave as a hybrid between fixed and uniform gating. During the early stages of LiDAR operation (first few cycles) we have little to no information about scene depth—all temporal bins have approximately equal probability of being the true depth. Thus, a hybrid scheme should mimic uniform gating to explore the entire depth range. During the later stages of LiDAR operation (last few cycles), we have rich information about scene depth from the detection times recorded during preceding cycles—only one or few temporal bins have high probability of being the true depth. Thus, a hybrid scheme should mimic (near-fixed) gating, to maximize the detection probability of the few remaining candidate temporal bins. At intermediate stages of LiDAR operation, the hybrid scheme should progressively transition from uniform towards fixed gating, with this progression adapting from cycle to cycle to the information about scene depth available from previously-recorded detection times.

Thompson sampling. To turn the above high-level specification into a formal algorithm, we use two building blocks. First, we use the probabilistic model of Section 3 to quantify the information we have about scene depth $d$ at each cycle. At the start of cycle $p$, the LiDAR has recorded detection times $\bar{s}_{p-1}\equiv\left\{s_{q},q=1,\dots,p\right\}$ using gate times $\bar{g}_{p-1}\equiv\left\{g_{q},q=1,\dots,p\right\}$. Then, the depth posterior $p_{\bar{s}_{p-1}}^{\bar{g}_{p-1}}\left(d\right)$ of Equation (11) represents all the information we have available about scene depth, from both recorded detection times and any prior information ($p_{\text{prior}}\left(d\right)$).

Reward function. We motivate our choice of reward function as follows: At cycle $p$, Thompson sampling assumes that the true depth equals the depth hypothesis $\tilde{d}$ sampled from the depth posterior. Equivalently, Thompson sampling assumes that the detection time we will measure after cycle $p$ concludes is distributed as $s_{p}\sim p_{\tilde{d}}^{g_{p}}\left(s\right)$ (Equation (5)). As we aim to infer depth, we should select a gate $g_{p}$ such that, if we estimated depth from only the next detection $s_{p}$, the resulting estimate would be in expectation close to the depth hypothesis $\tilde{d}$ we assume to be true. Formally,

In Equation (15), to estimate depth from the expected detection time $s$, we use the same MAP depth estimator of Equation (12) as for the final depth estimate $\hat{d}_{\text{MAP}}\left(\bar{s},\bar{g}\right)$. The MAP depth estimator is optimal with respect to the $\mathcal{L}_{0}$ loss (Equation (13)), thus we use the same loss for the reward function. Selecting the gate $g_{p}$ requires maximizing the reward function $R\left(\tilde{d},g\right)$, which we can do analytically.

Intuition behind Thompson sampling. Before concluding this section, we provide some intuition about how Thompson sampling works, and why it is suitable for adaptive gating. We can consider adaptive gating with Thompson sampling as a procedure for balancing the exploration-exploitation trade-off. Revisiting the discussion at the start of this section, fixed gating maximizes exploitation, by only gating at one temporal bin (or a small number thereof); conversely, uniform gating maximizes exploration, by gating uniformly across the entire depth range. During the first few cycles, adaptive gating maximizes exploration: as only few measurements are available, the depth posterior is flat, and depth hypotheses (and thus gate times) are sampled approximately uniformly as with uniform gating. As the number of cycles progresses, adaptive gating shifts from exploration to exploitation: additional measurements make the depth posterior concentrated around a few depth values, and gate times are sampled mostly among those. After a sufficiently large number of cycles, adaptive gating maximizes exploitation: the depth posterior peaks at a single depth, and gate times are almost always set to that depth, as in fixed gating. Figure 4 uses simulations to visualize this transition from exploration to exploitation. This behavior matches the one we set out to achieve at the start of this section. Lastly, we mention that Thompson sampling has strong theoretical guarantees for asymptotic optimality [41]; this suggests that our adaptive gating scheme balances the exploration-exploitation trade-off in a way that, asymptotically (given enough cycles), maximizes depth accuracy.

Adaptive gating with adaptive exposure. The adaptive gating scheme of Section 4 lends itself to a modification that also adapts the number of cycles $P$, and thus exposure time. In Algorithm 1, we can terminate the while loop early at a cycle $p\leq P$, if the depth posterior $p_{\bar{s}_{p}}^{\bar{g}_{p}}\left(d\right)$ becomes concentrated enough that we expect depth estimation to have low error. Formally, we define a termination function as the expected error with respect to the depth posterior:

In Equation (17), we use the same MAP depth estimator as for the final depth estimate $\hat{d}_{\text{MAP}}\left(\bar{s},\bar{g}\right)$, and the $\mathcal{L}_{0}$ loss for which the MAP estimator is optimal (Equation (13)). These choices are analogous to our choices for the definition of the reward function $R\left(\tilde{d},g\right)$ in Equation (15). At the end of each cycle $p$, we check whether the termination function $L\left(\bar{s}_{p},\bar{g}_{p}\right)$ is smaller than some threshold, and terminate acquisition if it is true. In Algorithm 1, we show in blue how we modify our original adaptive gating algorithm to include this adaptive exposure procedure.

Prototype. Our SPAD-based LiDAR prototype comprises a fast-gated SPAD (Micro-photon Devices), a $532\,\mathrm{nm}$ picosecond pulsed laser (NKT Photonics NP100-201-010), a TCSPC module (PicoHarp 300), and a programmable picosecond delayer (Micro-photon Devices). We operated the SPAD in triggered mode (adaptive gating) or free-running mode, with the programmable dead time set to $81\,\mathrm{ns}$. We set the laser pulse frequency to $10\,\mathrm{MHz}$, with measurements discretized to $500$ bins ($100\,\mathrm{ps}$ resolution). The raster scan resolution is $128\times 128$, and acquisition time per scan point is $100\,\mathrm{\mu s}$, for a total acquisition time of $1.6\,\mathrm{s}$. We note that, if gate selection happens during dead time—easily achievable with optimized compute hardware—our procedure does not introduce additional acquisition latency.

Outdoor scenes. Outdoor experiments (Figures 1 and 5) were under direct sunlight (no shadows or clouds) around noon (11 am to 2 pm in the United States), with an estimated background strength of 0.016 photons per pulse per bin. Figure 1 shows a set of 3D reconstructions for the “Leaf” scene, captured at noon with a clear sky and under direct sunlight. Figures 1(a)-(b) show depth reconstructions using free-running mode and adaptive gating under fixed exposure times: our method reduces RMSE by around $3\times$. Figure 1(c) shows the depth reconstruction using adaptive gating combined with adaptive exposure. This combination reduces both RMSE and exposure time (by around $3\times$) compared to free-running mode. Figure 5 shows depth reconstructions for the “Stairs” scene, captured under the same sunlight conditions. This scene has significant SBR variations, and we observe that adaptive gating outperforms free-running mode in all SBR regimes.

Indoor scenes. Figure 6 shows reconstructions of the “Horse” scene, an indoor scene of a horse bust placed in a light booth. Under fixed exposure time, our method achieves $35\%$ lower RMSE compared to free-running mode. We note that RMSE improvement is less pronounced than in outdoor scenes. As the indoor scene has significantly higher SBR, this suggests that adaptive gating offers a bigger advantage over free-running mode at low SBRs.

Active time. As Gupta et al. [15] explain, using free-running mode provides similar advantages as uniform gating, and at the same time maximizes the time during which the SPAD is active—and thus maximizes photon detections. Adaptive gating, likewise, also results in reduced active time compared to free-running mode. However, experimentally we found that, despite the reduced photon detections, our adaptive gating scheme still results in improved depth accuracy for all practical dead time values (see Section 6 and additional experiments evaluating the effect of active and dead time in the supplement).

Hardware considerations. Our adaptive gating scheme requires using a SPAD with a controllable gate that can be reprogrammed from pulse to pulse. We implemented this using the same gated SPAD hardware as Gupta et al. [15] did for their uniform gating scheme. The main additional hardware requirements are electronics that can produce the gate control signals at $f_{\text{ptp}}$ frequency and $\Delta$ resolution. Both gating schemes require considerably more expensive hardware compared to free-running mode operation, which only requires an ungated SPAD. Whether the improved depth accuracy performance justifies the increased hardware cost and complexity is an application-dependent consideration.

Conclusion. We introduced an adaptive gating scheme for SPAD-based LiDAR that mitigates the effects of pile-up under strong ambient light conditions. Our scheme uses a Thompson sampling procedure to select a gating sequence that takes advantage of information available from previously-measured laser pulses, to maximize depth estimation accuracy. Our scheme can also adaptively adjust exposure time per-pixel, as necessary to achieve a desired expected depth error. We showed that our scheme can reduce both depth error and exposure time by more than $100\%$ compared to previous SPAD-based LiDAR techniques, including when operating outdoors under strong sunlight.

Acknowledgments. We thank Akshat Dave, Ankit Raghuram, and Ashok Veeraraghavan for providing the high-power laser for experiments and invaluable assistance in its use; as well as Matthew O’Toole, David Lindell, and Dorian Chan for help with the SPAD sensor. This work was supported by NSF Expeditions award 1730147, NSF CAREER award 2047341, DARPA REVEAL contract HR0011-16-C-0025, and ONR DURIP award N00014-16-1-2906. Ioannis Gkioulekas is supported by a Sloan Research Fellowship.