Adaptive Illumination based Depth Sensing using Deep Superpixel and Soft Sampling Approximation

Given RGB images, early depth prediction methods relied on hand-crafted features and probabilistic graphics models. Karsch et al. [12, 13] estimate the depth based on querying an RGBD image database. A Markov random field model is applied in [14] to regress depth from a set of image features. Recently deep learning (DL) and convolutional neural networks (CNNs) have been applied to learn the mapping from single RGB images to dense depth maps [15, 16, 17, 18, 19, 20, 21, 22]. These DL-based approaches achieve state-of-the-art performance because better features are extracted and better mappings are learned from large-scale datasets [23, 24, 25].

Given sparse depth measurements, traditional image filtering and interpolation techniques [26] can be applied to reconstruct the dense depth map. Hawe [27] and Liu [28] study the sparse depth map completion problem from the compressive sensing perspective. DL techniques have also been applied to the sparse depth completion problem. A sparse depth map can either be fed into conventional CNNs [29] or sparsity invariant CNNs [30]. When the sampling rate is low, the sparse depth map completion task is challenging.

If both RGB images and sparse depth measurements are provided, traditional guided filter approaches [31, 32] can be applied to refine the depth map. Optimization algorithms that promote depth map priors while maintaining fidelity to the observation are proposed in [33, 34, 35]. Various DL-based methods have been developed [29, 36, 6, 37, 38, 39, 40, 41]. During training and testing, most DL approaches are trained and tested using random or regular grid sampling masks. Because depth completion is an active research area, we do not want to limit our adaptive sampling method to a specific depth estimation method.

Irregular sampling [42, 43, 44] is well studied in the computer graphics, image processing and computational imaging literature to achieve good representation of images, and we have witnessed its application in compressive sensing [45], ghost imaging [46], wireless imaging [47], quantitative phase imaging [48], Fourier ptychography [49], and etc. Making the sampling distribution adaptive to the signal can further improve representation performance. Eldar et al. [50] proposed a farthest point strategy which performs adaptive and progressive sampling of an image. Inspired by the lifting scheme of wavelet generation, several progressive image sampling techniques were proposed [51, 52]. Ramponi et al. [53] applied a measure of the local sample skewness. Lin et al. [54] utilized the generalized Ricci curvature to sample grey scale images as manifolds with density. A kernel construction technique is proposed in [55]. Taimori et al. [56] investigated space-frequency-gradient information of image patches for adaptive sampling.

Specific reconstruction algorithms are needed for each of these irregular or adaptive sampling methods [42, 50, 52, 51, 53, 54, 55, 56] to reconstruct the fully sampled signal. Furthermore, handcrafted features are applied to these sampling methods. Finally, these sampling techniques are all applied to the same modality (RGB or grey scale image). Recently, Dai et al. [57] applied DL technique to the adaptive sampling problem. The adaptive sampling network is jointly optimized with the image inpainting network. The sampling probability is optimized during training, and binarized during testing. Good performance is demonstrated for X-ray fluorescence (XRF) imaging at a sampling rate as low as $5\%$. Kuznetsov et al. [58] predicted adaptive sampling maps jointly with reconstruction of Monte Carlo (MC) rendered images using DL. A differentiable render simulator with respect to the sampling map was proposed. Huijben et al. [59, 60] proposed a task adaptive compressive sensing pipeline. The sampling mask is trained with respect to a specific task and is fixed during imaging. Gumbel-max trick [61, 62] is applied to make the sampling layer differentiable.

All of the above DL-based sampling methods predict a per pixel sampling probability [57, 59, 60] or a sampling number [58]. Good sampling performance has not been demonstrated under extreme low sampling rates $(<1\%)$. Directly enforcing priors on sampling locations is effective when the sampling rate is low. This requires the adaptive sampling network to predict sampling locations ($(x,y)$ coordinates) directly and the sampling process to be differentiable. For the RGB and sparse depth adaptive sampling task, Wolff et al. [63] use the SLIC superpixel technique [64] to segment the RGB image and sample the depth map at the center of mass of each superpixel. A bilateral filtering based reconstruction algorithm is proposed to reconstruct the depth map. A spatial distribution prior is implicitly enforced by superpixel segmentation, resulting in good sampling performance under low sampling rates. The sampling and reconstruction methods are not optimized jointly, leaving room for improvement. In this paper, we show that jointly training recent DL-based superpixel sampling networks [65, 63] and depth estimation networks [42, 50, 52, 51, 53, 54, 55, 56] can be adapted to the problem of dense depth map estimation from sparse LiDAR data with improved accuracy. Bergman et al. [8] warp a uniform sampling grid to generate the adaptive sampling mask. The warping vectors are computed utilizing DL-based optical flow estimated from the RGB image. A spatial distribution prior is enforced by the initial uniform sampling grid. End-to-end optimization of the sampling and depth estimation networks is performed and good depth reconstruction is obtained under low sampling rates. In the pipeline of [8], there are 4 sub-networks, 2 for sampling and the other 2 for depth estimation. They are jointly trained but only the final depth estimation results are demonstrated. The whole pipeline is bulky and expensive. More importantly, it is hard to assess if the improvement on depth estimation comes from the sampling part or the depth estimation part of the pipeline. In this paper, we decouple these two parts and study each individual module to better understand their contribution towards the final depth estimate. Finally, a bilinear sampling kernel is applied in [8] to make the optimization of the sampling locations differentiable. In contrast, we propose a novel differentiable relaxation of the sampling procedure and show its advantages over the bilinear sampling kernel.

As shown in Figure 2, the input RGB image is denoted by $I$. The mask generation network $NetM$ produces a binary sampling mask $B=NetM(I,c)$, where $c\in[0,1]$ is the predefined sampling rate. Elements in $B$ equal to $1$ correspond to sampling locations and $0$ to non-sampling location. The LiDAR system samples depth according to $B$ and produces the measured sparse depth map $D^{\prime}$. In synthetic experiments, if the ground truth depth map $D$ is given, the measured sparse depth map $D^{\prime}$ is obtained according to

where $\odot$ is the element-wise product operation. The reconstructed depth map $\bar{D}$ is obtained by the depth estimation network $NetE$, that is,

The overall adaptive depth sensing and depth estimation pipeline is shown in Figure 2. End-to-end training can be applied on $NetM$ and $NetE$ jointly. The adaptive depth sampling strategy is learned by $NetM$, while $NetE$ estimates the final dense depth map. An informative sampling mask is beneficial to depth estimation algorithms in general, not just to $NetE$. Given a limitted depth sampling budget and an RGB image, we want to sample depth value on the ambiguous regions in a balanced way. During testing, we can replace the inpainting network $NetE$ with other depth estimation algorithms. Network architectures and training details of $NetE$ and $NetM$ are discussed in the following subsections.

We use the network architecture in [29] for the depth estimation network. The network is an encoder-decoder pipeline. The encoder takes a concatenated $I$ and $D^{\prime}$ as input ($4$ channels) and encodes them into latent features. The decoder takes the low spatial resolution feature representation and outputs the restored depth map $\bar{D}=NetE(I,D^{\prime})$. Readers can refer to [29] for the detailed architecture of $NetE$.

Because method [29] is differentiable with respect to $D^{\prime}$ (unlike [37]) and its network architecture is standard without customized fusion modules [38, 6, 37], we choose it as $NetE$ and jointly train $NetM$ with it according to Figure 2. We found out that the trained $NetM$ can generalize well to other depth estimation methods during testing.

Existing irregular sampling techniques [50, 44] and adaptive depth sampling methods [8, 63] explicitly or implicitly make sampling points evenly distributed spatially. Such prior is important when the sampling rate is low. Inspired by the SLIC superpixel [64] based adaptive sampling method [63], we propose to utilize recent DL-based superpixel networks [66, 65] as $NetM$. As demonstrated in Figure 2, $NetM$ adapts to the task of depth sampling after being jointly trained with $NetE$.

Superpixel with fully convolutional networks (FCN) [66] is one of the DL-based superpixel techniques. It predicts the pixel association map $Q$ given an RGB image $I$. Its encoder-decoder network architecture is shown in Figure 3. Similar to the SLIC superpixel method [64], a combined loss that enforces similarity property of pixels inside one superpixel and spatial compactness is applied. Readers can refer to [66] for more details.

Given an RGB image $I$ with spatial dimensions $(H,W)$, under the desired depth sampling rate $c$, we have $N_{p}=H\cdot W$ pixels and $N_{s}=c\cdot H\cdot W$ superpixels. The sampled depth location is the weighted mass center of each superpixel. We denote the subset of pixels as $\mathcal{P}=\{\mathcal{P}_{0},...,\mathcal{P}_{N_{s}-1}\}$, where $\mathcal{P}_{i}$ is a set of pixels associated with superpixel $i$. Pixel $p$’s CIELAB color property and $(x,y)$ coordinates are denoted by $\mathbf{f}(p)\in\mathbb{R}^{3}$ and $\mathbf{c}(p)\in\mathbb{R}^{2}$, respectively. CIELAB color space is used here as we follow the FCN [66] and SLIC superpixel [64] setup. The loss function is given by

Here we have

where $m$ is a weight balancing term between the CIELAB color similarity and spatial compactness, $\mathcal{N}_{p}$ is the set of superpixels surrounding $\mathbf{p}$, $q_{s}(p)$ is the probability of a pixel $p$ being associated with superpixel $s$ and is derived from the associate map $Q$, $\mathbf{u}_{s}\in\mathbb{R}^{3}$ and $\mathbf{l}_{s}\in\mathbb{R}^{2}$ are the color property and locations of superpixel $s$, $\mathbf{f}^{\prime}(p)\in\mathbb{R}^{3}$ and $\mathbf{c}^{\prime}(p)\in\mathbb{R}^{2}$ are respectively the reconstructed color property and location of pixel $p$.

Defined in Equation 4(a), we denote the collection of $\mathbf{l}_{s}$, $s=0,...,N_{s}-1$ , as $S$. Depth values at locations $S$ would be measured during the depth sampling. In order to train $NetM$ and $NetE$ jointly, the sampling operation $g$, which computes the sampled sparse depth map $D^{\prime}$ from depth ground truth $D$ and sampling location $S$, $D^{\prime}=g(D,S)$, needs to be differentiable with respect to $S$. Unfortunately, such sampling operation $g$ is not differentiable in practice. Bergman et al. [8] apply a bilinear sampling kernel to differentiably correlate $S$ and $D^{\prime}$. The computed gradients rely on the $2\times 2$ local structure of the ground truth depth map $D$. The computed gradients are not stable when the sampling location is sparse. Thus limited sampling performance is obtained. We propose a soft sampling approximation (SSA) strategy during training. SSA utilizes a larger window size compared to the bilinear kernel and achieves better sampling performance.

As shown in Figure 4, during training, given a sampling location $\mathbf{l}_{s}\in S$, we find a local $h\times w$ window $W$ around $\mathbf{l}_{s}$. The depth value $d_{s}$ at $\mathbf{l}_{s}$ is a weighted average of the depth values in $W$,

where $\mathcal{N}_{W}$ includes the indices of all pixels in $W$, $\mathbf{w}_{i}$ is the $i^{th}$ pixel’s location in $W$, $d_{i}$ is the depth value of $\mathbf{w}_{i}$, the weights $k_{i}$ are computed according to the Euclidean distance $\rho_{i}$ between $\mathbf{l}_{s}$ and $\mathbf{w}_{i}$, scaled by a temperature parameter $t$,

When the temperature parameter $t\rightarrow 0$, the sampled depth value $d_{s}$ is equal to the depth value $d_{n}$ of the nearest pixel $\mathbf{w}_{n}$. When $t$ is large, the soft sampled depth value $d_{s}$ is different from $d_{n}$. We gradually reduce $t$ during the training process. During testing, we find the nearest neighbor pixel $\mathbf{w}_{n}$ of $\mathbf{l}_{s}$ and sample the depth value $d_{n}$ at $\mathbf{w}_{n}$.

Given the training dataset consisting of the aligned RGB image $I$ and the ground truth depth map $D$, we first train $NetE$ by minimizing the depth loss,

where $D^{\prime}$ is obtained by applying a random sampling mask on $D$ with sampling rate $c$.

Then we initialize the superpixel network $NetM$ using the RGB image $I$. $\mathcal{L}_{SLIC}$ is minimized according to Equation 3. The initialized $NetM$ approximates the SLIC superpixel segmentation on RGB image. If we sample the depth value on $\mathbf{l}_{s}$ of each superpixel, the sampling pattern would be similar to  [63].

Finally, we freeze $NetE$ and train $NetM$ in Figure 2 by minimizing

where $q$ is the weighting terms of $\mathcal{L}_{SLIC}$. The SSA trick shown in Figure 4 is applied and the temperature parameter $t$ gradually decreases during training.

We fix $NetE$ when training $NetM$. Optimizing $NetE$ and $NetM$ simultaneously would obtain $better$ depth reconstruction accuracy [8]. However, similarly to [57], we would utilize other depth estimation methods than $NetE$ during testing. We want to make the adaptive depth sampling mask be general and applicable to many depth estimation algorithms.

We use both the KITTI depth completion dataset [30] and the NYU-Depth-V2 dataset [23] for our experiments. The KITTI depth completion dataset consists of aligned ground truth depth maps (from LiDAR sensor) and RGB images. The original KITTI training and validation set split is applied. There are $42949$ and $3426$ frames in the training and testing sets, respectively. We only use the bottom center crop $240\times 960$ of the images because the LiDAR sensor has no measurements at the upper part of the images. The NYU-Depth-V2 dataset consists of RGB and depth images captured by a Microsoft Kinect. $48004$ synchronized RGB-depth image pairs are used for training. $654$ RGB-Depth image pairs from the small labeled test dataset are used for testing. Following  [29], the original frames of resolution $480\times 640$ are down sampled to half resolution, producing a final resolution of $240\times 320$.

For the KITTI depth completion dataset, the ground truth depth maps are not dense because they are measured by a velodyne LiDAR device. In order to perform adaptive depth sampling, we need dense depth maps to sample from. Similarly to [8], a traditional image inpainting algorithm [31] is applied to densify the depth ground truth. During evaluation, we compare the estimated dense depth maps to the original sparse ground truth depth maps. For the NYU-Depth-V2 dataset, the raw depth values are projected onto the synchronized RGB images and inpainted using a bilateral filter method available in the official toolbox.

During the training of $NetE$, we follow Ma et al.’s setup [29]. The batch size is set equal to $16$. The ResNet encoder is initialized with pretrained weights using the ImageNet dataset [67]. Stochastic gradient descent (SGD) optimizer with momentum $0.9$ is used. We train $100$ epochs in total. The learning rate is set to be equal to $0.01$ at first and reduced by $80\%$ at every $25$ epochs. $NetE$ is trained individually under different sampling rates $c=1\%,0.25\%$ and $0.0625\%$ using random sampling masks. We also train FusionNet [6] and SSNet [36] under different sampling rates using random sampling masks. The same training procedure in their original papers are used. They serve as alternative depth estimation methods.

We test the proposed sampling algorithm under $3$ sampling rates, $c=1\%,0.25\%$ and $0.0625\%$. For the KITTI depth completion dataset, they correspond to $N_{s}=2304,576$ and $144$ depth samples (superpixels) in the $240\times 940$ image. For the NYU-Depth-V2 dataset, they correspond to $N_{s}=768,192$ and $48$ depth samples (superpixels) in the $240\times 320$ image. $NetM$ is configured to output the desired number of samples. During the training of $NetM$, we pretrain it using the SLIC loss. $m$ in Equation 3 is set equal to be $1$. ADAM optimizer [68] is applied. Learning rate is set to be $5\times 10^{-5}$. We train $100$ epochs in total.

After $NetM$ is initialized, we finally jointly train $NetM$ and $NetE$ according to Figure 2. Loss defined in Equation 8 is optimized with $q$ equal to $10^{-6}$, resulting in $\mathcal{L}_{depth}$ being equal to about $10$ times of $q\cdot\mathcal{L}_{SLIC}$ in value. The window size of the soft depth sampling module is equal to $5$. Temperature $t$ defined in Equation 6 decreases from $1.0$ to $0.1$ linearly during training. We experimentally find that $NetM$’s performance is not sensitive to the SSA related settings, such as the window size, initial temperature and temperature decay policy. Batch size is set equal to $8$ and this is the largest batch size we can use for both $NetM$ and $NetE$ in an NVIDIA 2080Ti GPU (11GB memory). As discussed in Section III-E, $NetE$ is fixed during the training to make $NetM$ generalize well to other depth estimation methods. Learning rate of $NetM$ is assigned to be equal to $10^{-4}$ and is reduced by $50\%$ every 10 epochs. SGD optimizer with momentum $0.9$ is used. We found that 50 epochs in total are adequate for converge.

Our proposed adaptive depth sampling framework is implemented in PyTorch and our implementation is available at: https://github.com/usstdqq/adaptive-depth-sensing.

For the adaptive depth sampling and estimation task, we demonstrate the advantages of our proposed adaptive sampling mask $NetM$, over the use of random, uniform grid and Poisson [44] sampling masks, as well as other state-of-the-art adaptive depth sampling methods, such as SuperPixel Sampler (SPS) [63] and Deep Adaptive Lidar (DAL) [8].

$NetM$ is initialized by RGB images according to FCN [66]. To show the effectiveness of the proposed $NetE$ and $NetM$ joint training method, we also compare with the sampling mask computed by the initialized $NetM$. The sampling method is denoted as FCN.

To illustrate $NetM$ can generalize across datasets, we train $NetM$ on the NYU-Depth-V2 dataset and test it on the KITTI dataset. The depth sampling method is noted as $NetM-NYU$ when testing on the KITTI dataset. Similarly, we train $NetM$ on the KITTI dataset and test it on the NYU-Depth-V2 dataset. The sampling method is noted as $NetM-KITTI$ when testing on the NYU-Depth-V2 dataset. Noted that $NetM$ is fully convolutional, so it is straightforward to test on different image resolution.

Random, Uniform Grid, Poisson, SPS [63], DAL [8], FCN [66] and proposed $NetM$ (including $NetM-KITTI$ and $NetM-NYU$) sampling methods are applied to the test images. Sampling rates $c=1\%,0.25\%$ and $0.0625\%$ are tested. For the depth estimation methods, DL-based methods $NetE$ [29], FusionNet [6], SSNet [36] and traditional method Colorization [31] are used to estimate the fully sampled depth map from the sampled depth map and RGB image. It’s noted that all the DL-based depth estimation methods are trained using random sampling masks and the same training set of either KITTI depth completion or NYU-Depth-V2 dataset.

For the KITTI depth completion dataset, the average Root Mean Square Error (RMSE) and Mean Absolute Error (MAE) over all $3426$ test frames are shown in Table I. First, under all three sampling rates, the proposed $NetM$ mask outperforms the random, Poisson, Uniform Grid, SPS, DAL and FCN masks consistently over all depth estimation methods in terms of RMSE and MAE. This demonstrates the effectiveness of our proposed adaptive depth sampling network. Furthermore, $NetM$ is jointly trained with $NetE$ and it still performs well with other depth estimation methods, demonstrating that it can generalize well to other depth estimation methods. The performance advantage of $NetM$ is not tied to any specific depth estimation method. Finally, it can be concluded that the smaller the sampling rate, the larger the advantage of $NetM$ compared to other sampling algorithms. This implies that $NetM$ is able to handle challenging depth sampling tasks (extremely low sampling rates).

The depth sampling and reconstruction performance on the NYU-Depth-V2 dataset are shown in Table II. Similar conclusions as the ones from the KITTI depth completion dataset can be drawn. It is noticed that SPS method is comparable to $NetM$. NYU-Depth-V2 is an indoor dataset with a maximum $10$m depth range. The variance of the depth map is small compared to the KITTI depth completion dataset, so the even spatial distribution prior in SPS works well here. Moreover, NYU-Depth-V2 dataset is captured by Microsoft Kinect. The ground truth depth maps have low spatial resolution ($240\times 320$) and are relative noisy. Thus some improvement on fine details in the estimated depth map is not reflected. Nevertheless, $NetM$ outperforms SPS when sampling rate is low and more advanced DL-based depth completion algorithms are applied.

According to Table I and Table II, FusionNet [6] has the best depth estimation accuracy under various of sampling masks. FusionNet extracts both global and local information and is more complex than $NetE$. Depth estimation from RGB image and sparse depth input is an active research topic. $NetM$ is able to generalize with other depth estimation methods than $NetE$, so it is able to benefit from the latest depth estimation algorithms.

The visual quality comparison of various sampling strategies and depth estimation methods is shown in Figure 5 and Figure 6. For sampling rate equal to $c=0.25\%$, we can observe the advantages of the proposed $NetM$ mask over all other sampling masks by comparing the resulting depth maps by the same depth estimation algorithm. In Figure 5, $NetM$ samples densely around the end of the road, trees and billboard, resulting in accurate depth estimation in such areas. Compared to other adaptive sampling algorithms, such as SPS and DAL, $NetM$ samples more densely on distance objects, making the estimated depth more accurate. SPS uses SLIC [64] to segment the RGB image and such segmentation can not obtain distance information from the RGB images. DAL estimates a smooth motion field to warp a regular sampling grid. When the scene is complicated, it is not flexible enough to warp a regular sampling grid to optimal location. In Figure 6 , $NetM$ samples the distant vertical structure, as well as the table and chair on the left side. More detailed depth map reconstruction in those areas is obtained. Compared to SPS, $NetM$ tends to sample uniformly on the wall regions, while still capture the shape of the foreground objects.

According to Table I, $NetM-NYU$ is the second best performer in the KITTI dataset. From Table II, in the NYU-Depth-V2 dataset, $NetM-KITTI$ outperforms all the other sampling methods except SPS and $NetM$. This demonstrates that the proposed $NetM$ is able to generalize across different datasets. We visualize the sampling location difference between $NetM-NYU$ and $NetM$ in the KITTI dataset in Figure 7. It can be found that $NetM$ samples densely on the upper part of the image compared to $NetM-NYU$. $NetM-NYU$ does not learn the prior knowledge that upper part of the image has larger depth values and should be sampled more densely from the indoor NYU-Depth-V2 dataset. However, $NetM-NYU$ is still able to sample on such objects as the bus, cyclist and poles. The sampling location difference between $NetM-KITTI$ and $NetM$ on the NYU-Depth-V2 dataset is shown in Figure 8. $NetM-KITTI$ learns the prior knowledge that upper part of the image is more important from the KITTI dataset and samples densely on the upper part of the image, resulting sub-optimal reconstruction accuracy compared to $NetM$.

$NetM$ is initialized with RGB images trained FCN [66] superpixel network using the SLIC loss (Equation 3). SPS [63] uses the SLIC superpixel technique to segment the RGB images. Sampling locations are determined by the weighted mass center of superpixels. Different superpixel segmentations result in different sampling quality. In Figure 9, we visualize the superpixel segmentation results and the derived sampling locations for SLIC, FCN and $NetM$ when $c=0.0625\%$. SLIC and FCN segment the input RGB image based on the color similarity and preserve spatial compactness. The segmentation density is spatially homogeneous. $NetM$ is jointly trained with $NetE$, thus it has knowledge of distance given the RGB input image. Distance objects in the image are sampled denser. It also segments sparsely the pavement and grass areas. Such near objects as cars are segmented denser compared to the pavement and grass areas. We also observe that $NetM$ segmentation does not preserve color pixel boundaries as well as FCN, which is expected as $NetM$ also minimizes the depth estimation loss besides the SLIC loss in Equation 8. According to FCN and $NetM$’s reconstruction accuracy in Table I and Table II, it can be found that $NetM$ always outperforms FCN. This demonstrates the effectiveness of the proposed $NetM$ training mechanism (Equation 8).

In Section III-D, we propose the use of SSA to make the sampling process differentiable during training. Such differentiable sampling approximation is necessary to jointly train $NetM$ with $NetE$. Compared to the $2\times 2$ bilinear kernel based differentable sampling in [8], the proposed SSA provides better sampling performance. In order to show the advantages of SSA, we replace the SSA sampling of $NetM$ by the bilinear kernel based sampling and perform the exact same training procedures. As demonstrated in Table III, the lower the sampling rate, the bigger the advantage of SSA over the bilinear kernel sampling. When sampling points are sparse, the gradients derived from a $2\times 2$ local window are too small to train $NetM$ effectively. We empirically found that the $5\times 5$ window size for SSA provides reasonable sampling performance under all sampling rates.

In Table I, FusionNet [6] achieves the best depth estimation performance under various of sampling masks. The proposed global and location information fusion is effective and the network size is considerably larger than $NetE$ [29]. Best depth sampling and estimation results are obtained using $NetM$ sampling and FusionNet depth estimation under all sampling rates. It is noted that $NetM$ is trained jointly with $NetE$ and FusionNet is trained using random masks. Similar to DAL, $NetM$ and FusionNet can also be optimized simultaneously. Starting from the $NetE$ trained $NetM$ and random mask trained FusionNet, we alternatively train $NetM$ and FusionNet and denote the trained networks by $NetM*$ and FusionNet*, respectively. The joint depth sampling and reconstruction results are shown in Table IV. We also compare with the sampling and reconstruction methods proposed in SPS and DAL. $NetM*$ with FusionNet* slightly outperforms $NetM$ with FusionNet and achieves the best accuracy. Utilizing random sampling masks during the training of depth estimation methods (FusionNet, SSNet, $NetE$) makes the methods robust to other sampling patterns in testing. We also found that $NetM$ trained using different depth estimation methods has similar sampling patterns. So simultaneously training the sampling and reconstruction networks improves the results slightly.

In Figure 10, we visually compare the end to end depth sampling and reconstruction results. In the $2$ test scenes, $NetM*$ with FusionNet* properly sample and reconstruct distant and thin objects, resulting in the best accuracy compared to other methods. With the developing depth estimation algorithms, we can integrate better depth estimation methods into our system. We show in Section IV-B that the performance advantages of $NetM$ can generalize well to other than $NetE$ depth estimation methods.

Apart from the superior adaptive depth sampling quality to other state-of-the-art methods, the proposed sampling method also has advantages in fast computation efficiency. In this section we evaluate the computation efficiency of different methods. The test images from both the KITTI depth completion dataset ($240\times 960$) and the NYU-Depth-V2 dataset ($240\times 360$) are used. Non DL-based methods, Poisson and SPS, are tested using one Intel i9-9820X CPU with 64GB memory. DL-based methods, DAL and $NetM$, are tested using the same CPU as well as one NVIDIA 2080Ti GPU with 11GB memory. We also measure the floating point operations (FLOPs) and number of parameters of both models. All methods are tested on the same test image set under 3 different sampling rates for 100 times and the average run time in milliseconds (ms) are reported in Table V. It’s noticed that comparing to DAL, $NetM$ has smaller model size and faster run time. Also $NetM$’s run time is almost constant under different sampling rates, different from the other three methods. Such fast computation efficiency property makes $NetM$ practical for a real time adaptive depth sensing system. $NetM$ is also faster than Poisson, SPS and DAL when running on the same CPU.

All the experiments above assume the RGB images and the sampled depth maps are captured at the exact same time instant. Due to the system response time, $NetM$ computation time, etc, there are temporal registration issue between the sampled depth maps and the RGB images in practice. To make the adaptive depth sampling method practical, it is important to understand how $NetM$’s sampling performance degrades with respect to the capture time delay $\Delta t$ (between the sampled depth map and RGB image). Noted that the Random, Uniform Grid and Poisson sampling masks are free from such temporal registration issue, because the sampling masks are independent from the RGB images. In this section, we performed 2 sets of experiments to simulate the temporal registration issues.

For the first set of experiments, with $0.25\%$ sampling rate on the KITTI dataset, during the depth sampling process at frame $t$, the sampling mask is computed using RGB image at frame $t-\Delta t$, to simulate the capture time difference directly. The KITTI dataset captures synchronized RGB and Depth data every $100$ms. We simulate such time delay $\Delta t$ from $0$ to $500$ms and use $NetE$ to estimate the dense depth maps. The MAE and RMSE error is shown in Figure 11. It can be found that $NetM$’s sampling performance is fairly stable when the capture time difference increases, up to $500$ms. One reason is that the far objects’ motion is small given the temporal perturbation. Another reason is that the structure of the scene is relatively static in consecutive frames.

For the second set of experiments, also with $0.25\%$ sampling rate on the KITTI dataset, additional perturbation is added on $NetM$ predicted sampling location. The perturbation is done by adding uniform distribution noise under different ranges to the sampling location. As shown in Figure 12, it can be found that under the MAE and RMSE metrics, even with $+/-15$ pixels perturbation ($240\times 960$ full image resolution) on the sampling location, $NetM$ still outperforms Random, Uniform Grid and Poisson sampling masks (no perturbation added), under the same depth estimation method $NetE$. Such large pixel perturbation serves as a challenging test case because far objects’ motion can not reach this level in practice.

According to the above 2 sets of experiments, the sampling performance advantage still holds if we take the capture time difference into consideration. The difficulty in sampling fast moving objects is one of the limitations of the proposed adaptive sampling approach. Such techniques as motion prediction can be applied to compensate the temporal registration issue. They are beyond the scope of this paper.