Depth360: Self-supervised Learning for Monocular Depth Estimation using Learnable Camera Distortion Model

Fig. 2[a] presents the calculation of depth estimation for real images. Since there are no GT depth images, we employed self-supervised learning approach using time-sequence images. Unlike previous approaches[3, 25], our spherical camera can capture both the front- and back-side of the robot. Hence, we propose a cost function to blend the front- and back-side images and thereby reduce the negative effects of occlusion.

We feed the front-side images $I_{A^{f}}$ and back-side image $I_{A^{b}}$ at robot pose $A$ into the depth network $f_{\mbox{depth}}()$ to estimate the corresponding depth images as $D_{A^{f}},D_{A^{b}}=f_{\mbox{depth}}(I_{A^{f}},I_{A^{b}})$. By back-projection $f_{\mbox{backproj}}()$ with our proposed camera model, we can obtain the corresponding point clouds $Q_{A^{f}}$ and $Q_{A^{b}}$ for each camera coordinate $\Sigma_{A^{f}}$ and $\Sigma_{A^{b}}$, respectively.

$\displaystyle Q_{A^{f}}=f_{\mbox{backproj}}(D_{A^{f}}),\hskip 5.69054ptQ_{A^{b}}=f_{\mbox{backproj}}(D_{A^{b}})$

(2)

Opposed to the baseline methods, our method predicts both the front- and back-side images from a single side image to blend them in $J_{bimg}$. Hence, “Trans.” in Fig. 1[a] transforms the coordinates of the estimated point clouds as follows:

	$\displaystyle Q^{B^{f}}_{A^{f}}=T_{AB}Q_{A^{f}},\hskip 22.76219ptQ^{B^{f}}_{A^{b}}=T_{AB}\cdot T^{-1}_{fb}Q_{A^{b}},$		(3)
	$\displaystyle Q^{B^{b}}_{A^{f}}=T_{fb}\cdot T_{AB}Q_{A^{f}},\hskip 2.84526ptQ^{B^{b}}_{A^{b}}=T_{fb}\cdot T_{AB}\cdot T^{-1}_{fb}Q_{A^{b}}.$

Here, $Q^{Y^{\beta}}_{X^{\alpha}}$ denotes the point clouds on the coordinate $\Sigma_{Y^{\beta}}$ estimated from image $I_{X^{\alpha}}$. $T_{AB}$ is the estimated transformation matrix between $\Sigma_{A^{f}}$ and $\Sigma_{B^{f}}$. $T_{fb}$ is the known transformation matrix between $\Sigma_{X^{f}}$ and $\Sigma_{X^{b}}$. By projecting these point clouds with our learnable camera model, we can estimate four matrices $M^{B^{\beta}}_{A^{\alpha}}$,

$\displaystyle M^{B^{\beta}}_{A^{\alpha}}=f_{\mbox{proj}}(Q^{B^{\beta}}_{A^{\alpha}})$

(4)

where, $\alpha=\{f,b\}$ and $\beta=\{f,b\}$. According to [24], we estimate $I_{A^{\alpha}}$ by sampling the pixel value of $I_{B^{\beta}}$ as $\hat{I}^{B^{\beta}}_{A^{\alpha}}=f_{\mbox{sample}}(M^{B^{\beta}}_{A^{\alpha}},I_{B^{\beta}})$. Here $\hat{I}^{B^{\beta}}_{A^{\alpha}}$ denotes the estimated image of $I_{A^{\alpha}}$ by sampling $I_{B^{\beta}}$. Note that we estimated four images from the combination of $\alpha=\{f,b\}$ and $\beta=\{f,b\}$, as shown in Fig. 2[a]. We calculate the blended image loss $J_{bimg}$ to penalize the model during training.

	$\displaystyle J_{bimg}=\lambda_{1}J_{L1}+\lambda_{2}J_{SSIM}.$		(5)
	$\displaystyle J_{L1}=\sum_{\alpha\in\{f,b\}}f_{\mbox{min}}(M|I_{A^{\alpha}}-\hat{I}^{B^{f}}_{A^{\alpha}}|,M|I_{A^{x}}-\hat{I}^{B^{b}}_{A^{\alpha}}|),$
	$\displaystyle J_{SSIM}=\sum_{\alpha\in\{f,b\}}f_{\mbox{min}}(Md_{\mbox{ssim}}(I_{A^{\alpha}},\hat{I}^{B^{f}}_{A^{\alpha}}),Md_{\mbox{ssim}}(I_{A^{\alpha}},\hat{I}^{B^{b}}_{A^{\alpha}})).$

Here, $f_{\mbox{min}}(\cdot,\cdot)$ selects a smaller value at each pixel and calculates the mean of all the pixels. $d_{\mbox{ssim}}(\cdot,\cdot)$ is the pixel-wise structural similarity (SSIM) [37], following [25]. By selecting a smaller value in $f_{\mbox{min}}(\cdot,\cdot)$, we can equivalently select the non-occluded pixel value of $\hat{I}^{B^{f}}_{A^{\alpha}}$ or $\hat{I}^{B^{b}}_{A^{\alpha}}$ to calculate L1 and SSIM [25]. $M$ is a mask to remove the pixels without RGB values and those of the robot itself, which are depicted in gray color in Fig. 1[a].

Fig. 2[c] denotes the process to estimate $T_{AB}$. Unlike previous monocular depth estimation approaches, we use the GT transformation matrix $\bar{T}_{AB}$ from the integral of the reference velocities $\{v_{i},\omega_{i}\}_{i=0\cdots N_{g-1}}$ to move between poses $A$ and $B$. $J_{pose}$ is designed as $J_{pose}=\sum(\bar{T}_{AB}-T_{AB})^{2}$.

The mapping between $[x_{i},y_{i}]$ and $[u_{i},v_{i}]$ is defined by four independent parameters: $r_{x},r_{y}$ for the FoV, and $o_{x},o_{y}$ for the offset between $UV$ and $XY$. Assuming linear relationships, this part of $f_{\mbox{backproj}}()$ is written as follows,

$\displaystyle[x_{i},y_{i}]^{T}=R\cdot[u_{i},v_{i}]^{T}+[o_{x},o_{y}]^{T},$

(6)

where $R=\mbox{diag}(1/r_{x},1/r_{y})$. Owing to the linearity and regularity, defining the inverse transformation for $f_{\mbox{proj}}()$ as $R^{-1}\cdot([x_{i},y_{i}]^{T}-[o_{x},o_{y}]^{T})$ is straightforward.

Next, we show the process of the green double arrow between $[x_{i},y_{i}]$ and $[X_{i},Y_{i},Z_{i}]$ in Fig. 4 to model the distortion. At first, we design the learnable projection surface (grey color lines in Fig. 4). Then, we explain the computation procedure in $f_{\mbox{backproj}}()$ and $f_{\mbox{proj}}()$.

The robot footprint is almost horizontally flat, and its height is obtained from the camera mounting position. According to the method shown below, we obtain the GT depth $\{\bar{D}_{A^{\alpha}}\}_{\alpha=\{f,b\}}$ only around the robot footprint between $\pm M_{r}$ steps and provide the supervision as follows:

$\displaystyle J_{fcl}=\sum_{\alpha\in\{f,b\}}\frac{1}{N_{m}}\sum_{i=1}^{N_{m}}M_{f}|\bar{D}_{A^{\alpha}}-D_{A^{\alpha}}|,$

(11)

where $M_{f}$ is the mask, which masks out the pixels without the GT values in $\bar{D}_{A^{\alpha}}$, and $N_{m}$ is the number of pixel with the GT depth. $\bar{D}_{A^{\alpha}}$ can be derived as:

$\displaystyle\bar{D}_{A^{\alpha}}(x^{\alpha}_{j},y^{\alpha}_{j})=Z^{\alpha}_{j},$

(12)

where $[x^{\alpha}_{j},y^{\alpha}_{j}]=f_{\mbox{proj}}(T_{r\alpha}Q_{foot}[j])$ and $Z^{\alpha}_{j}$ are the values of the $Z$ axis of $T_{r\alpha}Q_{foot}[j]$. Here, $Q_{foot}$ is the point clouds of the robot footprint between $\pm M_{r}$ steps on $\Sigma_{X^{r}}$. To obtain $Q_{foot}$, we calculate the robot local positions using the teleoperator’s velocity between $\pm M_{r}$ steps. And, $T_{r\alpha}$ is the known transformation matrix from $\Sigma_{X^{r}}$ to $\Sigma_{X^{\alpha}}$. By assigning all point clouds into $\bar{D}_{A^{\alpha}}$ using (12), we can take $\bar{D}_{A^{\alpha}}$ to calculate $J_{fcl}$. Note that $\bar{D}_{A^{\alpha}}$ is a sparse matrix. No GT pixel in $\bar{D}_{A^{\alpha}}$ is excluded by $M_{f}$.

In indoor scenes, floor areas often reflect ceiling lights. This can cause it to appear that there are holes on the floor in the estimated depth image. To provide a lower boundary for the height of estimated point clouds, we observe two key points: 1) the camera is horizontally mounted on the robot, and 2) most objects around the robot are higher than the floor. One exception would be anything that is downstairs, which would be lower than the floor. However, it is rare in to find such occurrences in the dataset, because teleoperation around the stairs is dangerous and ill-advised.

To provide the constraint for the $Y$ axis ($=$ hight) value of the estimated point clouds, $J_{lbl}$ can be given as follows:

$\displaystyle J_{lbl}=\sum_{\alpha\in\{f,b\}}\frac{1}{N}\sum_{i=1}^{N}(\mbox{max}(0.0,Y_{A^{\alpha}}-h_{cam})),$

(13)

where $Q_{A^{\alpha}}=[X_{A^{\alpha}},Y_{A^{\alpha}},Z_{A^{\alpha}}]$. $J_{lbl}$ penalizes $Y_{A^{\alpha}}$, which is larger ($=$ lower) than the floor height ($=h_{cam}$).

We used the GS dataset [9, 10, 11] with time-sequence spherical camera images (256$\times$128) and the reference velocities, which were collected by teleoperating turtlebot2 with Ricoh THETA S. The GS dataset contains 10.3 hours of data from twelve buildings at the Stanford University campus. To train our networks, we used a training dataset from eight buildings, following [11].

In addition to the real images of the GS dataset, we collected pairs of simulator images and GT depth images for $J_{depth}$. We scanned 12 floors (e.g., office rooms, meeting rooms, laboratories) in our company buildings and ten environments (e.g., a traditional Japanese house, art gallery, fitness gym) for the simulator by Matterport pro2, as shown in Fig. 6. We separated them into groups: eight floors in our company buildings for training, four floors in our company buildings for validation, and ten environments not in our company for testing. For training, we rendered 10,000 data from each environment using interactive Gibson [41, 42]. In addition, we collected 1,000 data points for testing from ten environments.

Data collection of the GT depth from a real spherical image is a challenging task. Hence, in the quantitative analysis, we used the GT depth from the simulator. To evaluate the generalization performance, our test environment was not derived from our company building. Examples of the domain gaps between training and testing are shown in Fig. 6. In the qualitative analysis, we used both real and simulated images.

To evaluate our proposed camera model against the baseline methods, we employed the KITTI raw dataset [12] for the evaluation of depth estimation. Similar to the baseline methods, we separated the KITTI raw dataset via Eigen split [45] with 40,000 images for training, 4,000 images for validation, and 697 images for testing. To compare with baseline methods, we employed the widely used 640$\times$192 image size as input.

Besides, we used the KITTI odometry dataset with ground truth pose for the evaluation of pose estimation. It is known that the KITTI raw dataset for depth estimation partially includes test images from the KITTI odometry dataset. Hence, following the baseline methods, we trained our models with sequences 00 to 08 and conduct testing on sequences 09 and 10.

Table I shows the ablation study of our method and the results of three baseline methods for comparison. We trained the following baseline methods with the same dataset.

In quantitative analysis, we evaluate the estimated depth using common metrics. “Abs-Rel,” “Sq-Rel,” and “RMSE,” are calculated by means of the following values.

• •

  Abs-Rel        :   $|D_{gt}-\hat{D}|/D_{gt}$

• •

  Sq Rel          :   $(D_{gt}-\hat{D})^{2}/D_{gt}$

• •

  RMSE          :   $((D_{gt}-\hat{D})^{2})^{\frac{1}{2}}$

Here, $D_{gt}$ is the ground truth of the estimated depth image $\hat{D}$. The remaining metric is the ratios that satisfy $\delta<1.25$. $\delta$ is defined as $\delta=\mbox{max}(D_{gt}/\hat{D},\hat{D}/D_{gt})$.

From Table I, we can observe that our method significantly outperforms all baseline methods. In addition, we confirmed the advantages of our proposed components via the ablation study. The use of the GT depth from the simulator and the learning axisymmetric camera model were fairly effective. Even though the proposed method is evaluated without scaling using the GT depth, it outperforms the other methods with scaling. This suggests that our method learns the correct scaling via $J_{depth}$ with the GT depth from the simulator.

In Table II, we present the quantitative results for the KITTI dataset. Similar to our method, the baseline methods (shown in Table II) learned the camera model. All methods presented in Table II used ResNet-18 for their depth network to allow for fair comparisons.

In our method with known camera parameters, we set $\{h_{q}\}_{q=0\cdots N_{b}}$, $r_{x}$, $r_{y}$, $o_{x}$, and $o_{y}$ as the constant values for the GT camera’s intrinsic parameters. $\{b_{q}\}_{q=0\cdots N_{b}}$ was designed with equal intervals between 0.0 and 1.0. Our method improved the accuracy of depth estimation by learning the camera model. In addition, our method outperformed all baseline methods, including the original monodepth2.

Moreover, we trained our models by mixing the KITTI, Cityscape [47], bike [5], and GS datasets [11] to evaluate its ability to handle various cameras and evaluated on KITTI images. During training, all images were aligned into KITTI’s image size by center cropping. In GS dataset, we use front-side fisheye images. Our method showed improved performance by adding datasets from various cameras with various distortions, as shown at the bottom of Table II.

Figure 7 shows the estimated depth images from simulated images and real images from the GS dataset. The depth images estimated by monodepth2 are blurred. This is caused by the small size of the input image and the camera model’s error. The GT depth from the simulator can sharpen the simulated depth of images in monodepth2 with sim. GT. However, there are many artifacts, particularly on the reflected floor. Alhashim et al. observed errors in the estimated depth from real images. Alhashim et al. failed sim2real transfer because the depth network was trained only from simulated images. However, our method (rightmost side) can accurately estimate depth images of both real and simulated images by reducing these artifacts. Additional examples are provided in the supplemental videos.

Finally, we present the depth images of KITTI in Fig. 1 [b]. Our method can handle the pinhole camera image and estimate an accurate depth image without camera calibration.