Three-Filters-to-Normal: An Accurate and Ultrafast Surface Normal Estimator

Real-time 3-dimensional (3D) object recognition is a very challenging computer vision task [3]. Surface normal is an informative and important visual feature used in 3D object recognition [4]. However, not much research has been conducted thoroughly on surface normal estimation, as it is merely considered as an auxiliary functionality for other computer vision applications. Such applications are generally required to perform in an online fashion, and therefore, surface normal estimation must be carried out extremely fast [4].

The surface normals can be estimated from either a 3D point cloud or a depth/disparity image (see Fig. Three-Filters-to-Normal: An Accurate and Ultrafast Surface Normal Estimator). The former, such as a LiDAR point cloud, is generally unstructured. Estimating surface normals from unstructured range data usually requires the generation of an undirected graph [4], e.g., a $k$-nearest neighbor graph or a Delaunay tessellation graph. However, the generation of such graphs is very computationally intensive. Therefore, in recent years, many researchers have been focusing on surface normal estimation from structured range sensor data, e.g., depth/disparity images.

Existing surface normal estimators (SNEs) can be categorized as either geometry-based [3, 4, 5, 6] or machine/deep learning-based [7, 8]. The former typically computes surface normals by fitting planar or curved surfaces to locally selected 3D point sets, using statistical analysis or optimization techniques, e.g., singular value decomposition (SVD) or principal component analysis (PCA) [4]. On the other hand, the latter generally utilizes data-driven classification/regression models, e.g., convolutional neural networks (CNNs) to infer surface normal information from RGB or depth images [9].

In recent years, with rapid advances in machine/deep learning, many researchers have resorted to deep convolutional neural networks (DCNNs) for surface normal estimation. For instance, Xu et al.[10] utilized a so-called prediction-and-distillation network (PAD-Net) to 1) realize monocular depth prediction and surface normal inference, as well as 2) perform scene parsing and contour detection simultaneously. Recently, Huang et al.[11] formulated the problem of densely estimating local 3D canonical frames from a single RGB image as a joint estimation of surface normals, canonical tangent directions and projected tangent directions. Such problem was then addressed by a DCNN.

The existing data-driven SNEs are generally trained using supervised learning techniques. Hence, they require a large amount of hand-labeled training data to find the best CNN parameters [8]. Additionally, such CNNs were not specifically designed for surface normal estimation, because SNEs were only used as an auxiliary functionality for other computer vision applications, such as scene parsing, 3D object detection, and depth perception. Furthermore, many robotics and computer vision applications, e.g., autonomous driving [12], require very fast surface normal estimation (in milliseconds). Unfortunately, the existing machine/deep learning-based SNEs are not fast enough. Moreover, the accuracy achieved by data-driven SNEs is still far from satisfactory (the average proportion of good pixels is usually lower than $80\%$) [7, 8]. Most importantly, it can be considered more reasonable to estimate surface normals from point clouds or disparity/depth images rather than from RGB images. Hence, there is a strong motivation to develop a lightweight SNE for structured range data with high accuracy and speed.

The major contributions of this work are as follows:

1. 1.

   Three-filter-to-normal (3F2N), an accurate and ultrafast SNE. We published its Matlab, C++ and CUDA implementations at github.com/ruirangerfan/three_filters_to_normal. Compared with other geometry-based SNEs, 3F2N SNE greatly improves the trade-off between speed and accuracy.

2. 2.

   Three datasets (easy, medium and hard) created using 24 3D mesh models. Each mesh model is used to generate 1800–2500 depth images from different views. The corresponding surface normal ground truth is also provided, as 3D mesh object models (rather than the objects themselves) are available for surface normal ground truth generation.

This section provides an overview of geometry-based SNEs.

1) PlaneSVD SNE [13]: The simplest way to estimate the surface normal of an observed 3D point $\mathbf{p}_{i}=[x,y,z]^{\top}$ in the camera coordinate system (CCS) is to fit a local plane:

to the points in $\mathbf{Q}_{i}^{+}=[\mathbf{Q}_{i}^{\top},\mathbf{p}_{i}]^{\top}$, where $\mathbf{Q}_{i}=[{\mathbf{q}_{i}}_{1},\dots,{\mathbf{q}_{i}}_{k}]^{\top}$ (${\mathbf{q}_{i}}_{j}\neq\mathbf{p}_{i}$) is a set of $k$ neighboring points of $\mathbf{p}_{i}$. The surface normal $\mathbf{n}_{i}=[n_{x},n_{y},n_{z}]^{\top}$ can be estimated by solving:

where $\mathbf{b}_{i}=[\mathbf{n}_{i}^{\top},b]^{\top}$ and $\mathbf{1}_{m}$ is an $m$-entry vector of ones. (1) can be solved by factorizing $\mathbf{Q}_{i}^{+}$ into $\mathbf{U}\mathbf{\Sigma}\mathbf{V}^{\top}$ using SVD. $\hat{\mathbf{b}}_{i}$ (the optimum $\mathbf{b}_{i}$) is a column vector in $\mathbf{V}$ corresponding to the smallest singular value in $\mathbf{\Sigma}$ [4].

2) PlanePCA SNE [14]: $\mathbf{n}_{i}$ can also be estimated by removing the empirical mean $\bar{\mathbf{q}}_{i}=\frac{1}{k+1}(\mathbf{p}_{i}+\Sigma_{j=1}^{k}{\mathbf{q}_{i}}_{j})$ from $\mathbf{Q}^{+}_{i}$ and rearranging (2) as follows [15]:

where $\bar{\mathbf{Q}}_{i}^{+}=\mathbf{1}_{k+1}\bar{\mathbf{q}}_{i}^{\top}$. Minimizing (3) is equivalent to performing PCA on $\mathbf{Q}^{+}_{i}$ and selecting the principal component with the smallest covariance [4].

3) VectorSVD SNE [4]: A straightforward alternative to fitting (1) to $\mathbf{Q}^{+}_{i}$ is to minimize the sum of the inner dot products between ${\mathbf{r}_{i}}_{j}={\mathbf{q}_{i}}_{j}-\mathbf{p}_{i}$ and $\mathbf{n}_{i}$, namely,

This minimization is done by SVD.

4) AreaWeighted SNE [4]: A triangle can be formed by a given pair of ${\mathbf{r}_{i}}_{j}$ and ${\mathbf{r}_{i}}_{j+1}$, as defined above. A general expression of averaging-based SNEs is as follows [4]:

where $w_{j}$ is a weight and ${\mathbf{r}_{i}}_{k+1}={\mathbf{r}_{i}}_{1}$. In AreaWeighted SNE, the surface normal of each triangle is weighted by the magnitude of its area:

5) AngleWeighted SNE [4]: The weight $w_{j}$ of each triangle relates to the angle between ${\mathbf{r}_{i}}_{j}$ and ${\mathbf{r}_{i}}_{j+1}$:

where $\langle\cdot\rangle$ is a dot product operator.

6) FALS SNE [5]: The relationship between the Cartesian coordinate system and the spherical coordinate system (SCS) is as follows [5]:

where $r_{i}\geq 0$, $\theta_{i}\in(-\pi,\pi]$ and $\phi_{i}\in(-\frac{\pi}{2},\frac{\pi}{2}]$. Since all points in $\mathbf{Q}_{i}^{+}$ are in a small neighborhood [5], their $r_{i}$ are considered to be identical in FALS SNE. (2) and (8) result in:

where $\mathbf{V}_{i}^{+}=[\mathbf{v}_{i},{\mathbf{v}_{i}}_{1},\dots,{\mathbf{v}_{i}}_{k}]^{\top}$, $\tilde{\mathbf{n}}_{i}=\mathbf{n}_{i}/b^{2}$ and $\mathbf{s}_{i}=[{{r^{-1}_{i}}},{{{r_{i}}_{1}^{-1}}},\dots,{{{r_{i}}_{k}^{-1}}}]^{\top}$.

7) SRI SNE [5]: Similar to FALS SNE, SRI SNE first transforms the range data from the Cartesian coordinate system to the SCS. $\mathbf{n}_{i}$ is then obtained by computing the partial derivative of the local tangential surface $s$:

where $\mathbf{R}_{i}$ is an SO(3) matrix with respect to $\theta_{i}$ and $\phi_{i}$. $\mathbf{e}_{z}$, $\mathbf{e}_{x}$ and $\mathbf{e}_{y}$ are the unit vectors in the $z$, $x$ and $y$ coordinate axes, respectively. $\nabla s(\theta_{i},\phi_{i})$ can be obtained by applying standard image convolutional kernels.

8) LINE-MOD SNE [3]: Firstly, the optimal gradient $\nabla z=[{\partial z}/{\partial u},{\partial z}/{\partial v}]^{\top}$ of a depth map is computed. Then, a 3D plane is formed by three points $\mathbf{p}_{0}$, $\mathbf{p}_{1}$ and $\mathbf{p}_{2}$:

where $\mathbf{t}(\tilde{\mathbf{p}}_{i})$ is the vector along the line of sight that goes through an image pixel $\tilde{\mathbf{p}}_{i}=[u_{i},v_{i}]^{\top}$ and is computed using camera intrinsic parameters. The surface normal $\mathbf{n}_{i}$ can be computed using:

In this paper, we introduce 3F2N SNE, which is simple to understand and use. Our SNE can compute surface normals from structured range sensor data using only three filters: 1) a horizontal image gradient filter, 2) a vertical image gradient filter and 3) a mean/median filter.

A 3D point $\mathbf{p}_{i}=[x,y,z]^{\top}$ in the CCS can be transformed to $\tilde{\mathbf{p}}_{i}=[u,v]^{\top}$ using [16]:

where $\mathbf{K}$ is the camera intrinsic matrix, $\mathbf{p}_{\text{o}}=[u_{\text{o}},v_{\text{o}}]^{\top}$ is the image principal point, and $f_{x}$ and $f_{y}$ are the camera focal lengths (in pixels) in the $x$ and $y$ directions, respectively. Combining (1) and (13) results in:

Differentiating (14) with respect to $u$ and $v$ leads to:

which can be approximated by respectively performing horizontal and vertical image gradient filters, e.g., Sobel, Scharr and Prewitt, on the inverse depth image (an image storing the values of $1/z$). Rearranging (15) results in the following expressions of $n_{x}$ and $n_{y}$:

Given an arbitrary ${\mathbf{q}_{i}}_{j}\in\mathbf{Q}_{i}$, we can compute the corresponding ${n_{z}}_{j}$ by plugging (16) into (1):

where ${\mathbf{r}_{i}}_{j}={\mathbf{q}_{i}}_{j}-{\mathbf{p}_{i}}=[\Delta{x_{i}}_{j},\Delta{y_{i}}_{j},\Delta{z_{i}}_{j}]^{\top}$. Since (16) and (17) have a common factor of $-b$, they can be simplified as:

where $\Phi\{\cdot\}$ represents the manner of $\hat{n}_{z}$ (the optimum $n_{z}$) estimation. In our previous work [12], $\mathbf{n}_{i}$ is written in spherical coordinates and $\Phi(\cdot)$ is formulated as an energy minimization problem [12], which is computationally intensive. Hence, in this paper, $\Phi\{\cdot\}$ represents a mean/median filtering operation used to estimate $n_{z}$. Please note: if the depth value of $\mathbf{p}_{i}$ is identical to those of all its neighboring points ${\mathbf{q}_{i}}_{j}\in\mathbf{Q}_{i}$, we consider that the direction of its corresponding surface normal is perpendicular to the image plane and simply set $\mathbf{n}_{i}$ to $[0,0,-1]^{\top}$. The performances of estimating $\mathbf{n}_{i}$ using the mean filter and using the median filter will be compared in Section IV.

Specifically, for a stereo camera, $f_{x}=f_{y}=f$, and the relationship between depth $z$ and disparity $d$ is as follows [18]:

where $t_{c}$ is the stereo rig baseline. Therefore,

Plugging (19) and (20) into (18) results in:

Therefore, our SNE can also estimate surface normals from a disparity image using three filters.

An SNE can be applied in a variety of computer vision and robotics tasks. In this paper, we perform ElasticFusion [20], a real-time dense visual simultaneous localization and mapping (SLAM) algorithm, on the ICL-NUIM RGB-D dataset [21] with and without surface normal information incorporated, respectively. According to the quantitative analysis of our experimental results, the 3D geometry reconstruction accuracy can be improved by approximately 19%, when using the surface normal information obtained by 3F2N SNE. Examples of the experimental results are given in the supplement.

Moreover, we have also proven in [12] and [22] that surface normal information can be employed for various planar surface segmentation applications. Therefore, we believe that 3F2N SNE can be utilized to extract informative features for CNNs in various autonomous driving perception tasks, without affecting their training/prediction speed.

Finally, it is emphasized that the proposed SNE is essentially different from the approaches developed for dominant surface normal estimation (or Manhattan frame model inference [23]). The latter aims at estimating the surface normal of each planar surface instead of the one of every single pixel.

In this paper, we presented a precise and ultrafast SNE named 3F2N for structured range data. Our proposed SNE can compute surface normals from an inverse depth image or a disparity image using three filters, namely, a horizontal image gradient filter, a vertical image gradient filter and a mean/median filter. To evaluate the performance of our proposed SNE, we created three datasets (containing about 60k pairs of depth images and the corresponding surface normal ground truth) using 24 3D mesh models. Our datasets are also publicly available for research purposes. According to our experimental results, FD outperforms other image gradient filters, e.g., Sobel, Scharr and Prewitt, in terms of both precision and speed. FD-Median SNE achieves the best surface normal precision ($1.66^{\circ}$, $5.69^{\circ}$ and $15.31^{\circ}$ on easy, medium and hard datasets, respectively), while FD-Mean SNE is most effective for minimizing the trade-off between speed and accuracy. Furthermore, our proposed 3F2N SNE achieves better overall performance than all other geometry-based SNEs.