Self-optimizing loop sifting and majorization for 3D reconstruction

Due to rapid development in autonomous vehicles and consumer robots, there is an increasing need for precise 3D maps for route and action planning and navigation. Among 3D mapping methods, visual simultaneous localization and mapping (vSLAM) and 3D reconstruction methods are very promising because they can map large-scale environments such as cities and indoor environments without the need for much human effort involved.

vSLAM and 3D reconstruction methods have gone through impressive progress. In camera tracking, there are different methods, such as sparse keypoint point-based methods [1, 2, 3, 4, 5], direct methods [6, 7], and dense surface-based methods [8, 9]. Additionally, IMU are added to methods [10, 11, 12, 13, 14, 15, 10] to make tracking more accurate. Even though camera tracking algorithms have good performance and low drift, the build-up error can still not be ignored [16]. To solve this problem, loop closure detection [17, 18] and optimization [19] are often leveraged to counter the problem, and it has provided plenty of improvements. However, the problem is not fully solved yet. Intuitively, the more loops in data, the more information to recover more precise camera trajectories and 3D models. But, in practice, when running existing vSLAM systems on datasets with loopy motions, mismatches can always be found in the final results. This means that loops are not successfully detected and utilized.

vSLAM systems tend to add the loops very conservatively to reduce the severe influence of the false loops [16]. These conservative checks are the result of the non-perfect precision performance of loop detection methods. There are high chances that detected loops are incorrect ones.

To solve this challenging problem, we propose an algorithm that can sift and majorize loop detections so that only correct and essential loops are fed into the following optimization steps. The proposed method highly couples with the dense map posterior (DMP) metric [20] that can evaluate 3D reconstruction performance without ground truth measurement. Our proposed algorithm can compare the usefulness and effectiveness of different loops and ultimately sifts out false and unimportant loops. To the best of our knowledge, the contributions of the proposed algorithm are:

1. 1.

   The proposed algorithm can sift loop detections based on their impact on loop optimization results.

2. 2.

   It is the first algorithm that can marjorize loop detection only to keep the important ones while ignoring the less relevant ones.

3. 3.

   Experiments on public datasets show it outperforms state-of-the-art methods.

To avoid the severe consequence of optimizing with false loops, vSLAM and 3D mapping systems tend to add the loops very conservatively. ORB-SLAM2 [1] requires the presence of several consistent loops in consecutive keyframes to accept them, where at least one keyframe must be shared in order to be classified as consistent. With this consistency check, ORB-SLAM2 merely takes false loops into optimization but at the price that plenty of correct loops are rejected. ElasticFusion [9] evaluates several characteristics before taking a loop detection into optimization pipelines, including deformation cost and final state of the Gauss-Newton system. Even after all the evaluations, a good loop is often rejected, and not rare to see that an incorrect loop is accepted. BundleFusion [4] filters loop correspondences with cascade checks including local geometric and photometric consistency checks and check on correspondence residual after optimization. The local depth discrepancy check shares a small similarity with our work. However, the check is limited to a very local region with a downsampled depth resolution together with a user-specified threshold. Thus it is not informed about the effect of loop data correspondences impacting a full 3D model. A requirement on a user parameter also makes it ineffective and less adaptive. [2] do this by pruning edges after optimization based on the discrepancy between the individual transformation estimates before and after optimization. We share the idea of observing optimization consequences brought in with a loop, but their impact is measured on a sparse graph while ours is observed on a full 3D dense model.

Another approach to solving the problem is to treat false loops as outlier data and decrease their impact on the optimization [21, 22, 23, 24]. They work well in some cases, but the dependence on initial conditions and the ratio of outliers makes them prune to failures. Choi et al. further develop this idea into an algorithm that is highly coupled with the dense 3D reconstruction problem by specifying both pose graph construction and least square information calculation [25]. This method is very effective when a desired camera scan pattern is followed but it requires keeping surface within camera range all the time thus limiting its flexibility. It also suffers dependence on initial condition and outlier ratio.

Due to the difficulty of balancing precision and recall of loop detections, SUN3D [26] turns to a human-in-the-loop approach by labeling objects in scenes and connect the same objects across frames. This method performs very well in terms of loop precision and recall, but it requires too much effort in labeling; thus is not practical to process data on a large scale.

To solve the loop sifting problem, we propose an algorithm specified in Algorithm 1. In the algorithm, a given set of loop detections is denoted as $\mathbb{O}$ among which each individual one is denoted as $O$. The supporting optimization pose graph is denoted as $\mathcal{G}$. The sensor (e.g. camera and LiDAR) data are denoted as $\mathbb{Z}$.

There are two parts in the algorithm. In the first part, all the loops are tested and evaluated individually on the given initial pose graph. This step first runs optimization with a single loop and then fuses a model with the optimized results $\mathbb{T}^{i}$. Then a DMP value $r$ is evaluated for the fused 3D model $M$. This means that it tests each loop and sees how much improvement it provides by itself. Finally, all these loops are ranked by the calculated DMP value $r$ in ascending order (more effective $\rightarrow$ less effective $\rightarrow$ negative impacts).

In the second part, all the loops are tested and evaluated one more time, but in a way that is different from the first time. In this part, the loops are tested in sorted order: the ones that provide more improvements are tested first. When a loop can provide performance improvement on the previous result, it will be added to an accepted set, thus will also impact consequent loop tests. In this way, loops are accepted when they can provide performance improvement on the current status. The first accepted one should make an improvement to the original results from tracking.

The proposed method is general and agnostic to loop detection and optimization methods. Neither does it require a specific type of vSLAM system. For our experiments, we choose several well know implementations.

Extensive experiments are performed to evaluate our proposed method on two datasets: augmented ICL-NUIM [25] and SUN3D dataset [26]. SMD is short for surface mean distance.

In this work, an algorithm that can sift and majorize loop detections is proposed. The algorithm tests and decides the acceptance of each loop without a single user-defined threshold. Experiments are conducted on public datasets, including the Augmented ICL-NUIM dataset and the SUN3D dataset. Results show that the proposed method outperforms the state-of-the-art method. It can find key loops with $100\%$ precision and eliminate significant mismatches when processing SUN3D sequences.