Uncertainty-Driven Dense Two-View
Structure from Motion

Reliability of per-pixel matching from $\mathbf{I}^{r}$ to the corresponding pixel in $\mathbf{I}^{s}$ is the key to dense two-view SfM problem. Yet, relative camera pose estimation needs a few accurate pixel matches. Since there is no trivial way to identify the correct per-pixel correspondences directly, it motivates us to explore a neural-network-based data-driven solution to reason about the fidelity of the predicted flow, so that we can a priori decide the suitability of the estimated correspondence. To this end, we exploit the dense correspondence method PDC-Net [15] as our optical flow backbone. Using this backbone model, we compute dense matches relating to two frames and the pixel-wise uncertainty map representing the correspondences’ reliability. As outlined in Sec.III-B, such an idea can provide a better camera pose estimate than popular methods [39][40]. We introduce the associated notations with a brief recap of the PDC-Net as follows.

In the paper, we use the term forward flow and backward flow denoted as $\mathbf{Y}_{r\rightarrow s}\in\mathbb{R}^{H\times W\times 2}$ and $\mathbf{Y}_{s\rightarrow r}\in\mathbb{R}^{H\times W\times 2}$, for the optical flow estimated from $\mathbf{I}^{r}$ to $\mathbf{I}^{s}$ and vice-versa. We represent $\mathbf{Y}_{r\rightarrow s}=F_{r\rightarrow s}(\mathbf{X};\theta)$, $\mathbf{Y}_{s\rightarrow r}=F_{s\rightarrow r}(\mathbf{X};\theta)$, where $F(\mathbf{X};\theta)$ symbolizes the neural network parameterized by $\theta$. For notation convenience, we use the symbol $\mathbf{Y}$ for denoting optical flow in general and will explicitly denote the forward-backward flow symbol if required.

Given $\mathbf{X}$, instead of predicting the single flow vector values for each pixel, the goal is to predict the conditional probability density of the optical flow as $p(\mathbf{Y}|\mathbf{X};\theta)$. Denoting $\mathbf{y}_{ij}\in\mathbb{R}^{2\times 1}$ and $\mathbf{\phi}_{ij}\in\mathbb{R}^{n\times 1}$ as the optical flow and predicted distribution parameter for the $(i,j)^{\textrm{th}}$ pixel, we write $p(\mathbf{Y}|\mathbf{X};\theta)=p(\mathbf{Y}|\Phi(\mathbf{X};\theta))=\prod_{ij}p(\mathbf{y}_{ij}|\phi_{ij}(\mathbf{X};\theta))$. To estimate confidence value $c\in[0,1]$ for a pixel $\mathbf{p}$ from the predicted distribution, the total probability of predicted optical flow within a radius $R$ of the estimated mean flow $\mathbf{\mu}$ is computed as $c=P(|y-\mu|<R)=\int_{\{y\in\mathbb{R}^{2}:|y-\mu|<R\}}p(y|\phi)dy.$ The confidence map $\mathbf{C}=\{c_{ij}\}~\forall~(i,j)\in[H,W]$ can be obtained by calculating the confidence value for every pixel.

With the dense correspondences, the next step is to estimate the relative camera pose $\mathbf{T}$. However, as the estimated correspondences are generally noisy and of varying quality, classical SfM pipelines mostly rely on off-the-shelf RANSAC algorithm [16] or other robust inlier maximization approaches [30][31]. Without losing the robustness from outlier rejection, we further assess the quality of correspondence matches in the optical flow network design so that we can reason about correspondence inliers at test time. Such a hybrid idea is not only effective for camera pose estimation but also helps refine the correspondence’s measure itself. To realize such an idea, we introduce a weighted bundle adjustment module (WBA).

The proposed WBA optimizes for camera pose and depth such that the induced optical flow computed from the current estimation of camera pose and depth should be consistent with the network’s predicted flow. Despite such an idea has some notion of similarity with [6] that obtains weights from convolutional layers without probabilistic modeling, we consider the uncertainty obtained from a mixture of probability models with additional robust outlier rejection, bidirectional flow consistency, careful pose initialization, and independent depth estimation, which significantly improves the accuracy of pose and depth estimates—refer to Sec.IV.

Let $\mathbf{p}_{ij}\in\mathbb{R}^{2\times 1}$ and $\bar{\mathbf{p}}_{ij}\in\mathbb{R}^{3\times 1}$ denote $(i,j)^{\textrm{th}}$ image pixel coordinate and its homogeneous coordinate. We compute $\mathbf{p}^{s}$, i.e., pixel in the source image corresponding to $\mathbf{p}^{r}$ in the reference image with predicted flow $\mathbf{Y}_{r\rightarrow s}$ as

As stated, our aim of using WBA is to mindfully solve for camera pose; therefore, we propose exploiting the consistency between the induced flow and predicted flow. By induced flow, we mean the optical flow derived from the camera pose and the intermediate depth estimates. Assume $\mathbf{T}=[\mathbf{R}|\mathbf{t}]$ as the relative camera pose from $\mathbf{I}^{r}$ to $\mathbf{I}^{s}$, where $\mathbf{R}\in{SO}(3)$, and $\mathbf{t}\in\mathbb{R}^{3\times 1}$ represents the translation vector. Let $\mathbf{Z}=(\mathbf{Z}^{r},\mathbf{Z}^{s}),~\mathbf{Z}^{r}\in\mathbb{R}^{H\times W},~\mathbf{Z}^{s}\in\mathbb{R}^{H\times W}$ be the intermediate depth representations used in WBA corresponding to $\mathbf{I}^{r}$ and $\mathbf{I}^{s}$, respectively. Now, we are ready to define the forward induced flow by back-projecting the pixel in $\mathbf{I}^{r}$ to 3D space and projecting it back to $\mathbf{I}^{s}$ using the $\mathbf{T}$ and $\mathbf{Z}^{r}$. We compute the induced-flow corresponding pixel in the source image as

Combining Eq.(1) and Eq.(2) gives us the reprojection error constraint, i.e., $\sum_{ij}||\mathbf{p}_{\textrm{flow},ij}^{s}-\mathbf{p}_{\textrm{ind\_flow},ij}^{s}(\mathbf{T},\mathbf{Z}^{r})||^{2}$ which helps refine the intermediate depth and pose. Meanwhile, we also have prior information pertaining to the quality of optical flow, i.e., the confidence map $\mathbf{C}$, which we have estimated using our uncertainty-aware flow estimation network (Sec III-A). While $\mathbf{C}$ does provide self-contained evidence of the correspondence quality, we further boost the robustness by combining it with the robust outlier filtering from RANSAC. Therefore, we introduce an additional binary mask $\mathbf{M}=\{m_{ij}\}~\forall~(i,j)\in[H,W]$ that includes high-value score $\mathbf{C}$ as well as RANSAC inliers. Accordingly, we compute the binary mask as

where $\gamma$ is the confidence threshold, $[\cdot]$ is the Iverson bracket, and $A$ symbolizes the set of RANSAC inliers computed from $\mathbf{Y}$. Using the defined binary mask and $c_{ij}\in\mathbf{C}$, we define the weights for our WBA as $w_{ij}=m_{ij}c_{ij}$. This leads us to our WBA formulation defined as

Eq.(4) is due to forward induced optical flow. Similarly, we can compute backward induced optical flow and have one more constraint $E_{s\rightarrow r}(\mathbf{T}^{-1},\mathbf{Z}^{s})$ for WBA. Combining both WBA constraints, we write the overall WBA objective as

We use the Gauss-Newton method [41] to optimize Eq.(5) for $K$ iterations. The initial pose $\mathbf{T}^{(0)}$ is obtained from the same RANSAC pose estimation step in the previous outlier filtering, and the initial depth map $\mathbf{Z}^{(0)}$ is set to all value one. At the $k^{\text{th}}$ iteration, we compute the camera pose update $\Delta\xi^{(k)}\in\mathfrak{se}(3)~\textrm{(lie-algebra corresponding to}~\mathbf{T})$, and the depth update $\Delta\mathbf{Z}^{(k)}$ for two depth maps. By vectorizing the camera pose and depth parameters, the updates are efficiently computed using the Schur decomposition [6]. The camera pose and depth are then updated via retraction on $SE(3)$ and addition, respectively over the iterations as follows

By now, we have detailed how we are predicting the uncertainty-aware dense optical flow and utilizing it for accurate pose estimation. In fact, those are critical to dense SfM nonetheless; the final goal is to recover an accurate depth map of the scene. To that end, we utilize the backbone architecture of MVS2D [42] with EfficientNet [43]. Given the estimated pose $\mathbf{T}$ and $\mathbf{X}$, the proposed depth network predicts the reference image depth as $\mathbf{D}^{r}=G(\mathbf{T},\mathbf{X};\theta_{D})$. However, if we switch the order of the reference and source image, i.e., $\mathbf{X}^{\prime}=(\mathbf{I}^{s},\mathbf{I}^{r})$, we can predict the source image depth map as $\mathbf{D}^{s}=G(\mathbf{T}^{-1},\mathbf{X}^{\prime};\theta_{D})$.

Additionally, we know there is an inherent dependency between camera pose, optical flow, and depth. As mentioned before (Sec. I), most previous methods overlook exploiting this dependency to refine pose and depth. On the contrary, we propose to exploit the relation between them over iteration to improve our camera pose and depth. The key idea is to utilize the current pose and depth estimates to update flow confidence. Given $\mathbf{T}$ and $\mathbf{D}=(\mathbf{D}^{r},\mathbf{D}^{s})$, we compute a new induced flow using Eq.(2), which provides additional evidence about the dense correspondences. The weights are penalized according to the distance between the predicted flow $\mathbf{Y}$ and the new induced flow¹¹1Alternatively, iterative reweighted least squares can also be used.. We adopt the RBF kernel with standard deviation $\sigma$ to penalize large distances. Taking forward flow as an example, the penalization factor can be computed as

where the weights are updated as $w_{r\rightarrow s,ij}:=\Delta w_{r\rightarrow s,ij}\cdot w_{r\rightarrow s,ij}$. We also update the predicted flow through a simple mixup as $\mathbf{Y}_{r\rightarrow s,ij}:=\frac{1}{2}\mathbf{Y}_{r\rightarrow s,ij}+\frac{1}{2}(\mathbf{p}_{\textrm{ind\_flow},ij}^{s}(\mathbf{T},\mathbf{D}^{r})-\mathbf{p}^{r}_{ij})$. We re-run the WBA with the updated flow and weight to obtain a better pose and further improve the depth.

The uncertainty-aware flow estimation network is trained using the negative log-likelihood loss, while the depth estimation network is trained using standard L1 loss.

We trained the two networks separately using ground truth pose $\mathbf{T}^{gt}$ and depth $\mathbf{D}^{gt}$ at train time. The induced ground truth flow $\mathbf{Y}^{gt}$ is estimated from $\mathbf{T}^{gt}$ and $\mathbf{D}^{gt}$ on-the-fly.

Following the previous methods [21][25][44], we evaluated our proposed pipeline on four standard two-view SfM benchmark datasets, namely YFCC100M [45], ScanNet [46], DeMoN [20], and KITTI VO [47].

We implemented our approach in Python 3.9 with PyTorch 1.11. The confidence threshold for weighted bundle adjustment is set to $\gamma=0.1$ [44]. For RANSAC pose initialization, we adopt the OpenCV implementation with threshold 1.0 and confidence 0.99. For camera pose estimation on YFCC100M-ScanNet and KITTI VO, we use the provided models of PDC-Net+ (H) [44] and PDC-Net (D) [15] pre-trained on Megadepth, respectively. For camera pose estimation on DeMoN dataset, we use the PDC-Net (D)  [15] pre-trained on Megadepth and fine-tune on DeMoN train set for 50 epochs using Adam optimizer with $lr=5e^{-5}$. On DeMoN, for training the depth estimation network, we use the DeMoN train set and train our model for 50 epochs using Adam optimizer with $lr=8e^{-4}$, which takes 48 hours on 4 NVIDIA TITAN RTX GPUs. To maintain the stability of fine-tuned flow confidence for pose estimation, we use the additional $4\times 4$ local grid filtering with confidence quantile in each local grid.

Unlike [21] which trains depth networks using their network’s predicted pose to better fit the ground-truth depth, we train the depth estimation network using ground truth pose alone. This helps assess the robustness of the depth estimation network against the predicted pose at test time. Contrary to [21], our depth-network training methodology is close to the conventional way of training a deep neural network.