AA-RMVSNet: Adaptive Aggregation Recurrent Multi-view Stereo Network

According to output scene representations, traditional MVS reconstruction methods can be categorized into three types: volumetric [18, 28], point-based [19, 9] and depth-based [5, 10, 26, 29]. Volumetric methods first discretize the whole 3D space into regular cubes and then decide whether a voxel belongs to the surface or not with the photometric consistency metric. The space discretization is memory intensive, thus these methods are not scalable to large-scale scenarios. Point-based methods focus on the 3D points, usually start from a sparse set of matched key points and use the propagation strategy to gradually densify the reconstruction, which limits the capacity of parallel data processing. In contrast, depth-based methods have shown more flexibility in modeling the 3D geometry of scene. It reduces the MVS reconstruction into relatively small problems of per-view depth map estimation, and can be further fused to point cloud [22] or the volumetric reconstructions [23]. Many successful traditional MVS algorithms yielding depth maps have been proposed. Schönberger et al. present COLMAP [26], which uses hand-crafted features and jointly estimates pixel-wise view selection, depth map and surface normal to utilize the photometric and geometric priors. Xu et al. propose ACMM [29] with multi-scale geometric consistency, adaptive checkerboard sampling and multi-hypothesis joint view selection. Although traditional MVS methods yield impressive results, they utilize hand-crafted features which are not suitable for non-Lambertian surfaces, low-textured and texture-less regions where photometric consistency is unreliable.

Rather than using traditional hand-crafted image features, recent studies on MVS apply deep learning for better reconstruction accuracy and completeness. Volumetric methods SurfaceNet [12] and LSM [13] are first proposed. They construct a cost volume using multi-view images and use 3D CNNs to regularize and infer the voxel. However, SurfaceNet and LSM are restricted to only small-scale reconstructions due to the common drawback of the volumetric representation. Depth-based method MVSNet [33] improves the MVS reconstruction performance a lot compared with SurfaceNet and LSM. MVSNet takes one reference image and several source images as input and extracts deep image features, then encodes camera geometries in the network to build the 3D cost volumes via differentiable homography. To reduce the huge memory consumption of MVSNet, some variants of MVSNet have been proposed recently and can be divided into: multi-stage methods and recurrent methods. Multi-stage methods, such as CasMVSNet [11], CVP-MVSNet [32], UCS-Net [7], Vis-MVSNet [37], use the coarse-to-fine strategy that first predict a low resolution depth map with large depth interval and iteratively upsamples and refines the depth map with a narrow depth range. Though the coarse-to-fine architectures successfully reduce memory consumption, they are not suitable for high-resolution depth reconstructions as the depth prediction of coarse stage may be wrong with a large depth interval. To this end, recurrent methods, such as R-MVSNet [34] and $D^{2}$HC-RMVSNet [31], are proposed. They sequentially regularize cost maps along the depth dimension with recurrent networks to avoid using memory-intensive 3D CNNs; thus they can infer depth maps within a very large depth range. R-MVSNet regularizes cost volumes in a sequential manner using convolutional gated recurrent unit (GRU). $D^{2}$HC-RMVSNet improves R-MVSNet with more powerful recurrent convolutional cells, ConvLSTMCells, and a dynamic consistency checking strategy.

Though achieving promising results, most of the aforementioned learning-based MVS methods still have difficulties in handling challenging regions and severe occlusion problems in MVS.

As have been covered, 3D cost volumes are constructed by matching 2D feature maps so extracting recognizable and reliable features is of great significance in MVS. As for 3D reconstruction, it is universally acknowledged that reflective surfaces and low-textured or texture-less areas are main difficulties for a common CNN to handle which is operated on regular 2D grids with fixed receptive fields. For those challenging regions that are generally lacking in texture, we expect the receptive fields of convolutions to be larger while smaller receptive fields are favored for regions with rich texture. We introduce an intra-view AA module illustrated as Fig. 3 for adaptive aggregating features of different scales and regions with varying richness of texture. In the intra-view AA module, 3 feature maps of different spatial scales, whose sizes are $H\times W\times 16$, $\frac{H}{2}\times\frac{W}{2}\times 16$ and $\frac{H}{4}\times\frac{W}{4}\times 16$ respectively, are processed by 3 one-stride deformable convolutions [8, 38] with exclusive parameters. The definition of a deformable convolution is defined as

$$\mathbf{f}^{\prime}(\mathbf{p})=\sum_{k}w_{k}\cdot\mathbf{f}(\mathbf{p}+\mathbf{p}_{k}+\Delta\mathbf{p}_{k})\cdot\Delta m_{k},$$

(3)

where $\mathbf{f}(\mathbf{p})$ denotes the feature value pixel $\mathbf{p}$; $w_{k}$ and $\mathbf{p}_{k}$ represent the kernel parameter and fixed offset defined in a common convolution operation; $\Delta\mathbf{p}_{k}$ and $\Delta m_{k}$ are the offset and modulation weight yielded adaptively by learnable sub-networks of deformable convolution. By interpolating smaller feature maps to $H\times W$, we obtain 3 feature maps with 16, 8, 8 channels respectively and these features are concatenated to construct a feature map of $H\times W\times 32$.

After per-view cost volumes have been constructed, the next step is to aggregate all cost volumes into one for regularization.

A common practice is to average $N-1$ cost volumes, whose underlying principle is that all views should be of equal importance. However, this is not reasonable enough since varying shooting angles may lead to problems such as occlusion and different lighting conditions of non-Lambertian surfaces that make depth estimation more difficult.

Therefore, as illustrated in Fig. 4, we design an inter-view AA module to handle unreliable matching costs, which is defined as

$$\mathbf{C}^{(d)}=\frac{1}{N-1}\sum_{i=1}^{N-1}[1+\boldsymbol{\omega}(\mathbf{c}_{i}^{(d)})]\odot\mathbf{c}_{i}^{(d)},$$

(4)

where $\odot$ denotes Hadamard multiplication and $\boldsymbol{\omega}(\cdot)$ is pixel-wise attention maps adaptively yielded according to per-view cost volumes. In this way, pixels that are likely to be confusing for matching will be suppressed while those with crucial context information will be assigned with larger weights. $1+\boldsymbol{\omega}(\cdot)$ better prevents over-smoothness than $\boldsymbol{\omega}(\cdot)$ alone.

Cost regularization is to leverage spatial context information and to turn matching costs into a probability distribution of $D$ depth hypotheses. The regularization network adopts a RNN-CNN hybrid fashion where a cost volume ($H\times W\times D\times 32$) is sliced at the dimension of $D$. As is illustrated in Fig. 2, feature passing in the regularization network has both horizontal direction and vertical direction. Horizontally, each slice of 3D cost volume is regularized by a CNN with encoder-decoder architecture; on the vertical direction, there are 5 parallel RNNs to deliver intermediate outputs of former ConvLSTMCells to later ones.

Considering a cost volume slice of depth hypothesis $d$ to be processed by the $j$-th convolution layer, denoted as $\mathbf{v}_{j-1}^{(d)}$, the output of this layer with depth hypothesis $d-1$ as $\mathbf{v}_{j}^{(d-1)}$ and memory maintained (or hidden state) as $\mathbf{m}_{j}^{(d-1)}$, operations within a ConvLSTMCell are as follows.

Firstly, $\mathbf{v}_{j-1}^{(d)}$ and $\mathbf{v}_{j}^{(d-1)}$ are concatenated and after being processed by a convolution layer, the tensor is split into 4 tensors from the feature dimension, namely $\mathbf{w}$, $\mathbf{x}$, $\mathbf{y}$ and $\mathbf{z}$. The 4 signals within a LSTM cell are defined as

$$\left\{\begin{aligned} \mathbf{i}&=\sigma(\mathbf{w})\\ \mathbf{f}&=\sigma(\mathbf{x})\\ \mathbf{o}&=\sigma(\mathbf{y})\\ \mathbf{g}&=\tanh(\mathbf{z})\end{aligned}\right.$$

(5)

where all signals are two-dimensional in space and Sigmoid function $\sigma(\cdot)$ and hyperbolic tangent function $\tanh(\cdot)$ are all element-wise operations. Then the memory of LSTM is updated by

$$\mathbf{m}_{j}^{(d)}=\mathbf{m}_{j}^{(d-1)}\odot\mathbf{f}+\mathbf{i}\odot\mathbf{g},$$

(6)

while the output of the cell is

$$\mathbf{v}_{j}^{(d)}=\mathbf{o}\odot\tanh(\mathbf{m}_{j}^{(d)}).$$

(7)

Since cost volume regularization turns matching costs into a pixel-wise probability distribution of depth hypothesis, the task of depth estimation is now similar to a pixel-wise classification problem. Therefore, by encoding the ground truth with one-hot pattern, we adopt cross entropy to calculate the training loss, defined as

$$L=\sum_{\mathbf{p}\in\{\mathbf{p}_{v}\}}\sum_{d=d_{0}}^{d_{D-1}}-G^{(d)}(\mathbf{p})\ \log[P^{(d)}(\mathbf{p})],$$

(8)

where $G^{(d)}(\mathbf{p})$ and $P^{(d)}(\mathbf{p})$ denote ground truth probability and predicted probability of depth hypothesis $d$ at pixel $\mathbf{p}$. $\{\mathbf{p}_{v}\}$ is the set of valid pixels with reliable depth.

In this section, we provide ablation experiments to quantitatively analyze the effectiveness and memory cost of each adaptive aggregation method. The following ablation studies are performed on DTU dataset using the same parameters as Sec. 4.2. We compare four different network architectures with or without the proposed adaptive aggregation modules. Baseline applies general 2D CNN for feature extraction and the same hybrid LSTM structure for cost volume regularization without any additional modules.

Validation results of the mean average depth error with different components during training are shown in Fig. 8. It is clear that each individual module can significantly lower the depth error, and the two modules are complementary in full AA-RMVSNet to achieve the best performance.

We also test the point cloud results generated by different network models as shown in Tab. 3. Both intra-view AA and inter-view AA can improve the accuracy and completeness of 3D reconstruction results. Specifically, intra-view AA takes about 1.74GB additional memory and improves completeness by 0.28, while inter-view AA only costs extra 0.11 GB and gains 0.31 more in accuracy. The overall error drops from 0.391 to 0.357 with both two modules. Full AA-RMVSNet only takes 4.25GB to obtain dense and accurate depth maps with $800\times 600$ resolution, indicating that our method is fairly memory efficient.

Regarding ablation study for different experiment settings, please refer to the appendices for detailed results.