Unsupervised Domain Adaptation for Point Cloud Semantic Segmentation via Graph Matching

In unsupervised domain adaptive point cloud semantic segmentation, we are able to access the source domain $\mathcal{X}_{s}=\{\mathbf{x}^{s}_{i}\}^{M^{s}}_{i=1}$ of ${M^{s}}$ point clouds with its segmentation labels $\mathcal{Y}_{s}=\{\mathbf{y}^{s}_{i}\}^{M^{s}}_{i=1}$ and the unlabeled target domain $\mathcal{X}_{t}=\{\mathbf{x}^{t}_{j}\}^{M^{t}}_{j=1}$ of ${M^{t}}$ point clouds, where $\mathbf{x}^{s}_{i}\in\mathbb{R}^{{N^{s}_{i}}\times 3}$ is the set of ${N^{s}_{i}}$ points and $\mathbf{x}^{t}_{j}\in\mathbb{R}^{{N^{t}_{j}}\times 3}$ is the set of ${N^{t}_{j}}$ points. Given data $\mathcal{X}_{s}$, $\mathcal{Y}_{s}$ and $\mathcal{X}_{t}$, our goal is to train a model which can precisely categorize the point of target data into one of the common semantic categories in the source data, and to alleviate the performance drop problem caused by the domain gap at the same time. As illustrated in Fig.1, we first construct the local feature distributions of the source and target domains with the proposed dynamic feature graphs. Then, by building a source-domain feature graph memory bank, we employ graph matching to obtain graph pairs between the graphs in the memory bank and the target graphs. Finally, with the obtained matching association, we utilize the designed losses for local-level feature alignment.

Different from the global-level feature alignment methods [6, 7, 15, 16, 17] roughly aligning two domains, we consider the differences of local neighborhood context in the target domain for the fine alignment. The main idea of our method is to use the learned dynamic local feature graphs to capture the multi-level features in different neighborhoods of point clouds. Then, based on the local-graph similarity, the correlation between local neighborhoods from two domains can be viewed as the knowledge transferring from the labeled source domain to the unlabeled target domain.

We leverage PointWeb [12] as our semantic segmentation backbone, which contains a classifier and a feature extractor with an encoder and a decoder. We use the feature extractor to extract the multi-level features and then build dynamic feature graphs on the sampled centroid points by the feature similarity. Specifically, given a point cloud sample $\mathbf{x}\in\mathbb{R}^{{N}\times 3}$ with $N$ points, we first extract its local-context feature using the feature extractor. Then, we select ${N/64}$ centroid points using the farthest point sampling for three iterations, where the centroid points are then as the kernel points of graphs for feature aggregation at each level. In detail, for each kernel point at different levels, we gather its k-nearest neighbors in the feature space ($k$ is different at each level). Thereby, a feature graph can be constructed by setting the k-NN features as its vertices $\{\mathbf{v}_{i}^{k_{1}},\mathbf{v}_{i}^{k_{2}},\mathbf{v}_{i}^{k_{3}},\mathbf{v}_{i}^{k_{4}}\}$ and the k-NN feature distances as the edges $\{\mathbf{e}_{i}^{k_{1}},\mathbf{e}_{i}^{k_{2}},\mathbf{e}_{i}^{k_{3}},\mathbf{e}_{i}^{k_{4}}\}$, where $i$ is the index of centroid points and $\{k_{j}\}_{j=1}^{4}$ is the different values in k-NN. We obtain ${N/64}$ dynamically updated local feature graphs to represent the local neighborhood context of the given point cloud, which can be formulated as,

$\displaystyle\mathcal{G}={\{\mathbf{g}_{i}\}}^{N_{c}}_{i=1}=\{\{\mathbf{f}_{i}^{k_{j}}\}^{4}_{j=1}\}^{N_{c}}_{i=1}=\{\{\mathbf{v}_{i}^{k_{j}},\mathbf{e}_{i}^{k_{j}}\}^{4}_{j=1}\}^{N_{c}}_{i=1},$

(1)

where $N_{c}$ denotes the number of ${N/64}$ centroid points, and $\mathbf{f}_{i}^{k_{j}}$ represents the feature embedding containing the vertice and the edge information. As a result, given a source sample and a target sample, the generated graphs are represented as $\mathcal{G}^{s}={\{\mathbf{g}^{s}_{i}\}}^{N_{c}}_{i=1}$ and $\mathcal{G}^{t}={\{\mathbf{g}^{t}_{i}\}}^{N_{c}}_{i=1}$.

In this section, based on the constructed dynamic graphs above, we aim to find the intrinsic correlation between the source domain and the target domain. We use the graphs from the source domain to guide the model to extract semantic discriminative features on the target domain. However, at the training stage, the sample category in a batch is limited and their graph patterns tend to present significant structure differences, which may potentially introduce the alignment bias. To address the issue above, we build up a feature graph memory bank $\mathcal{G}^{b}$ and store a graph $\mathbf{g}^{s}_{i}$ into the bank according to the corresponding category of the centroid point. Therefore, benefiting from such memory bank mechanism, we can sufficiently mine the rich source information from it for reliable target-domain feature learning. The memory bank provides the same capacity $B$ for each category of graphs. Once the number of graphs exceeds the capacity $B$ of the corresponding category, we will update the memory bank by replacing the oldest graphs with the new ones.

Given the graphs $\mathcal{G}^{t}={\{\mathbf{g}^{t}_{j}\}}^{N_{c}}_{j=1}$ from target domain, we consider finding the most similar graph from $\mathcal{G}^{b}$ to each graph in $\mathcal{G}^{t}$ for feature alignment. In particular, we use optimal transport for graph matching. Specifically, the total transport cost of optimal transport is used to measure the similarity between two graphs, and the assignment matrix $\mathbf{A}\in\mathbb{R}^{K\times K}$ is used to find the point-level correspondences for $K$ nodes in graphs. In the graph matching formulation, we first compute the distance matrix $\mathbf{D}\in\mathbb{R}^{K\times K}$, where the element $\mathbf{D}_{(m,n)}$ indicates the distance between the point $m$ in one graph and the point $n$ in the other graph. Here, we use the squared Euclidean distance in the feature space to measure the pairwise distance between points in graphs, where the points are composed of the corresponding features with edge and vertice information. Once we obtain the distance matrix $\mathbf{D}$, we apply the Sinkhorn algorithm [22] to obtain the final assignment matrix $\mathbf{A}$ and the total transport cost through solving the optimal transport problem. In this way, we can measure the relevance between each target graph with all the graphs in the memory bank. As a result, we can find the most similar graph $\mathbf{g}^{b}_{(a,l)}\in\mathcal{G}^{b}$ for the target graph $\mathbf{g}^{t}_{j}$ according to the sorting result of the transport cost for knowledge transferring, where $a$ indicates the category and $l$ indicates the index in the memory bank.

Based on the generated graph pairs, we formulate a local feature loss based on the assignment matrix for the local-level feature alignment. Given a target graph $\mathbf{g}^{t}_{j}$, we first select the most similar graph $\mathbf{g}^{b}_{(a,l)}$ from the memory bank. At the same time, we are able to access the corresponding assignment matrix $\mathbf{A}_{j}\in\mathbb{R}^{K\times K}$, which can decode the point-level corresponding feature assignment between two graphs. In detail, for each point in target graph $\mathbf{g}^{t}_{j}$, we can obtain the corresponding transport weights for every point in graph $\mathbf{g}^{b}_{(a,l)}$ from the assignment matrix $\mathbf{A}_{j}$. Then, we perform a weighted sum of $\mathbf{g}^{b}_{(a,l)}\in\mathbb{R}^{K\times D}$ to guide the learning of $\mathbf{g}^{t}_{j}\in\mathbb{R}^{K\times D}$, where $D$ is the number of feature channels. The key point is that the local neighborhood areas with similar semantic contexts need to have similar feature distributions. In this way, we can effectively align the indiscriminate feature distributions of the unlabeled target domain to the source domain. Therefore, we propose the following assignment matrix based local feature loss for feature graph learning in the target domain:

$\displaystyle\mathcal{L}_{loc}=\frac{1}{N_{c}}\sum_{j=1}^{N_{c}}{\left\|\mathbf{g}^{t}_{j}-\mathbf{A}_{j}\mathbf{g}^{b}_{(a,l)}\right\|}_{1}.$

(2)

Owing to a variety of feature graphs from different categories in our memory bank, we can further exploit the contrastive learning for more discriminative target-domain feature learning. Here, we select the category $a$ of the matched graph $\mathbf{g}^{b}_{(a,l)}$ as the positive category and the other categories as the negative categories. In order to obtain the representative features of each category, all graphs in the memory bank are used for calculating the feature representations. Although we have achieved the assignment matrix for each positive or negative pair, it is meaningless for point-level adaptation on unmatched pairs. Therefore, we use the mean of all features in the graphs with the same category to represent the feature representation of the corresponding category. It is worth noting that our graph is composed of multi-level features, so we calculate the mean features of different levels separately and then concatenate them as the final feature representation. The positive and negative features for graph $\mathbf{g}^{t}_{j}$ can be formulated as,

$\displaystyle\mathbf{f}^{+}_{j}=\frac{1}{B}\sum_{b=1}^{B}\sum_{c=1}^{C}\mathbb{I}_{[c=a]}\Theta\left({\mathbf{f}_{(b,c)}^{k_{1}},\mathbf{f}_{(b,c)}^{k_{2}},\mathbf{f}_{(b,c)}^{k_{3}},\mathbf{f}_{(b,c)}^{k_{4}}}\right),$

(3)

$\displaystyle\mathbf{f}^{-}_{j}=\frac{1}{BC}\sum_{b=1}^{B}\sum_{c=1}^{C}\mathbb{I}_{[c\neq a]}\Theta\left({\mathbf{f}_{(b,c)}^{k_{1}},\mathbf{f}_{(b,c)}^{k_{2}},\mathbf{f}_{(b,c)}^{k_{3}},\mathbf{f}_{(b,c)}^{k_{4}}}\right),$

(4)

where $C$ is the number of categories and $B$ is the capacity size of each category. The indicator function $\mathbb{I}$ returns 1 if the condition is satisfied or returns 0 if unsatisfied. We use $\Theta$ to represent the mean and concatenation operators.

Then, with the generated positive and negative features, we formulate the following contrastive loss for increasing the intra-category compactness and inter-category separability between the target graph $\mathbf{g}^{t}_{j}$ and the graphs in $\mathcal{G}^{b}$.

$\displaystyle\mathcal{L}_{con}=\frac{1}{N_{c}}\sum_{j=1}^{N_{c}}\left[{\left\|\mathbf{f}^{t}_{j}-\mathbf{f}^{+}_{j}\right\|}_{1}-{\left\|\mathbf{f}^{t}_{j}-\mathbf{f}^{-}_{j}\right\|}_{1}+\alpha\right]_{+},$

(5)

where $\alpha$ is the margin of the contrastive loss and $\mathbf{f}^{t}_{j}$ is the mean feature of graph $\mathbf{g}^{t}_{j}$. Therefore, with the proposed local feature loss and the contrastive loss, we consider the feature alignment from two complementary perspectives, which can significantly reduce the domain discrepancy in feature space.

For the unsupervised domain adaptation, the core challenge is how to learn the discriminative target features without labels. First of all, for the source domain, we use the standard cross-entropy loss for supervised training:

$\displaystyle\mathcal{L}_{seg}=-\frac{1}{N}\sum_{n=1}^{N}\sum_{c=1}^{C}\mathbf{y}^{s}_{(n,c)}\log\hat{\mathbf{y}}^{s}_{(n,c)},$

(6)

where $\mathbf{y}^{s}\in\mathbb{R}^{N\times C}$ is the semantic labels for $N$ points with $C$ semantic categories and $\hat{\mathbf{y}}^{s}$ is the outputs from the model.

In addition, in our framework, in order to identify the relationship of local features between the source domain and the target domain, we construct dynamic feature graphs and the generated graph pairs based on the graph matching to find correspondences between the two domains. With our developed local feature loss based on the assignment matrix and the category-guided contrastive loss, we can effectively align the local features between the two domains. The overall loss can be formulated as:

$$\begin{aligned} \mathcal{L}_{all}=\mathcal{L}_{seg}+\lambda_{1}\mathcal{L}_{loc}+\lambda_{2}\mathcal{L}_{con}\end{aligned},$$

(7)

where $\lambda_{1}$ and $\lambda_{2}$ are hyperparameters balancing the proposed losses with semantic segmentation loss.

Furthermore, our framework can be extended into a two-stage method in a self-training manner, where we follow Jaritz et al. [20] to use a pseudo-label training strategy. We first use our framework to train a model with the loss in Eq. 7, where the source data $\mathcal{X}_{s}$, $\mathcal{Y}_{s}$ and the target data $\mathcal{X}_{t}$ are available. Then we fix the parameters of the model and generate pseudo labels $\mathcal{\hat{Y}}_{s}$ for target data. After that, the supervised semantic segmentation loss with pseudo labels is used on the target domain.

vKITTI to SemanticPOSS. The synthetic dataset vKITTI [23] contains 6 sequences of outdoor scenes in urban settings, where the point cloud are generated from the synthetic 2D depth images. The SemanticPOSS [24] dataset was obtained in dynamic driving scenarios. It is composed of 6 sequences of scenes with a total of 2988 LiDAR scans. Therefore, there is a large gap in data distribution between the vKITTI and the real-world SemanticPOSS. For the domain adaptation scenario from vKITTI to SemanticPOSS, we select 6 semantic categories for domain adaptation: plants, building, road, traffic sign, pole, and car. The point clouds are sampled into blocks of 15m$\times$15m, and each block contains 4096 points.

We use the official PyTorch implementation for PointWeb as our segmentation backbone. Stochastic Gradient Descent (SGD) optimizer is selected for training with the momentum 0.9 and the weight decay 0.0001, respectively. Also, we apply the weight decay to the learning rate, where the drop factor is 0.1 and the step size is 30. The initial learning rate for indoor and outdoor scenarios are 0.05 and 0.005. The capacity size $B$ of the memory bank is set to 16. The parameters $\lambda_{1}$ and $\lambda_{2}$ are set to 1.0 and 0.1. The margin $\alpha$ of the contrastive loss is set to 0.4. The values in k-NN for different levels are set to 1, 4, 16 and 64. To train and test our model, we use a single TITAN RTX GPU and the batch size is set to 4.

We report the performance of point cloud semantic segmentation with mean Intersection-over-Union (mIoU). Tab.I shows the quantitative results of the comparison between our method with other domain adaptation methods. As shown in the table, our method achieves the highest performance, which shows the effective domain transferability of our method. Specifically, the Supervised means the model of PointWeb is trained on the target domain with semantic labels. The Source Only means the model trained with the source domain and directly tested at the target domain. Due to the domain gap, the performance of Source Only has a significant drop compared to the Supervised, which shows the necessity of unsupervised domain adaptation.

In order to verify the effectiveness of our method, we compare our method with a series of general unsupervised domain adaptation methods: MinEnt [16], MaxSquare [15], and ADDA [17]. For a fair comparison, these methods are reproduced with the same setting in our framework, where the hyperparameters are adjusted to obtain the best performance on all domain adaptation scenarios. The PL is the same Pseudo-Label training strategy in the [20] with our framework, which is a two-stage method with extra training cost. Furthermore, we introduce the geodesic correlation alignment used in [5] into our segmentation framework to construct additional comparison method 3DGCA. It can be observed from the Tab. I that although the above methods can alleviate the domain discrepancy, they are not efficient for point cloud semantic segmentation. Especially with the global-level feature alignment methods, the model produces the confused semantic information in the feature space. In the S3DIS to ScanNet scenario, these methods even produce negative effects on domain adaptation.

Furthermore, we compare our method with the 3D unsupervised domain adaptation methods: SqueezeSegV2 (SQSGV2) [5] and Complete & Label (C&L) [3]. Because these methods use different point cloud semantic segmentation backbones, we directly use the results of SQSGV2 and C&L reported in [3] for comparison. Since SQSGV2 requires spherical projection and C&L requires sequences of point cloud for the completion network, it is limited to reproduce in the vKITTI to SemanticPOSS and S3DIS to ScanNet domain adaptation scenarios. As shown in the Tab. I, compared with the above methods, our method achieves state-of-the-art performance on three domain adaptation scenarios.

In order to further verify the effects of each module of our method and the effectiveness of the proposed assignment matrix based local feature loss, we conduct ablation studies on the vKITTI to SemanticPOSS scenario.

As shown in Tab. II, we first report the performance improvement brought by each proposed loss, where the quantitative results of each loss can show its effective domain transferability. Particularly, the integration of the two losses can benefit the overall domain adaptation framework, and further improve the domain adaptation performance. It can be observed that our framework can benefit from a simple pseudo-label training strategy with additional 3.0% improvement, and they play a complementary role in unsupervised domain adaptation.

Secondly, in order to verify that the feature distributions of different categories has been separated, we draw t-SNE [29] visualization with the features from the target domain to show qualitative results. As shown in Fig. 3, our proposed method can effectively enhance the discrimination of features from the target domain.

Thirdly, we conduct the ablation study without using the assignment matrix named $L_{loc}$ w/o $A$. In this case, we use the mean features mentioned in Sec.III-C to represent the local feature graphs, and directly select the nearest neighbor from the memory bank to find the relationship between the graphs from the source domain and the target domain. Instead of using the assignment matrix for the alignment, we use the mean features to directly align the two graph features. As shown in Tab. II, the proposed assignment matrix based local feature loss can achieve a better performance.

At last, we show the visualization of point cloud semantic segmentation results to qualitatively illustrate the effectiveness of our method. It can be clearly observed in Fig. 2, compared with the Source Only, only a few noise predictions are produced in our method, which shows the proposed framework can effectively alleviate the domain gap problem and significantly improve the segmentation performance.