Dense Multimodal Alignment for Open-Vocabulary 3D Scene Understanding

Learning a robust 3D model that is generalizable to open vocabularies is challenging since it is unclear how to acquire the dense text labels for point clouds. Although well-trained human annotators could potentially provide detailed language descriptions of 3D scenes, such a method is costly and lacks scalability. To overcome this limitation, we leverage a tagging model and an MLLM to extract complete category information and scalable scene descriptions, respectively.

Complete Category Information. The scene tagging process is illustrated in Fig. 2. Firstly, we use the image tagging foundation model such as RAM [23] to extract all possible categories from an image, and utilize category names and short descriptions derived from the metadata as the text query, referred to as $T_{tag}$, such as “There is a {category name} in the scene”, “A photo of a {category name}”, etc. Unlike image captioning models [28, 1] that can only identify sparse and salient objects in a scene, RAM can recognize as many tags as possible without missing important parts, ensuring a high recall rate and alignment with complete semantic patterns. The more complete and accurate the detected categories are, the easier we can establish precise dense correspondences between text and 3D modalities, and hence the open-vocabulary capability of the 3D model can be enhanced.

Reliable GPT-based Denoising. While there are many tags recognized by RAM, some redundant or irrelevant tags are also included, such as non-noun words (“purple”, “blue”, “lay”, etc.) and objects that do not exist in the image (“bed”, “ukulele”, etc.), as shown in Fig. 2. We address this issue in two steps. Firstly, we utilize GPT to filter potential noisy vocabulary. Given the input list, we instruct GPT to examine the words one by one and perform reasoning according to the chain of thought, outputting a boolean list indicating whether a word is an outlier. Please refer to Fig. 1 of supplemental material for the detailed instructions and examples to reduce the noisy tags. Secondly, to decrease the non-existent categories in a scene, we conduct multi-view voting and neglect categories that appear in fewer than five views. Please refer to Fig. 2 of the supplemental material for the denoised scene tagging results and the corresponding visualizations of 3D label maps.

Scalable Scene Description. Although scene-level tags have already covered most of the categories, the limited scalability and variation of category names hinders their provision of rich contexts and details. To address this limitation and enable arbitrary queries for 3D networks, we additionally leverage MLLMs such as LLaVA [35] to generate diverse and linguistically expressible descriptions, denoted by $T_{llm}$. Owing to the exposure to a diverse range of linguistic patterns and contextual nuances, the MLLMs can generate comprehensive and in-depth descriptions based on input images. Consequently, these LLMs can enhance the richness and granularity of the generated textual representations, thereby facilitating a more comprehensive understanding of the 3D scenes. Please refer to Fig. 3 of supplemental material for the examples of scene-level captions and corresponding visualizations.

Finally, we generate the text embeddings $\textbf{f}^{T}_{tag}$ and $\textbf{f}^{T}_{llm}$ using CLIP text encoder based on the generated tags $T_{tag}$ and scene descriptions $T_{llm}$, respectively, which are utilized to supervise the training of 3D networks subsequently.

Compared to language modality, the image modality offers a wealth of contextual information and exhibits significant variations among different pixels, which could provide more effective supervision. Inspired by this observation, OpenScene [42] distills the knowledge from frozen open-vocabulary 2D segmentation models, such as LSeg [27] and OpenSeg [14]. However, these methods suffer from two major limitations. Firstly, they are fine-tuned on in-vocabulary datasets, which leads to a misalignment between image and text features and consequently results in poor performance on open-vocabulary categories. Secondly, all of these methods freeze 2D networks, failing to perceive the 3D structure of objects and leading to inaccurate supervision. As shown in Fig. 3, we visualize the segmentation results using 2D and 3D features. One can observe that although 2D features are more advantageous in segmenting background objects with ambiguous geometry, such as “bookshelf”, “door” and “blackboard”, they are less effective in segmenting objects with distinct shapes, such as “table” and “chair”. Therefore, it is necessary to distill the structural priors of 3D networks into 2D ones as well in order to facilitate fine-grained scene understanding.

In this paper, we adopt FC-CLIP [56] as the backbone to extract image features. On one hand, we use the frozen CLIP visual encoder to ensure the intactness of image-text alignment, obtaining CLIP features $\textbf{f}^{2D}_{clip}$. On the other hand, to facilitate the synergistic benefits of both 2D and 3D modalities, we fine-tune the mask head and attain the mask features $\textbf{f}^{2D}_{mask}$. In contrast to previous methods that rely on potentially noisy fixed image features for supervision, the fine-tuned mask features enhance the adaptability to downstream 3D tasks. We explore different fine-tuning strategies, such as LoRA [18], Adapter [17], and full parameter fine-tuning and compare them in experiments.

Once the text and image modalities are constructed, the subsequent step is to associate each point to its corresponding pixel and text. We utilize the image modality as a bridge to establish separate associations between pixels and other modalities. Firstly, we construct the associations between image and language modalities by taking $C$ different text embeddings $\textbf{f}^{T}=\{\textbf{f}^{T}_{1},\cdots,\textbf{f}^{T}_{C}\}$ as classifier to assign text labels to each pixel, obtaining a 2D semantic score map, denoted by $S^{2D}\in\mathbb{R}^{H\times W\times C}$. This process can be formulated as follows:

where $S^{2D}_{c}(u,v)$ denotes the probability that the pixel at location $(u,v)$ belongs to the $c$-th text label, and $<\cdot>$ represents the cosine similarity between two $\ell_{2}$-normalized feature vectors. $\tau_{1}$ is a temperature parameter. Then we establish the associations between images and point clouds by back-propagating 3D points $\textbf{p}=(x,y,z)$ onto 2D positions $(u,v)$ using a projection matrix $T\in\mathbb{R}^{3\times 4}$, i.e., $[u,v,w]=T[x,y,z,1]$, where $w$ is a scaling factor.

Finally, we associate the text and 3D modalities by taking image as the bridge. Given $K$ different projection views for one point p, we compute its average semantic score, denoted as $\bar{S}^{3D}$, across $K$ views $[S^{2D}_{1}(u_{1},v_{1}),\cdots,S^{2D}_{K}(u_{K},v_{K})]$. Based on the aggregated 3D semantic scores, the final text-to-3D label map, denoted as $M^{3D}\in\mathbb{R}^{N\times C}$, can be derived by:

where $N$ denotes the number of points and $M^{3D}_{c}$ indicates whether the point belongs to the $c$-th text label or not. $M^{3D}$ can be regarded as the pseudo label map for point cloud, serving as the supervision signal for training 3D models.

It is noteworthy that instead of generating one-hot label through the argmax operation, we select all confident text labels whose scores exceed the threshold. This is because the generated text categories may exhibit similarities in semantics (like ‘suitcase’ and ‘luggage’) or inclusion relationships (such as ‘kitchen’ and ‘stove’). As a result, it is highly possible that one single point corresponds to multiple text labels simultaneously.

After obtaining the triplets of different modalities and their dense correspondences, the subsequent objective is to align the 3D points with their corresponding text and pixel embeddings. This alignment process involves several steps. Firstly, we extract 3D features for the point cloud by utilizing a 3D network, denoted as $\varepsilon_{3D}$. These features are then projected to match the dimension of the CLIP features.

Next, we assign text labels to different 3D points by computing the cosine similarities between point and text embeddings $\textbf{f}^{T}$, yielding a 3D segmentation probability map $P^{3D}$:

where $\textbf{f}^{3D}_{i}$ denotes the feature of the $i$-th point, and $P^{3D}_{i,c}$ denotes the probability that the $i$-th point belongs to the $c$-th text label. Here we employ the Sigmoid activation function $\sigma(\cdot)$ since it will not lead to mutually exclusive relationships among different categories.

Text-to-3D Supervision. We use the text-to-3D label map $M^{3D}$ as the pseudo label to facilitate the alignment of point and text features. Different loss functions are employed for aligning the point embeddings with the tag ${\bf f}_{tag}^{T}$ and scene description ${\bf f}_{llm}^{T}$ embeddings. As can be seen in Fig. 1, we build dense associations between ${\bf f}_{tag}^{T}$ and the entire point cloud, resulting in $M_{tag}^{3D}$. Consequently, The Binary Cross Entropy (BCE) loss is used to effectively penalize both positive and negative samples:

As for ${\bf f}_{llm}^{T}$, since it corresponds only to salient objects, we can only obtain the mask for partial points, denoted as $M_{llm}^{3D}$. (Visualizations of $M_{tag}^{3D}$ and $M_{llm}^{3D}$ are given in Fig. 2 and Fig. 3 of the supplementary material, respectively.) We utilize the cosine similarity loss to supervise only the positive samples:

Mutually Inclusive Loss. In this work, we do not employ the Cross-Entropy loss because it would result in a mutually exclusive relationship between different classes, meaning that each point is assigned to only one class of interest. However, in text-to-3D alignment, one point may simultaneously associate with multiple text prompts, such as ‘bed’ and ‘bedroom’, ‘chair’ and ‘office chair’, ‘curtain’ and ‘drape’, etc. To handle this issue, we employ mutually inclusive losses (MIL), such as BCE loss and cosine similarity loss, to ensure that each point is aligned with all its corresponding tags/descriptions simultaneously, avoiding the potential conflicts between categories with overlapping or similar semantics.

2D-to-3D Supervision. For 3D-2D pairs, we follow the previous work [42] to fuse pixel embeddings across $K$ different views, represented as $[\textbf{f}^{2D}_{1},\cdots,\textbf{f}^{2D}_{K}]$, into a single feature vector $\bar{\textbf{f}}^{2D}$, and align 2D and 3D features by minimizing the cosine similarity loss:

Since 2D mask head is also trainable, we additionally add the text-to-2D supervision and compute the BCE loss between 2D predictions and 2D masks, obtaining $\mathcal{L}_{text-2d}$. Finally, the overall objective function to perform dense multimodal alignment is defined as:

where the language modality provides comprehensive textual descriptions, and the image modality gives precise supervision on object edges and contextual information. Additionally, the 3D modality reveals crucial structural information of objects. By densely aligning these modalities in a shared space, our method can maximize the synergistic benefits among them and achieve outstanding segmentation performance without compromising the open-vocabulary classification ability of the model.

Datasets. To demonstrate the effectiveness of our proposed method, we employ three popular datasets, i.e., ScanNet [11], Matterport3D [6], and nuScenes [4]. The first two datasets are indoor ones, comprising RGBD images and 3D meshes. The third one is an outdoor dataset, consisting of data collected from two sensors, i.e., LiDAR and camera. We conduct comparisons with state-of-the-art methods on each of these datasets. The mean Intersection-of-Union (mIoU), mean Accuracy (mACC), Precision, and Recall are employed as the evaluation metrics.

Implementation Details. In this work, MinkowskiNet [10] is employed as the 3D backbone, whose voxel size is set to 2cm for ScanNet and Matterport3D and 5cm for nuScenes. As for the 2D backbone, we use OpenSeg [14] and FC-CLIP [56] that perform mask-wise classification. The parameters $\tau_{1}$ in Eq. 1 and $\tau_{2}$ in Eq. 3 are both set to 0.1. We use Adam [25] as the optimizer and the initial learning rate is set to $1e-4$. The model is trained for 100 epochs. We set the batch size as 8 for indoor datasets and use one single NVIDIA RTX A6000 for training. As for nuScenes dataset, we use 8 GPUs for training and set the batch size as 16.

We compare the proposed DMA with fully-/weakly-supervised and zero-shot methods [42, 13, 55]. Tab. 2 presents the segmentation results on the ScanNet [11] dataset. To facilitate comparison, we measure the results of OpenScene by using 3D and 2D-3D integrated features as supervision. As can be seen in Tab. 2, although OpenScene(LSeg) attains better results (54.2% mIoU) by using both 2D and 3D encoders, it results in significantly increased inference latency. This is because the parameter size of 2D encoder is much larger than 3D encoder, and the 2D encoder needs to perform inference on multi-view images of the scene. Our DMA(OpenSeg) using only 3D model for prediction outperforms OpenScene(OpenSeg)-2D3D by 5.4% mIoU at a significantly lower latency, wherein the mIoU (F) and mIoU (B) are improved by 8.8% and 0.5%, respectively. This is because we perform additional alignment with text modality, thereby compensating for the decreased open-vocabulary ability of 2D model. When using text supervision only, our method outperforms the text-supervised approach RegionPLC [55] by 9.5%, and even surpasses OpenScene(OpenSeg)-2D3D by 2.6% in terms of mIoU. This indicates that, compared to previous methods that generate image- and region-level captions, our method establishes dense and precise correspondences between text and 3D points by taking 2D modality as the bridge, achieving more precise supervision. The suboptimal performance of our method using FC-CLIP as the 2D encoder may be attributed to the low resolution of the images (320$\times$240), which limits the capabilities of FC-CLIP.

Outdoor Scenes. To validate the effectiveness of our method on outdoor point clouds, we evaluate the performance of DMA on the nuScenes dataset [4]. Due to the highly imbalanced class distribution of outdoor scenes, we additionally measure the performance on base and long-tail categories. As shown in Tab. 2, by densely aligning with the tagging information and the detailed description extracted from each scene, our DMA(OpenSeg) using only 3D encoder significantly improves the performance over OpenScene(OpenSeg)-2D3D by 3.0% mIoU. Additionally, the final performance is further improved by 2.3% and attains 47.4% mIoU by employing FC-CLIP [56] to extract 2D features. This is because FC-CLIP could achieve more precise segmentation while maintaining outstanding open-vocabulary recognition ability of CLIP. Besides, by fine-tuning the mask head, FC-CLIP could incorporate the 3D structural priors into mask features and produce better results.

Long-Tail Datasets. As shown in Tab. 3, we validate the open-vocabulary methods on the more challenging long-tail 3D scene understanding datasets, i.e., ScanNet200 [45]. Following [45], we partition the 200 categories into three splits, i.e., head, common, and tail sets, facilitating a more comprehensive comparison across categories with different frequencies. When training on head classes (ceiling, curtain, window, etc.), the fully-supervised method performs much better than zero-shot methods due to the sufficient 3D labels for supervision. However, on the common and tail splits, our DMA method approaches to or even surpasses the fully-supervised competitors. This is because there are only a few instances available on these long-tail categories, which is not sufficient to train a robust model from scratch. Our method does not rely on ground truth 3D labels but instead distill knowledge from pretrained vision-language models, thus it is more robust to rare objects.

To further validate the robustness of our method on rare objects/classes, we evaluate on the most frequent $K$ classes of Matterport3D [6], where $K=21,40,80,160$. We train a 3D model by taking our generated textual descriptions and image features as supervision, and perform inference on different $K$ categories. As shown in Tab. 4, when employing the same 2D network, i.e., OpenSeg, our method demonstrates superior zero-shot segmentation capability on both common and rare categories. Specifically, our DMA(OpenSeg)-3D surpasses OpenScene(OpenSeg)-3D by 3.8%, 4.5%, 1.6%, and 1.2% in terms of mIoU at different $K$. This can be attributed to that OpenScene heavily relies on 2D model for supervision without aligning with text prompts, which limits its open-vocabulary ability. Our method, however, directly aligns with the textual modality, overcoming the limitations of 2D models.

Qualitative Comparison. Fig. 4 visualizes the segmentation results of different methods. We can observe that OpenScene [42] with only 3D encoder exhibits poor performance in segmenting objects that lack spatial structures, such as “door”, “window”, “counter”, etc. In contrast, text supervision offers more refined guidance by establishing dense correspondences between the texts and points, thereby enabling more precise alignment. Our approach leverages the advantages of both language and 2D modalities, and achieves excellent segmentation results for both foreground and background classes using only the 3D model.

2D Features vs. 3D Features. In Fig 6, we compare the segmentation performance on ScanNet by using different features. ‘F’ and ‘B’ denote foreground and background classes, respectively. For OpenScene [42], we observe that its 2D features are more advantageous for segmenting background categories with ambiguous geometry than 3D ones, i.e., 49.1% vs. 47.1% mIoU(B), while 3D features excel at segmenting the foreground objects with distinct shapes, i.e., 50.0% vs. 42.5% mIoU(F). Although the 2D-3D hybrid feature can leverage the strengths of both features simultaneously, utilizing 2D models for inference introduces significant computational overhead (please refer to the latency in Tab. 2). By additionally aligning with our generated text modality, our method can achieve outstanding performance on both foreground (58.3%) and background (51.5%) categories using only 3D features. Besides, DMA achieves comparable performance to using both 2D and 3D encoders by solely utilizing the 3D encoder, i.e., 53.3% vs. 53.5% mIoU(F), and hence significantly reducing inference time.

Tagging Models vs. MLLMs. In Tab. 6, we compare the results of using different tagging models and MLLMs on ScanNet. For the enhanced version, we replace RAM with RAM++ [22], and LLaVA-7B with LLaVA-13B. We can observe that our method outperforms RegionPLC [55] by a large margin (about 6.7%) by building dense point-to-text correspondences. The tagging model plays a key role for performance improvement since it encompasses extensive semantic patterns, while MLLM further enhances the final performance by incorporating rich contextual information. By filtering out noisy tags with GPT, the performance can be improved by 2.6% and 1.3% for the basic and the enhanced versions, respectively. The final performance can be further improved when stronger tagging models/MLLMs are employed.

CLIP Features vs. Mask Features. In addition to OpenSeg [14], we employ FC-CLIP [56] to extract 2D features due to its effectiveness. As show in Tab. 6, we compare the performance by using CLIP and Mask features as supervision. ‘In’ and ‘Out’ denote in-vocabulary and out-vocabulary classes, respectively. FC-CLIP contains an in-vocabulary classifier and an out-vocabulary classifier, which correspond to the seen and unseen categories in the training process, respectively. As can be seen, the fixed CLIP feature is more advantageous in segmenting unseen categories, which outperforms mask feature by 10.5% in terms of mIoU(Out). This demonstrates that the fixed CLIP visual encoder could maintain the strong generalization ability on novel classes. While for in-vocabulary classes, mask features outperform the CLIP feature by 11.6%. By combining these features, we can simultaneously achieve competitive results on in- and out-vocabulary categories, attaining 44.8% mIoU over all classes.

Comparisons of Different Fine-Tuning Methods. We fine-tune the mask head of FC-CLIP with different strategies for incorporating 3D structural priors into mask features. As can be seen in Tab. 6, by fully fine-tuning the mask head, the performances of 2D and 3D masks are improved by 3.8% and 1.9%, respectively. LoRA [18] and Adapter [17] can also achieve obvious improvements by tuning a small amount of parameters.

Open-Vocabulary Segmentation for Different Text Queries. We finally investigate the ability of our method to segment rare categories. As shown in Fig. 7, our method can accurately segment the corresponding regions for the given texts/queries in 3D scenes, even for unseen categories. For instance, our well-trained model can quickly locate the position of new categories such as “Snoopy”, or functional areas such as “kitchen”, etc. On one hand, we align with 2D CLIP features that have been trained with a vast corpus of text. On the other hand, we construct a comprehensive and scalable textual modality by using VLMs, further enhancing the understanding ability.