3D Scene Graph Prediction on Point Clouds Using Knowledge Graphs

Scene graph prediction problems have been predominantly tackled on 2D image datasets. Two main methods are prevalent for integrating knowledge into the model. The first method is to extract the high-level structure inherent in the training dataset. For instance, MotifNet [13] leverages the most frequent relations between labeled object pairs in the training set for scene graph prediction. Knowledge-embedded routing network (KERN) [14], on the other hand, employs the co-occurrence probabilities between objects implicitly to aid in resolving the long-tail problem in the dataset.

The second method involves utilizing external knowledge sources for the task. For example, Gu et al.[15] used ConceptNet [9] to refine object and relationship features prior to training on scene graph generation. In [11], Zareian et al.introduced GB-Net, which unifies scene graphs and knowledge graphs by learning the node encoding and edge encoding both within and between graphs. In their model, a scene graph represents an ”image-conditioned instantiation” of a commonsense knowledge graph. They employed multiple knowledge bases as external knowledge sources, including WordNet [10], ConceptNet [9], and Visual Genome [8]. This represents one of the latest attempts to merge knowledge graphs and scene graphs, enabling learning of object features and predictors with neighboring nodes, while also constructing a ”bridge” that infuses information between the two graphs through message passing.

Our work adopts the second method since we aim to generalize beyond the training data and deploy the algorithm in a real-world setting. we also aim to explore the effectiveness of external commonsense knowledge in generating 3D scene graphs, and as such, we adapt the bridging structure similar to [11]. However, due to inherent differences in our dataset (image vs. point cloud), our method also takes into account the incompleteness of the point cloud dataset and innate 3D relationships in the reconstructed scene. Moreover, the knowledge graphs we construct differ from [11]., as the indoor robot task employs a distinct set of objects and relationships.

The application of deep learning methods for 3D point cloud recognition can be traced back to PointNet [16], a model still extensively to encode and predict point cloud data. In our study, we likewise use PointNet as a backbone for point cloud embedding.

Recently, there has been some research into indoor scene graphs specifically designed for indoor robot mapping. One example is the 3D scene graph [17], which establishes a semi-automatic framework to create a dataset unifying objects, rooms, and cameras in a structured manner. While some traditional approaches [18, 19] use SLAM for semantic mapping at the object level, their maps do not include labeled semantic relationships. Certain methods [5, 3] construct a scene graph from each image, and then merge the 2D scene graphs into a global 3D graph. However, these approaches are tested on a small selection of objects and minimal relationships. The works most similar to ours include [12, 1], with the 3DSSG dataset first introduced in these studies. [12] proposed the Scene Graph Spatial Network (SGPN), which learns the 3D scene graph from the reconstructed point clouds. [1] further extended the method to accommodate RGB-D images as input and demonstrated that a model initially trained on reconstructed point cloud could be used for online scene graph prediction with a minor decline in prediction performance.

Based on these insights, our work focuses on generating scene graphs using the point cloud dataset, considering that the image-to-reconstructed point cloud can be accomplished either by traditional SLAM or by the graph fusing method. Notably, our approach differs from all previously mentioned methods by explicitly incorporating common-sense knowledge.

As shown in Figure 2, the entire pipeline consists of two streams: one originating from the point cloud and the other from knowledge. The first stream generates the scene representation (SR) of the scene graph. Given the input of a room’s point cloud, we extract the point cloud of each object and the predicate, in the form of the union of two clusters. These are encoded with a three-layer PointNet structure, constituting the first part of the SR node features. Additionally, we concatenate a set of contextual vectors to the PointNet feature embedding, as described in section IV-B. Edges of the SR graph are kept if two objects in the 3d space are within a certain Euclidean distance threshold, with both directions preserved at this stage.

The second stream originates from knowledge sources and thus is named knowledge representation(KR). Each node, representing either an object, subject, or predicate, is encoded using GloVe [20] embedding. The edges are provided by three different knowledge sources: Visual Genome, Conceptnet, and WordNet. The process of constructing a knowledge graph is explained in detail in section IV-C.

Next, both SR and KR are fed to a Knowledge-Scene Graph Network(KSGN). Through message passing, the node feature of both SR and KR will update alongside their neighboring node features, and the edge within and between graphs will update as well (Section IV-D). The resulting updated SR node feature of the object and predicate will finally pass through two layers of multi-layer perceptron(MLP) to predict relation triplets.

The nodes features of the input point cloud consist of the PointNet encoding and the contextual vector. The contextual vector is an 11-dimension vector first introduced in [1]. The vector includes the centroid of the point cloud $(x,y,z)\in\mathbb{R}^{3}$, the standard deviation that describes the sparsity of the segment $(\sigma_{x},\sigma_{y},\sigma_{z})\in\mathbb{R}^{3}$, size of the bounding box $(b_{x},b_{y},b_{z})\in\mathbb{R}^{3}$, maximum length of the segment $l=max(b_{x},b_{y},b_{z})\in\mathbb{R}$, and bounding box volume $v=b_{x}\cdot b_{y}\cdot b_{z}\in\mathbb{R}$. The vector is calculated for each segment in the room as well as the union of two segments for the predicate.

The knowledge graphs are constructed from three different sources:

Visual Genome [8]: This contains labeled scene graphs for image datasets. We filtered the object and relationship vocabulary for the training data and constructed four different matrices: subject-object, object-subject, subject-predicate, and predicate-object.

ConceptNet [9]: A multilingual crowd-sourcing-based knowledge graph database. We retrieved an object-object matrix from it.

WordNet[10]: A lexical database that contains semantics explanations and synonyms for nouns, verbs, and adjectives. We use WordNet for predicate-predicate relationships.

The knowledge graph in our study is defined using four types of adjacency matrices, as depicted in Figure 2 and Table I. For each object in the dataset, we assume they can either be an object type or a subject type. Thus for both subject-object and object-subject relationships, the adjacency matrices are square matrices. We list the details of the knowledge graph in Table I. Because the dataset we use contains 160 objects and 27 relationships, we also listed the matrix dimension for clarity.

Regrading ConceptNet, we use relatedTo for between-object relationships, with a binary weight. As for the predicate-predicate relationship, we have created a manually crafted categorical adjacency matrix. This matrix connects all directional spatial relationships (such as left, right, behind, front), comparison relationships (smaller than, bigger than, higher than, lower than), and relationships that imply an attachment (attached to, standing on, lying on). This manually created matrix is binary as well. For determining similarity scores between predicates, we use WordNet to get the Wu-Palmer (WUP) scores. For all other adjacency matrices, weights are normalized in a range from 0 to 1.

The Knowledge-based Scene Graph model (KSGN) is a modified version of Gb-Net [11]. As shown in Figure 2, KSGN accepts two types of graph input: the scene graph and the knowledge graph. Notably, both inputs are heterogeneous graphs.

The detailed model is depicted in Figure 3. The scene representation consists of Scene Entities(SE) and Scene Predicates (SP), and the common-sense knowledge representation consists of Common-Sense Entities (CE) and Common-Sense Predicates (CP). In the following, we use ${\Delta}\in(SE,SP,CE,CP)$ to represent different types of nodes.

Each node is encoded by 2-layers of Multi-Layer Perceptron (MLP) to form the node features as the Message Send:

$$\mathbf{m}_{i}^{\Delta\rightarrow}=\phi_{\text{send }}^{\Delta}\left(\mathbf{x}_{i}^{\Delta}\right)$$

(1)

where $\phi_{\text{send }}$ is the MLP that is named as ”send head”, It is trained and shares the weight across four types of nodes.

With each outgoing message, we compute the message along each incoming edge. This is done by first summing the weight of the same types of edges, and then concatenating across different types of edges.

$$\mathbf{m}_{j}^{\Delta\leftarrow}=\phi_{\text{receive }}^{\Delta}\left(\bigcup_{\Delta^{\prime}}\bigcup^{\mathcal{E}_{k}\in\mathcal{E}^{\Delta^{\prime^{\prime}\rightarrow\Delta}}}\sum_{\left(i,j,a_{ij}^{k}\right)\in\mathcal{E}_{k}}a_{ij}^{k}\mathbf{m}_{i}^{\Delta^{\prime}\rightarrow}\right)$$

(2)

Finally, given the original input nodes and the received message, we use Gated Relu Unit (GRU) to update the node representations. The updated node vector is used to classify objects and predicts through training and back-propagation.

The model is trained in an end-to-end fashion, and the total loss consists of the classification loss for both objects and predicates:

$$\mathcal{L}_{\text{total }}=\lambda\mathcal{L}_{\text{obj}}+\mathcal{L}_{\text{pred}}$$

where $\lambda$ is a user-defined weight factor. Because a subject can have multiple relationships with an object, we used cross-entropy loss for the $\mathcal{L}_{\text{pred}}$.

The task objective is twofold: to generate scene graphs by (1) predicting the labels of segmented clusters of point cloud, and (2) predicting the labels of relationships between two point cloud clusters. For this, we use the 3RScan[21] dataset, which consists of 1482 scans across 478 different scenes. Each scene is recorded using Google Tango, yielding sequences of RGB-D images with accurately calibrated camera poses. The ground truth scene graph annotations are provided by 3DSSG [12].

The entire dataset is divided based on the number of scans, including 1061 scenes for training, 117 for validation, and 157 for testing. The original dataset contains 534 object classes and 40 relationship classes. The type of relationships captured in the data include supporting relationships (e.g. standing, lying), proximity relationships (e.g. next to, in front of), and comparative relationships (e.g. bigger than, taller than). In this project, we trained and evaluated the algorithm in two different settings. The first setting consists of 160 object classes and 26 predicate classes ($O_{160}R_{26}$), and the second setting consists of 27 object classes and 7 predicate classes($O_{27}R_{7}$).

Scene Graph Spatial Network (SGPN) [12] The network takes as input the object point cloud and the predicate point clouds and uses Graph Neural Network(GNN) to predict the scene graph.

Scene Graph Fusion Network (SGFN) [1] The network uses only contextual vectors as predicate input instead of PointNet encoded feature. It uses Graph Attention Network(GAT) among the triplets for the prediction.

Ours We present two versions of models, one uses internal knowledge graphs, and the other use external knowledge graphs. Both models have the same structure. The only difference is that for the internal knowledge graphs, we initialize the knowledge representation with zero matrices, which allows the model to capture the relationship within the dataset.

The model is implemented in PyTorch. The learning rate is set to 0.001. The input graphs are generated based on the distance between entities (0.5 meters). We use 100K epochs for our models and 300K epochs for SGPN and SGFN. The weight loss factor $\lambda$ is 0.5.

To evaluate the model’s performance on scene graph generation, we compare the predicted Subject-Predicate-Object triplets with the ground truth triplet. Considering that multiple predicates can co-exist between two objects, we employ two measures for evaluation: RE (abbreviation for relationship): This measure evaluates if the predicted triplets exactly match the ground truth. RE_single: This is a relaxed form of evaluation where at least one predicted relationship matches the ground truth.

Most of the current 2D scene graph prediction problems adopt recall rate as the evaluation metric [22, 13, 15, 11]. Note that $Recall=TP/(TP+FN)$, where $TP$ is true positive and $FN$ is false negative. The major reason is that the ground-truth annotations of relationships are likely to be incomplete, and thus using metrics like accuracy or precision, which penalize false positive predictions, is unfair to reflect the actual performance of a model. In the table, we use Obj@K to represent if the prediction for objects exists in top K predictions.

Table II summarizes the qualitative results for our model in comparison with state-of-art models. Our model outperforms in triplet prediction. The results indicate that incorporating knowledge graphs enhances overall performance, with external knowledge graphs offering greater improvements than internal knowledge graphs. Nevertheless, in terms of object classification accuracy, our model does not always give better predictions. This may be attributed to the inherent challenge of classifying objects using point cloud data, which in turn complicates the association between scene graphs and knowledge graphs for objects.

Fig 4 illustrates how the use of knowledge graphs can expedite model convergence. When compared to the SGFN model, our model reaches convergence in earlier epochs, implying a reduced need for training epochs.

We performed error analysis for both object classification and predicate classification. The top five misclassifications for objects, denoted as ground truth/prediction, were wall/curtain, wall/wardrobe, wall/blinds, pillow/cushion and chair/side table. The difficulty in recognizing curtains and blinds was also a common challenge faced during semantic classification for the ScanNet point cloud dataset [23]. Regarding predicate classes, the most frequently occurring mistakes were learning against/close by, cover/lying on, and cover/standing on.