Visual Object Recognition in Indoor Environments Using Topologically Persistent Features

For TDA, data is often represented by cubical or simplicial complexes depending on the type of data. Images can be considered as point clouds by treating every pixel as a point in $\mathbb{R}^{2}$. Such a point cloud is commonly represented using a simplicial complex. However, since images are made up of pixels, they have a natural grid-like structure to them. Therefore, they are more efficiently represented as cubical complexes in various ways [23, 21].

A cubical complex in $\mathbb{R}^{n}$ is a finite set of elementary cubes aligned on the grid $\mathbb{Z}^{n}$, where an elementary cube is a finite product of elementary intervals with dimension given by the number of its non-degenerate components [24]. An $n$-dimensional image is a map $\mathcal{I}:I\subseteq\mathbb{Z}^{n}\longrightarrow\mathbb{R}$. A voxel is an element $v\in I$, and its value $\mathcal{I}(v)$ is the intensity. When $n=2$, the voxel is known as a pixel, and the intensity is known as the grayscale value. While there are various ways of constructing a cubical complex from an $n$-dimensional image, Garin et al. [21] adopt a method in which an $n$-cube represents a voxel, and all the adjacent lower-dimensional cubes (faces of the $n$-cube) are included. The values of the voxels $v$ are extended to all the cubes $\lambda$ in the resulting cubical complex $C$ as

After the complexes are generated, a filtration is constructed, as described next.

For any cubical (or, simplicial) complex $K$, let $K_{i}$ denote the $i$-th sublevel set of $K$. A filtration is then defined as a collection of complexes $\left\{K_{s}\right\}_{s\in\mathbb{R}}$ such that $K_{s}$ is a subcomplex of $K_{t}$, for each $s\leq t$.

The pixel values in a grayscale image embed a natural filtration, which are used to obtain the sublevel sets of the corresponding cubical complex. The $i$-th sublevel set of the cubical complex $C$ is then obtained as

which defines the filtration as $\left\{C_{i}\right\}_{i\in\mathcal{I}}$. For binary images, various functions known as descriptor functions (or, filtration functions) are used to construct such grayscale images [21]. Similarly, filtration functions are also defined for generating filtrations from point cloud data or mesh data [19].

Persistent homology is a common tool in TDA that studies the topological changes of the sublevel sets $C_{i}$ as $i$ increases from $(-\infty,\infty)$. During filtration, topological features, interpreted as $h$-dimensional holes, appear and disappear at different scales, referred to as their birth and death times, respectively. This information is summarized in an $h$-dimensional persistence diagram (PD). An $h$-dimensional PD is a countable multiset of points in $\mathbb{R}^{2}$. Each point $(x,y)$ represents an $h$-dimensional hole born at a time $x$ and filled at a time $y$. The diagonal of a PD is a multiset $\Delta=\left\{(x,x)\in\mathbb{R}^{2}|x\in\mathbb{R}\right\}$, where every point in $\Delta$ has infinite multiplicity.

Several other stable representations of persistence can be obtained from a PD [25, 26]. One such representation is the Persistence Image (PI), a stable and finite dimensional vector representation of persistent homology [26]. To obtain a PI, an equivalent diagram of birth-persistence points, i.e., $(x,y-x)$, is computed. The birth-persistence points are then regarded as a sum of Dirac delta functions, which are convolved with a Gaussian kernel over a rectangular grid of evenly sampled points to compute the PI.

To generate the object segmentation maps for an input scene image, we follow a foreground segmentation method that is similar to the one proposed in [27]. In particular, we use the state-of-the-art DeepLabv3+ architecture [28], which is pre-trained on a large number of classes and is hypothesized to have a strong representation of ’objectness’. Subsequently, we train the network using pixel-level foreground annotations for a limited number of images from our datasets.

A shape-based object recognition method relies on the objects’ contours for distinguishing among multiple objects. However, the number of foreground pixels is very low as compared to the background when segmentation is performed on images taken from distances as large as two meters. Hence, it is difficult for a segmentation model to capture the minor details in the objects’ shapes in a single shot. Therefore, we employ a two-step segmentation framework to preserve the objects’ contours details. In the first step, the segmentation model predicts a segmentation map for the input scene image. Contour detection is performed on this scene segmentation map to obtain the bounding boxes for the objects. These bounding boxes are used to divide the scene image into multiple sub-images, each of which contains a single object. In step two, these sub-images are fed to the same trained segmentation model for predicting the individual object segmentation maps.

Segmentation maps are essentially binary images comprising only black and white voxels. A grayscale image, suitable for building a filtration of cubical complexes, can be generated from such binary images using various filtration functions. We select a commonly used filtration function, known as the height function, which computes a sufficient statistic to uniquely represent shapes in $\mathbb{R}^{2}$ and surfaces $\mathbb{R}^{3}$ in the form of the persistent homology transform [29].

For the cubical complexes in our case, we define the height function as stated in [21]. Consider a binary object segmentation map $\mathcal{B}:I\subseteq\mathbb{Z}^{n}\longrightarrow\left\{0,1\right\}$, a grayscale image $\mathcal{H}:I\longrightarrow\mathbb{R}$ for the segmentation map $I$, and a direction $p\in\mathbb{R}^{n}$ of unit norm. The values of all the voxels of $\mathcal{H}$ are then reassigned as

Here, $\langle v,p\rangle$ is the distance of voxel $v$ from the hyperplane defined by $p$ and $H_{\infty}$ is the filtration value of the voxel that is farthest away from the hyperplane.

In step three, we obtain $d$ such grayscale images for each object segmentation map by considering $d$ directions that are evenly spaced on a unit $1$-sphere $\mathbb{S}^{1}$. We construct cubical complexes from each grayscale image according to Eq. (1). The sublevel sets of the $d$ complexes are then computed according to Eq. (2) to obtain $d$ filtrations. Persistent homology is applied to these filtrations to obtain $d$ persistence diagrams (PDs) for every object segmentation map. We only consider $0^{th}$ order homology for generating the PDs. We investigate the performance of two types of persistent features from the generated PDs. The following subsections III-B1 and III-B2 describe their computation details.

We first choose the widely used MPEG-7 Shape Silhouette dataset to solely characterize the shape recognition capability of the two persistent features-based networks for (almost) ideal object segmentation maps. In particular, we use a subset of this dataset, namely, the MPEG-7 CE Shape 1 Part B dataset, which is specifically designed to evaluate the performance of 2D shape descriptors for similarity-based image retrieval [31]. It includes the shapes of 70 different classes and 20 images for each class, for a total of 1,400 images. Fig. 3 shows sample images from the dataset.

The shapes in the MPEG-7 are detailed and fairly distinguishable from each other. However, common objects in indoor environments are often less detailed and, therefore, more challenging for topological methods. Most deep learning models work exceptionally well in recognizing such everyday objects in their training environments. However, they face challenges when used in new (previously unseen) environments without any retraining, even if the objects remain the same. Therefore, we choose the widely used benchmark, the RGB-D Scene Dataset v1[32], to evaluate the performance of our proposed methods. The dataset consists of eight scene setups with objects that belong to the RGB-D Object Dataset[32]. The scenes are shot in five different environments: desk, kitchen_small, meeting_small, table, and table_small. In particular, we select the table_small and desk environments for our evaluation. Among all the objects present in the scenes, we consider six object classes for recognition corresponding to the six objects types that appear in both the environments. For our current analysis, we only consider RGB images where the objects are not occluded.

While the RGB-D Scenes Dataset consists of scenes from multiple environments, all of them are tabletop environments with limited variation in terms of lighting conditions and object types. On the other hand, the dataset in [33] provides images with several objects but only in a single environment. Although there are quite a few other indoor scene datasets in the literature, none of them includes a large enough set of object types, poses, and arrangements with varying backgrounds and lighting conditions. Therefore, we introduce a new RGB dataset, which we call the UW Indoor Scenes (UW-IS) dataset, for evaluating object recognition performance in multiple, typical indoor environments. For this purpose, we select a fixed set of fourteen different objects from the benchmark Yale-CMU-Berkeley (YCB) object and model set [34].

The UW-IS dataset consists of indoor scenes taken in two completely different environments. The first environment is a living room scene where the objects are placed on a tabletop. The second environment is a mock warehouse setup where the objects are placed on a shelf. For the living room environment, we have a total of 347 scene images. The images are taken in four different illumination settings from three different camera perspectives and varying distances up to two meters. Sixteen out of the 347 scene images are with two different objects, 135 images are with three different objects, 156 images are with four different objects, and 40 images are with five different objects. For the mock warehouse environment, we have a total of 200 scene images taken from distances up to 1.6 meters. Sixty out of 200 images are images with three different objects, 68 images with four different objects, and 72 images are with five different objects. Fig. 4(a) shows some sample living room scene images, and Fig. 4(b) shows sample images from the mock warehouse environment. Fig. 4(c) shows all the fourteen objects used in our dataset. The dataset is publicly available at https://data.mendeley.com/datasets/dxzf29ttyh/.

We perform five-fold training and testing on all images of the MPEG-7 Shape Dataset. For evaluation on the RGB-D Scenes dataset, we use the table_small environment for both training and testing, and the desk environment only for testing. For the UW-IS dataset, we use the living room images for training and testing, and the mock warehouse images only for testing. All the training and testing is performed using GeForce GTX 1080 and 1080 Ti GPUs on workstations running Windows 10 and Ubuntu 18.04 LTS, respectively. The code for the proposed methods is available at https://github.com/smartslab/objectRecognitionTopologicalFeatures.

For the MPEG-7 Shape Dataset, we divide the 1,400 images into five sets of 280 images each, with four images of each class included in every set. We perform five-fold training and testing using these sets, such that each set is used once as a test set while the remaining four sets are used for training and validation. We use the giotto-tda [35] library to generate the PDs and the Persim package in Scikit-TDA Toolbox to generate the PIs. The PDs are generated using height functions along 8 evenly spaced directions on $S^{1}$. We choose a grid size equal to 50 $\times$ 50, a spread of 10, and a linear weighting function for generating the PIs. We use three-layered and five-layered fully connected networks for recognition using amplitude features and sparse PI features, respectively. We use the ReLU activation (last layer uses softmax activation), Adam optimizer and categorical cross-entropy loss function for training. The learning rate is set to 0.01 for the first 500 epochs. It is decreased by a factor of 10 after every 100 epochs for the next 200 epochs, and by a factor of 100 for the last 100 epochs.

For the RGB-D Scenes dataset and the UW-IS dataset, we use the DeepLabv3+ segmentation model with Xception-65 backbone following the implementation in [36] to obtain object segmentation maps. The network is initialized using a model pre-trained on the ImageNet and PASCAL VOC2012 datasets available from [36]. We use 147 images from the table_small environment to train the network for the RGB-D Scenes dataset and 200 living room images to train the network for the UW-IS dataset. Horizontally flipped counterparts of the images are also used for training the models. We generate the corresponding foreground annotations using LabelMe [37]. We train the network for 20,000 steps using the categorical cross-entropy loss with 10% hard example mining after 2,500 steps in the case of the RGB-D Scenes dataset. For the UW-IS dataset, we use 1% hard example mining¹¹1Unlike the UW-IS dataset, the smaller training set for the RGB-D Scenes dataset leads to poor quality object segmentation maps that are unsuitable for extracting topological features. Therefore, we use a separate DeepLabv3+ model for Step 2, which is fine-tuned on images of individual objects obtained from the original scene images.

We then perform five-fold training and testing of the proposed methods on both datasets by diving the scene images from the training environment into five folds. The resulting five models are also tested separately on all the images of the test environment. First, the trained DeepLabv3+ segmentation models are used to obtain object segmentation maps from all the images. To ensure the quality of segmentation maps remain consistent across training and test environments, we generate object segmentation maps for the test environments using segmentation models fine-tuned in these environments. We then appropriately pad all the segmentation maps with zeros and consistently resize them to obtain 125 $\times$ 125 binary images. We augment the training data by rotating every object segmentation map by $90^{o}$, $180^{o}$, and $270^{o}$. The PDs and PIs are generated in the same manner as for the MPEG-7 dataset, except that the spread value is chosen to be 20. We also use the same fully connected network architectures and hyperparameters as for the MPEG-7 dataset.

Comparison with deep learning-based object recognition methods: To solely characterize the recognition performance of the topologically persistent features, we compare their performance with features extracted using two other widely used object recognition methods, namely, ResNetV2-56[1] and EfficientNet-B4[2], on the RGB-D Scenes benchmark. We use the same sets of cropped objects obtained using DeepLabv3+ in Step 1 of our proposed method for five-fold training and testing of both the methods. The networks for both the methods are trained using the implementations available in Keras [38].

Comparison with end-to-end object detection methods: We also compare the overall performance of both the persistent features-based methods against end-to-end object detection methods on both the RGB-D Scenes dataset and the UW-IS dataset. Particularly, we compare performance against Faster R-CNN[3], a widely used object detection method, and its state-of-the-art variant for cross-domain object detection, Domain Adaptive Faster R-CNN [12]. For Faster R-CNN (referred to as FR-CNN), we use a pre-trained model with InceptionResNet-V2 feature extractor and hyperparameters available with the TensorFlow Object Detection API[39]. For Domain Adaptive Faster R-CNN, we use a modified implementation from [40], where the VGG backbone is replaced with ResNet-50, and the RoI-pooling layer is replaced with the more popular RoIAlign. The modified implementation has been shown to outperform other state-of-the-art methods for cross-domain object detection[40]. We call this improved method DA-FR-CNN*. Similar to the proposed methods, we perform five-fold training and testing of these methods on both datasets. In the case of DA-FR-CNN* for the UW-IS dataset, all the training set images with artificial lighting (e.g., bottom row in Fig. 4(a)) are used as the source domain, while all the images with natural lighting (e.g., top row in Fig. 4(a)) are used as the target domain. Since such a split is not possible in the RGB-D Scenes dataset, we randomly divide the images into the source and target domains. The ground truth bounding boxes for both the datasets are generated using LabelImg[41].

We first examine the recognition performance of both amplitude features and sparse PI features on the MPEG-7 dataset. We use the weighted F1 score, weighted precision, weighted recall, and accuracy for evaluating performance. Table I shows the test-time performance of the trained recognition networks. We observe that sparse PI-based recognition performance is quite impressive (more than 0.85) with respect to all the four metrics. It is also consistently better than amplitude-based recognition, indicating the usefulness of sparse sampling in selecting the key features. It is worth noting that 100% recognition performance using 2D shape knowledge is not possible for this dataset, since some of the classes contain shapes that are significantly different from the others in the same class but are similar to certain shapes in other classes [31].

We then examine the performance²²2We do not account for the false negatives and false positives resulting from errors of the segmentation model to ensure a fair judgement of our methods’ effectiveness. Equivalently, for object detection methods, false positives arising from incorrect region proposals are also not considered. of both the persistent features-based methods along with other object recognition and detection methods on the RGB-D Scenes benchmark. Table II reports the performance of all the methods (trained in the table_small environment) on the unseen desk environment. We observe that sparse PI-based recognition, which achieves an overall accuracy of $0.71\pm 0.01$, has the highest performance among all the methods in terms of recall and accuracy. Particularly, for the same set of object images obtained using the DeepLabv3+ model, recognition using sparse PI features computed from object segmentation maps is better than recognition using features learned by ResNetV2-56 and EfficientNet-B4 from RGB inputs with respect to accuracy, recall, and F1-score. Additionally, amplitude features-based recognition is comparable to both ResNetV2-56 and EfficientNet-B4. Additionally, the difference between the recognition performance using sparse PI features and amplitude features is, however, much lower than that for the MPEG-7 dataset. We also note that the sparse PI features-based method outperforms Faster R-CNN (referred to as FR-CNN) substantially and DA-FR-CNN* by a small margin in terms of accuracy and recall.

We then assess the performance of both the persistent features-based methods, FR-CNN, and DA-FR-CNN* on the living room scene images from the UW-IS dataset reported in Table III. We observe that both the persistent features-based methods, which only use the information in object segmentation maps, perform reasonably well. Sparse PI-based recognition achieves an overall accuracy of $0.71\pm 0.01$ and is marginally better than amplitude-based recognition, whose accuracy is $0.69\pm 0.01$. Moreover, both FR-CNN and DA-FR-CNN*, which are trained on this environment and use RGB images as inputs, outperform these methods with respect to all the metrics including class-wise F1 scores.

We believe that the quality of the object segmentation maps has substantial impact on the performance of the persistent features-based methods. Notably, performance is considerably worse for the fork as compared to the other objects. To assess this impact, we compare the recognition performance of both these methods with that of a human given an identical set of segmentation maps. Table IV summarizes the comparison results. For the persistent features based-methods, we only report the accuracy for those images where the human recognizes the object correctly. We observe that a human achieves an accuracy of $0.84\pm 0.01$, which is lower that FR-CNN performance, and finds it difficult to recognize the objects based on the generated segmentation maps, especially for the spoon and fork classes. We refer the reader to Section VI for further discussion on segmentation map quality.

Despite this challenge, we expect recognition performance to be relatively unaffected, when the environments vary considerably but the objects are identical, provided the quality of the segmentation maps remains consistent. Accordingly, we test the performance of the persistent features-based methods on all the warehouse scene images of the UW-IS dataset without retraining the recognition networks. We also test the performance of FR-CNN and DA-FR-CNN* on all the warehouse images without any retraining. Table V summarizes the performances of all the four methods on the mock warehouse test environment. We observe that the performance of sparse PI-based recognition is unchanged from the living room scenario, and is substantially better than that of amplitude-based recognition, whose accuracy reduces by 4%. However, the performance of FR-CNN degrades a lot without fine-tuning (accuracy drops by 25%). The performance of DA-FR-CNN* also degrades considerably (accuracy drops by 18%). Similar to the RGB-D Scenes benchmark case, the sparse PI features-based method outperforms FR-CNN substantially and DA-FR-CNN* by a small margin with respect to accuracy and recall.

We also implement our proposed framework on a LoCoBot platform built on a Yujin Robot Kobuki Base (YMR-K01-W1) and powered by an Intel NUC NUC7i5BNH Mini PC. We mount a ZED2 camera with stereo vision on top of the LoCoBot and control the robot using the PyRobot interface [42]. The camera images are fed to the trained segmentation model and recognition networks, which are run on an on-board NVIDIA Jetson AGX Xavier processor, equipped with a 512-core Volta GPU with Tensor Cores and 8-core ARM v8.2 64-bit CPU. We use TensorRT [43] for optimizing the trained segmentation model. Fig. 5 shows a screenshot of the platform. The sparse PI features-based recognition runs at a speed of $1.1s$ per frame on this platform. A video demonstration of object recognition on this platform is included in the Supplementary Materials.