Ensembling Instance and Semantic for Panoptic Segmentation

Panoptic segmentation is a challenging task in the computer vision community which can help solving problems of scene understanding including autonomous driving. The coherent scene segmentation problem can be solved by combining instance and semantic segmentation. The aim for instance segmentation is to first detect and locate all foreground objects by bounding boxes, then for a detected object, assign a category label and an index number on all related pixels of that object. The goal of semantic segmentation is to classify all pixels in an image into different category labels, whereas it does not consider indices of objects. Both tasks have their own limitations for image segmentation. In order to generate a dense and coherent scene segmentation, panoptic segmentation was introduced in [8] combining the strengths of instance and semantic segmentation with the aim to assign a category label and an instance index for each pixel of a given image.

We use MS COCO[9] dataset for this challenge. For image segmentation tasks, MS COCO provides both training and validation data in the form of images and comprehensive annotations for all three segmentation tasks. These data are commonly used by the vision community, and related vision competitions for object detection and segmentation are yearly held.

In our solution to panoptic segmentation, we use a standard approach of combining instance and semantic segmentation as described in [8]. To realize this, we adopted the commonly used Mask R-CNN[6] and the powerful Hybrid Task Cascade (HTC)[2, 3] for instance segmentation. Mask R-CNN extends Faster R-CNN[12] originally designed for object detection by adding a separate branch for mask prediction, thereby predicting pixel level object masks in parallel to predicting the bounding box positions of objects. The HTC framework offers a special cascaded combination of R-CNN and gives the best performance in 2018 COCO object detection challenge¹¹1http://cocodataset.org/##detection-leaderboard, in SEGM Chal18. In semantic segmentation, we adopted DeepLabv3[4] network, which combines concepts of spatial pyramid pooling and dilated convolutions to extract information in the image at different scales and to capture information from a larger effective field of view. In the final combination step, we use the panopticAPI[8] to combine the predictions generated from instance and semantic models to yield the final panoptic results.

For our implementation, we adopted the open source machine learning framework Pytorch[11] as well as the accompanying torchvision library. They are widely used by the vision community, which allows us to quickly experiment with different pre-trained models and network architectures.

Segmentation experiments were conducted on a fully automated cloud pipeline on AWS including training, prediction, evaluation, as well as real time visualization of segmentation results.

In this section, we report our experiment settings and evaluation results. In the first step, we conducted experiments on instance and semantic segmentation using the standard evaluation measure of mean average precision (mAP) and mean Intersection over Union (mIoU), respectively. In the second step, we tested the various combination of them to study how we achieve our best panoptic segmentation results. All the following experiments are trained on dataset train2017, and the reported evaluation results are validated on val2017. No external data is used. All trainings were executed on 8xGPU V100 AWS EC2-instances and testing is done locally on GPU machines with 4x Titan Xp.

We represent various experimental setup in Table 1. Our first setup denoted as Ins1 is the trained Mask R-CNN with ResNet50 backbone, where we get mAP $37.9$ for bounding box (bbox), and mAP $34.6$ for mask. Then, we changed the backbone to ResNet152 in Ins2 and get mAP for bbox $41.3$ and for mask $37.1$. This shows ResNet152 outperforms ResNet50 for Mask R-CNN. Next, we tackle the data imbalance problem in Ins3, and trained three expert models using Mask R-CNN with ResNet152 on three subsets of training data. This yields a further increase in mAP of $0.8$ for bbox and $0.9$ for mask compared to Ins2, which shows that expert models indeed help for the case of data imbalance. Furthermore, with Ins4 strategy, we adopt HTC in our pipeline to improve instance performance. We achieve bbox $50.7$ mAP and mask $43.9$ mAP using X-101-64x4d-FPN backbone.

In the first semantic segmentation experimental setup (Sem1: Table 2), we get mIoU $36.7$ for FCN architecture, which is less than the other setups using DeepLabv3. In Sem2 and Sem3, we test DeepLabv3 with backbones ResNet101 and ResNet152 respectively. Their performance is quite similar, converging to mIoU around $43$. Although the model with the larger backbone (ResNet152) promises better mIoU score, we could not observe an improvement compared to ResNet101 in our benchmarks. Another approach is to ensemble three DeepLabv3 models in Ins4, one with ResNet101, the other two with ResNet152, yielding our highest mIoU of $44.5$. Compared to the best single semantic model, we improve mIoU of 1.1 using ensemble technique.

We now discuss our final panoptic segmentation results by various combination of instance and semantic segmentation. As shown in Table 3, our chosen baseline using the best single model of Mask R-CNN and DeepLabv3 yields PQ of $42.7$ on dataset val2017 and $42.9$ on dataset test-dev. In setup Pan1 we enhanced the baseline by adding expert models in instance segmentation yielding a higher PQ in $43.7$ for both data-sets. Similarly, in setup Pan2 we enhanced the semantic segmentation using ensemble models, but keep instance segmentation unchanged as the baseline. We get $43.2$ and $43.4$ for the two data-sets respectively. This confirms that expert models and ensemble strategy enhanced the baselines in different aspects. Next, we use both enhancements in strategy Pan2 and Pan3 to achieve our highest result for Mask R-CNN and DeepLabv3 with PQ of $44.2$ on both data-sets. Finally, changing the instance framework to HTC, we get our highest PQ for the panoptic segmentation challenge for $46.2$ and $47.1$ on the two data-sets, respectively. This means that HTC not only contributes in instance segmentation but also have a crucial influence on panoptic results.

As an image example to illustrate our panoptic segmentation result, we show in Figure 1 for a comparison between baseline method and enhanced methods. Their segmentation behavior among different categories are roughly similar. However, strategy Pan4 works better than other methods by detecting small parts of object like persons’ leg as shown in Figure 1d.

We achieved PQ $47.1$ for the 2019 COCO panoptic segmentation task using combined approach of instance and semantic segmentation. In our study, we proofed that ensemble techniques and expert models indeed improve panoptic segmentation quality. Albeit our combined segmentation approach is computationally expensive compared to unified approach, it is more flexible in incorporating and improving on recent developments in both instance and semantic segmentation individually. Our future work may include more investigation on expert models, ensemble techniques and more framework or architecture, where we believe that there is still space for improvement in these aspects.