Comprehensive Saliency Fusion for Object Co-segmentation

Object co-segmentation is the task of extracting common objects as foregrounds in a group of images. Compared to the single image segmentation, where it’s difficult to ascertain what’s foreground, we have a clear-cut definition of what foreground we wish to extract in the object co-segmentation task. Rother $et.al.$ [1] first introduced the concept of co-segmentation by developing a histogram matching method to extract common parts from a pair of images. However, Vicente $et.al.$ [2] were the first to propose that co-segmentation should be about things (or objects) and there is a need to incorporate a measure of objectness in the models. They proposed a Random Forest classifier to find the similarity between a pair of proposed segmentation of objects in each image of a group followed by the A* search algorithm [3] to find the segmented objects with maximum similarity score. The major applications of object co-segmentation include image grouping[4], object recognition[5], and object tracking[6] which are among the fundamental tasks of Computer Vision .

The problem of object co-segmentation is highly correlated to finding the common regions of interest in a group of images. [7, 8] approach the problem of co-segmentation by trying to highlight these common regions of interest. [9] proposed a shortest path algorithm between the salient regions of images, [10] found saliency maps and the computed dense correspondence for all the similar images for obtaining matching scores for pixels across the images. Geometric Mean Saliency (GMS) [7] uses dense SIFT Correspondence [11] for each pair of images to align saliency maps and fuses them. [8] extended [7] by making it more efficient through introducing the key image concept. [12] use saliency prior and superpixels to partition images into foreground and background. They calculate several feature level descriptors on superpixel level and organized in Bag of Words (BoW) to perform clustering of common foregrounds. All of these use single saliency source. Rather than going through the same trend, we propose the usage of multiple saliency sources to benefit from all of them.

Notations and Overview: Let’s consider a group of $m$ images $\mathcal{I}=\{I_{1},I_{2},…I_{m}\}$, which is divided into $K$ sub-groups, following [7]. If $V_{i}$ denotes the sub-group label of $I_{i}$, we can define $k^{th}$ sub-group of images as $G_{k}=\{I_{i}|V_{i}=k\}$, where $k\in\{1,\cdots,K\}$. Let’s denote the cluster representative of $k^{th}$ cluster with the notation $g_{k}$.

As far as saliency maps are concerned, let’s denote $j^{th}$ saliency map of $i^{th}$ image as $S^{j}_{i}$, where $j\in\{1,\cdots,L\}$, if we consider $L$ saliency sources. Let the warping function between two images, say $I_{u}\rightarrow I_{v}$, be denoted as $W^{I^{v}}_{I_{u}}$.

In Fig. 1, we show how different fused saliency maps are generated in a sub-group and eventual object co-segmentation masks are obtained. The core idea is to fuse the saliency maps available in a sub-group through appropriate alignments, which are of two types: centralized and decentralized. The centralized and decentralized alignments lead to fused saliency maps for the key image and member images, respectively. Let’s discuss all this in detail.

Sub-group Formation: In a given group of images, there may be lot of shape and pose variance across the images, which is not ideal for saliency fusion. Also, we need not process all of them together. We must note that an image needs only a couple of other neighboring images to carry out object co-segmentation. Hence, we divide the group into $K$ sub-groups via k-means clustering on global features extracted using ResNet50 pre-trained model (on ImageNet dataset[21]). We perform silhouette analysis [22] to find the best $K$. We also identify the images nearest to the cluster centers as key images.

Saliency Map Generation: Current saliency fusion-based co-segmentation approaches mostly utilize a single saliency source. In contrast, we extract four saliency maps (which means $J=4$) using different deep saliency detection methods, namely PoolNet [23], EGNet [24], BASNet [25], U2Net [26]. The main motivation behind using multiple methods is to benefit from all the sources, instead of just one.

Saliency Fusion: Our proposed framework has two stages: (i) centralized alignment, and (ii) decentralized alignment.

For generating fused saliency map of the key image, we align all the available saliency maps in the sub group to the key image. We call this centralized alignment. For making alignments, we use the pre-trained DGC-Net [19] model throughout our work. Such learning-based models provide us excellent correspondences between the pixels from two different images, which can be used to align saliency maps as well, not just images. First let’s collect all the candidate saliency maps for the key-image in a sub-group after proper alignment, as given below:

$$C_{k}=\{W^{g_{k}}_{I_{i}}(S_{i}^{j})|I_{i}\in{G_{k}},j\in\{1,\cdots,L\}\}$$

(1)

where $C_{k}$ denotes the collection. There will be a total of $L*|G_{k}|$ candidate saliency maps in the sub-group $G_{k}$ for the key image $g_{k}$. Here $|\cdot|$ denotes cardinality. Since all these candidate saliency maps will be of same size, because of aligning to one image, the key-image, we can denote the collection of candidate saliency values of a pixel $p$ as $C_{k}^{p}$. We fuse all these candidate values to generate fused saliency map $F_{g_{k}}$ for the key image $g_{k}$ in the following way:

$$F_{g_{k}}(p)=median(C_{k}^{p})$$

(2)

where we are basically finding the median of all the candidate saliency values for any pixel $p$. Having obtained the fused saliency map for the key-images, to obtain such map for sub-group members, we perform what we call decentralized alignment, i.e., aligning fused saliency map of key image to the member images. Note here that, in the centralized alignment discussed earlier, the alignment was opposite, i.e. aligning saliency maps of member images to the key image. Once we know the key-image to member image correspondences, we can align the fused saliency obtained for the key image to the member images. For instance, we can obtain fused saliency map for a member image $I_{i}$ in the following way:

$$F_{I_{i}}=W^{I_{i}}_{g_{V_{i}}}(F_{g_{V_{i}}})$$

(3)

where $F_{I_{i}}$ denotes the fused saliency map for image $I_{i}$.

Object Co-segmentation Masks: Having obtained fused saliency map for every image, we now need to generate their object co-segmentation masks. For that, we first apply the Otsu algorithm on those maps to generate foreground and background seeds. These seeds are then passed along with the image to Grabcut [27] algorithm to extract our expected foreground, the shared object.

We conduct all our experiments on three standard benchmark datasets of object co-segmentation research, namely MSRC, iCoseg and Internet Images. The most common evaluation metrics used for this task in literature are Jaccard Similarity Score ($\mathcal{J}$) and Precision Score ($\mathcal{P}$). We use both of them. Jaccard Similarity is defined as the intersection over union (IoU) of the resultant co-segmentation mask and groundtruth mask whereas Precision is used to measure the percentage of correctly labeled foreground and background pixels.

We use the standard subset [10] of MSRC with classes cat, bird, dog, sheep, car, plane and cow, where each class contains 10 images. It can be seen from Table I that our proposed approach obtains the best $\mathcal{J}$ score and the second best $\mathcal{P}$ score. We provide our sample qualitative results on this dataset in Fig. 5.

The iCoseg dataset [36] was proposed initially for the task of interactive co-segmentation but is often used to analyze the performance of automatic co-segmentation approaches as well. It contains 643 images belonging to 38 classes and is sufficiently challenging due to multiple objects and varying camera angles. It can be seen from Table I that our proposed approach gives the best $\mathcal{J}$ and $\mathcal{P}$ scores among all the methods. We provide a few sample qualitative results of our method on this dataset in Fig. 5.

Discussion: In Table III, we show the efficacy of using multiple saliency sources by comparing with the cases when there is only single saliency source available, i.e. $L=1$. It can be seen that our proposed approach performs better than all the cases involving just one saliency source. It clearly demonstrates that using multiple saliency sources is indeed beneficial. In Fig. 5, we also demonstrate how the proposed method performs better than individual saliency methods, if we were to simply apply Otsu algorithm followed by GrabCut on them, without performing any type of fusion.

We propose a comprehensive saliency fusion framework for performing object co-segmentation. In this framework, unlike previous saliency fusion-based methods, we rely on multiple saliency sources and use deep learning based saliency extraction and fusion processes. Our experimental analysis indicates that we are able to perform better than several state-of-the-art methods.

This work was supported by Infosys Centre for Artificial Intelligence, IIIT-Delhi, and IIIT-Delhi PDA Grant.