Towards Stable Co-saliency Detection and Object Co-segmentation

Salient object detection (SOD) is a fundamental task in computer vision, which is derived with the goal of detecting and segmenting the most distinctive objects from visual scenes. Over the past decades, a large amount of SOD algorithms have been developed, which can be roughly classified into traditional methods and deep learning based methods. Traditional models [28, 29, 30, 31, 32] detect salient objects by utilizing various heuristic saliency priors with hand-crafted features. Deep learning based models use various feature enhancement strategies to improve the ability of localization and awareness of salient objects [33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44], or take advantage of edge features to restore the structural details of salient objects [45, 45, 46, 47, 48]. Different from the above methods, some methods [49, 50, 51] consider leveraging predict-refine architecture or the fixation prediction framework [52, 53] to generate fine salient objects. Beyond of the scope of the paper, more detailed introduction of salient detection can be referenced in recent surveys [54, 9].

Compared to SOD task, CoSOD needs to model inter-saliency relations among an image group. Therefore CoSOD task is more challenging than saliency detection. To model inter-saliency relations between an image group, conventional approaches [55, 56, 57, 2, 58, 59, 60, 61] utilize handcrafted features, such as color, texture and SIFT descriptors etc., and these methods rely on researcher’s prior knowledge to model the interaction between the group images, like inter-image saliency. However, low-level features and fixed hand-designed interaction models are too subjective to face the multiple challenges including background clutter, appearance variance of co-salient object across images, and similarity between co-object and non-common object, etc. Recently deep-based models simultaneously explore the intra-saliency and inter-image consistency in a supervised manner with different approaches, such as concatenation operation [15, 18], graph convolution networks (GCN) [62, 20, 21], self-learning methods [63, 64], co-clustering [65] or Transformer-based methods [66]. However, the main drawback of these methods is that they suffer from sub-group instability. While the size of each group is not fixed in real-world scenarios as well as the experimental co-saliency dataset, so only partial inter-saliency relationships will be captured, limiting the robustness of the model. To address this problem, some methods design sophisticated modules, like gradient feedback [23], correlation techniques [22], to adjust unfixed number of images in each image group. In general, these two methods simply use the adding (fusion) to generate final inter-saliency relations from sub inter-saliency relations. While the location of co-salient region and noisy region vary in differnet image, only simply adding operation cannot make fully interaction between differnet images and retain co-salient regions. RCAN [14] proposes RCAU mechanism, which can better suppress non-salient background and retain co-salient regions. As we know, recurrent architecture makes the network sensitive to input orders. In this paper, we propose MSRU to make our proposed network stable and be less sensitive to input orders. For more about CoSOD tasks, please refer to [8, 9, 10].

The concept of co-segmentation was first introduced by the Rother [67], who used histogram matching to simultaneously segment out the common object from a pair of images. Following this work, a number of researchers have made further efforts to develop more effective object co-segmentation models by comparing foreground color histograms [68] or adopting more diverse features like Gabor filters [69] and SIFT [70]. In order to better explore the correspondence relationship among common objects, some existing methods [71, 72, 73, 74, 75, 76] additionally introduced prior constraints to better distinguish them from the undesired image backgrounds. However, these methods cannot obtain robust performance in real-world scenarios, where the handcrafted low-level features are too subjective to face the multiple challenges including intra-class variations and background clutters and the predefined prior knowledge cannot always provide adequate and precise constraint on the common objects.

Recent researches [77, 78, 79] use deep visual features to improve object co-segmentation and they also try to learn more robust synergetic properties among images in a data driven manner. Yuan [79] introduced a DNN-based dense conditional random field framework for object co-segmentation by cooperating co-occurrence maps which are generated using selective search [80]. Hsu [77] proposed a DNN-based method which uses the similarity between images in deep features and an additional object proposals algorithm [81] to segment the common objects. These methods achieved state-of-the-art results by substituting the features learned by DNN for engineered features. However, as feature learning and object segmentation are somehow separated in these approaches, the learned features are not tailored for segmenting the co-occurring objects, resulting in suboptimal performance. The very recent works [82, 83] proposed end-to-end deep learning methods for co-segmentation by integrating the process of feature learning and co-segmentation inferring as an organic whole. By introducing the correlation layer [83] or a semantic attention learner [82], they can utilize the relationship between the image pair and then segment the co-object in a pairwise manner. However, their siamese network structures limit their use of group-wise information which contains more sufficient synergetic relationships than image pairs. Consequently, co-segmenting common objects from image pairs has very limited robustness and practical application value when extending beyond pairwise relations. Unlike the previous methods, by introducing the recurrent architecture, our co-segmentation network is able to make use of all available information including individual image properties and the group-level synergetic relationships to meet the need for real-world applications. Recently, method [11] introduces region correspondence module which can help the network handle unfix input, while the drawback of this work is it omits how to make the model robust to different orders. One closely related topic to object co-segmentation is co-saliency detection, which aims at generating co-saliency maps for each of the images from the given image collection to highlight the common and salient objects. Compared with co-saliency detection, object co-segmentation only aims at segmenting the co-occurring objects without constraining those objects to be (co-)salient. By altering the datasets, we are able to use one network to address both CoSOD and CoSEG problems simultaneously in this paper. By the way, we find that the existing method [84] also tries to design a unified network to address these two problems. More details can be seen in Section.V and Section.VI.

RNN have been widely used in NLP(e.g., [25]) and speech recognition (e.g., [85]) to understand sequence data. The most popular variants of RNN included Long Short-Term Memory (LSTM [86]) and Gated Recurrent Unit (GRU [27] ). The common LSTM unit is composed of a cell and three gates (forget gate, input gate and output gate), which is designed to be capable of learning long term dependencies. Recently, the RNN (especially the LSTM and GRU) has been introduced in spatiotemporal tasks (known as convolutional RNN) for precipitation nowcasting [87], pattern recognition [88, 89], trajectory prediction [90], medical image analysis[91, 92, 93] and video saliency detection [94, 95, 26], et cetera. However, all these works process sequential data, so the variation between different images is little. While in co-saliency detection task, the location of co-salient objects varies greatly in different images. And the different gates in LSTM or GRU cannot handle these complex variations. To address this problem, RCAN propose co-attention mechanism to well model these inter-relations. However, the main drawback of the RCAN is that it makes the network sensitive to input orders. Hence, in this paper, we further design a MSRU to address order-sensitive problem.

The overall architecture of the proposed approach is illustrated in Figure.2. Co-saliency detection aims at discovering the co-occurring object masks $\mathcal{M}=\{M_{n}\}^{N}_{n=1}$ from a group of $N$ relevant images $\mathcal{I}=\{I_{n}\}^{N}_{n=1}$. For an input image group with an arbitrary size, our network first uses CNN to extract the semantic features of all images. Then the single image representation (SIR) branch processes each image individually to learn the intra-saliency $\{S_{n}\}^{N}_{n=1}$. Meanwhile, the multi-path stable recurrent unit (MSRU) recurrently explores all images in the image group to learn the robust group representation $G_{N}$. Finally, the outputs of these two branches are further fused through a non-local fusion module for robust co-saliency detection. The loss function of our proposed network contains co-saliency loss and cross-order contrastive loss (COCL).

As a basic rule in co-saliency detection, it is important to learn the unique properties of each image to capture potential co-occurring objects in the individual image. For each image $I_{n}$ in the input group $\mathcal{I}$, we first use a pre-trained VGG16 [96] to extract semantic features. Following [34, 46], we connect another side path to the last pooling layer in VGG-16. Hence, we obtain six side features Conv1-2, Conv2-2, Conv3-3, Conv4-3, Conv5-3 and Conv6-3 from the backbone network. Before sending the side features extracted from VGG to the SIR, we first use a $1\times 1$ convolutional operation to reduce their channel numbers for saving computation. In this paper, we set channel number $C=64$ suggested by works [47, 97]. For simplicity, we name side feature Conv6-3 as ${X_{n}}\in\mathcal{R}^{C\times H\times W}$. So the side features of image group can be written as: $\mathcal{X}=\{X_{n}\}^{N}_{n=1}$. Then we construct the SIR block on $\mathcal{X}=\{X_{n}\}^{N}_{n=1}$, to capture the intra-saliency $\mathcal{S}=\{S_{n}\}^{N}_{n=1}$ for each image. The SIR contains three convolutional blocks, and each block contains a $3\times 3$ convolutional operation with $stride=1$, followed by a batch normalization and a ReLU activation. Note that we capture intra-saliency and inter-saliency relations on three levels of backbone network (Conv4-3, Conv5-3 and Conv6-3). In this paper, we only show the intra-saliency learning and inter-saliency relations capturing in Conv6-3 for simplicity.

As images within an image group are contextually associated with each other in different ways such as common objects, similar categories, and related scenes, learning a robust group representation which contains the relevance and interaction between group images is extremely important for co-saliency referring. Our previous work RCAN [14] proposes to use a RCAU to learn the group representation for an arbitrary size group $\mathcal{I}$. Since the main drawback of the previous RCAU is that it makes the network sensitive to input orders, leading to unstable training and inferring procedures. To address this problem, we propose a new multi-path stable recurrent unit (MSRU, Figure.3) which can collect features from different orders for final group features generation. Next, we will describe the MSRU in detail.

The feature representations of an image sequence that contains $N$ images are written as:

$$\{X_{1},X_{2},...,X_{N}\}.$$

(1)

Previous RCAU gradually update each single image feature representation to the final group feature, and the performance is easily influenced by the order of input images. An intuitive idea to solve this problem is to generate all the different orders of an image group, and then concatenate group features from these different orders for final group representations. However, this idea is impractical because totally $N!$ orders are generated. Therefore, we propose a compromise approach in this paper. Specifically, the original sequence is first divided into several sub-groups by slide window with $stride=1$:

$$\{\{X_{1},X_{2},X_{3}\},...,\{X_{N-1},X_{N},X_{1}\},\{X_{N},X_{1},X_{2}\}\}.$$

(2)

Here the window size is set as 3, and we treat the image sequence as a cycle. Hence, the group features of each sub-group is denoted as $\{G_{n}^{s}\}_{n=1}^{N}$, which can be written as:

$$\{G_{1}^{s},...,G_{N-1}^{s},G_{N}^{s}\}.$$

(3)

In this paper, we first generate multi-path feature representations of each sub-group $G_{n}^{s}$ from three different orders, then use $\{G_{n}^{s}\}_{n=1}^{N}$ to generate the final group feature $G_{N}$ (Figure.2).

To generate the multi-path group feature representations of each sub-group $G_{n}^{s}$, we first propose a MSRU, which contains dummy orders mechanism (DOM) and a recurrent unit (RU). As can be seen in Figure.3, if a sub-group $G_{N}^{s}$ only contains three images $\{X_{N},X_{1},X_{2}\}$, the DOM would generate three different orders { $``X_{1}\to X_{2}\to X_{N}"$, $``X_{2}\to X_{N}\to X_{1}"$, $``X_{N}\to X_{1}\to X_{2}"$} of this group. Then we propose a RU to generate three group feature representations of these three different orders. Finally, we concat these three group feature representations to achieve multi-path group feature representations of this sub-group $G_{N}^{s}$.

Before introducing our proposed RU, we make some modifications on Eq.2 and Eq.3, which can help readers better understand the process of RU. In Eq.2, we can get several sub-groups, and rewrite their notations as:

$$\{\{X_{(1)1},X_{(1)2},X_{(1)3}\},...,\{X_{(N)1},X_{(N)2},X_{(N)3}\}\}.$$

(4)

The images in each sub-group can be written as $X_{nk}$. $X_{nk}$ means the feature of $k$-th image in $n$-th sub-group, where $k\in[1,3]$ and $n\in[1,N]$ in this paper. So the group feature of one order belong to each sub-group can be written as $G^{s}_{nK}$ ($K=3$). Although $G^{s}_{nK}$ is the group feature of one order, to avoid the abuse of notations, $G^{s}_{nK}$ also means the concated multi-path group feature representation of sub-group $G_{n}^{s}$. Next we will show the details of our proposed RU.

The two key modules of previous RCAU are reset gate and update gate. The goal of reset gate ($g_{d}$) is to use the synergetic relationships between the group feature and the current image to suppress the noise data in current image. The previous RCAU only uses a convolutional operation to achieve this purpose. However, this simple convolutional operation cannot capture long-range relations between group feature and current image feature. So if the co-salient regions of these two features vary largely, simple convolutional operation cannot well suppress the noise data. Inspired by non-local network, we design a new non-local cross-attention (NLCA) to fully suppress the noise data. Specifically, the input of NLCA is the group feature representation $G_{nk}^{s}\in\mathbb{R}^{C\times H\times W}$ and single image representation $X_{nk}\in\mathbb{R}^{C\times H\times W}$. In the first step, $G_{nk}^{s}$ is initialized with $X_{nk}$. For Query branch, we first add a $1\times 1$ convolution layer on $X_{nk}$ and reshape the feature to $\mathcal{R}^{C\times L}$, where $L=H\times W$. Meanwhile, for Key branch, we also use a $1\times 1$ convolution layer on $G_{nk}^{s}$ and reshape the feature to $\mathcal{R}^{C\times L}$. After that, we perform a matrix multiplication between the transpose of $X_{nk}$ and $G_{nk}^{s}$, then apply a softmax function to calculate the spatial attention map $\textbf{A}\in\mathcal{R}^{N\times N}$. Each pixel value in A is defined as:

$$A(i,j)=\frac{exp((X_{nk})^{i}\cdot(G_{nk}^{s})^{j})}{\sum_{j=1}^{L}exp((X_{nk})^{i}\cdot(G_{nk}^{s})^{j})},$$

(5)

where $i\in[1,L]$, $A(i,j)$ measures the $j^{th}$ position in the group feature impact on $i^{th}$ position in single image feature. Meanwhile, like Key branch, we generate feature $\hat{G}_{nk}^{s}$ from Value branch and perform a matrix multiplication between A and the transpose of $\hat{G}_{nk}^{s}$ to get denoised feature $X_{nk}^{out}\in\mathcal{R}^{C\times L}$, which is defined as:

$$X_{nk}^{out}(i)=\sum_{j=1}^{N}A(i,j)\hat{G}_{nk}^{s}(j),$$

(6)

Finally, we reshape $X_{nk}^{out}$ to $\mathcal{R}^{C\times H\times W}$, and add it with $X_{nk}$.

The goal of update gate ($g_{z}$) is to determine what group information should be retained in group feature and what new information should be updated from current image feature. Previous RCAU uses co-attention mechanism to explore the spatial-channel-wise variation of the co-salient object between group feature $G_{nk}^{s}$ and image feature $X_{nk}$. However, the appearance as well as the location of co-salient object varies across different image and group features. Therefore, it is difficult to capture the consistency information between two differently distributed features with a spatial-channel-wise attention map. So we propose a co-attention feature projection module (CFPM) to project image and group features to the common feature subspace, which can help bridge the gap between group feature and image feature. At first, we employ pyramid pooling module (PPM [98]) to reduce the dimension of feature maps $G_{nk}^{s}$ and $X_{nk}$ to save the computational cost. The PPM is composed of four-scale feature bins, which are then flattened and concatenated to form a matrix of size $C_{1}\times L_{g}$, $L_{g}\ll HW$, $C_{1}=C/2$. Here, the sizes of the feature bins are set to $1\times 1$, $3\times 3$, $6\times 6$ and $8\times 8$, respectively. Thus, the self-affinity matrixes of $G_{nk}^{s}$ and $X_{nk}$ can be calculated as:

		$\displaystyle A_{G}=(PPM(W_{G}^{1}G_{nk}^{s}))^{\top}(W_{G}^{2}G_{nk}^{s}),$		(7)
		$\displaystyle A_{S}=(PPM(W_{S}^{1}X_{nk}))^{\top}(W_{S}^{2}X_{nk}),$

where $A_{G}$ and $A_{S}$ denote the feature-specific similarity matrixes. Their sizes are fixed to $L_{g}\times(HW)$ through the PPM, which is asymmetric. $W_{G}^{1}$, $W_{G}^{2}$, $W_{S}^{1}$ and $W_{S}^{2}\in\mathbb{R}^{C_{1}\times C}$, indicate the learnable parameters. We further combine these two matrices as follows:

$\displaystyle Z=softmax((A_{G}+A_{S})^{\top}).$

(8)

Finally, the row-wise normalized matrix $Z$ $\in$ $\mathbb{R}^{L_{g}\times(HW)}$ is used to assist the update of group and image features:

		$\displaystyle\widetilde{G}_{nk}^{s}=Z(W_{G}^{3}G_{nk}^{s})^{\top},$		(9)
		$\displaystyle\widetilde{X}_{nk}=(1-Z)(W_{S}^{3}X_{nk}^{out})^{\top}.$

We add $\widetilde{G}_{nk}$ and $\widetilde{X}_{nk}$ to get the features $G_{nk+1}^{s}\in\mathbb{R}^{C\times H\times W}$:

$$G_{nk+1}^{s}=\widetilde{G}_{nk}^{s}+\widetilde{X}_{nk}.$$

(10)

In this paper, the above process would be repeated three times to get group features of one order. Finally, we concat the group features from three different orders to generate multi-path group feature ($G^{s}_{nK}$ or $G_{n}^{s}$) of each sub-group.

After generating sub-group features $\{G_{n}^{s}\}_{n=1}^{N}$ from $\mathcal{X}=\{X_{n}\}^{N}_{n=1}$, we apply another RU to generate the final group feature $G_{N}$ from $\{G_{n}^{s}\}_{n=1}^{N}$, which is written as:

$$G_{N}=RU(\{G_{n}^{s}\}_{n=1}^{N}).$$

(11)

As described previously, the group feature is then broadcasted to each image, which allows the network to leverage the synergetic information and unique properties between the images. So the interaction of group representation $G_{N}$ and single representation $\{S_{n}\}^{N}_{n=1}$ are sufficiently exploited to facilitate the robust co-saliency reasoning. Thus, inspired by classic non-local network [99], we propose a non-local fusion module to well fuse single representations and group features, and get the final co-saliency maps $\mathcal{M}=\{M_{n}\}^{N}_{n=1}$.

Let $\mathcal{I}=\{I_{n}\}^{N}_{n=1}$ and their groundtruth $\{\mathcal{G}^{T}_{n}\}^{N}_{n=1}$ denote a collection of training samples where $N$ is the number of images. After co-saliency detection, the co-saliency results are $\{M_{n}\}^{N}_{n=1}$. We use the cross-entropy loss as the individual supervision for each image $I_{n}$:

$$L_{s}=-\big{(}\ \mathcal{G}^{T}_{n}log(M_{n})+(1-\mathcal{G}^{T}_{n})log(1-M_{n})\ \big{)}.$$

(12)

In addition to the cross-entropy losses, we propose a perceptual group-wise training objective to further explore the interactive relationships of whole images in the training group. Two criteria are jointly considered in the design of group-wise training objective, including 1) high cross-image similarity between the co-occurring objects and 2) high distinctness between the detected co-occurring objects and the rest of the images like background and non-common objects. We apply triplet loss as the group-wise constraint. Specifically, for a image $I_{n}$, we can generate three masked images with its co-salient mask $M_{n}$ and groundtruth $\mathcal{G}^{T}_{n}$

$$I^{o}_{n}=M_{n}\otimes I_{n},\leavevmode\nobreak\ I^{\mathcal{G}}_{n}=\mathcal{G}^{T}_{n}\otimes I_{n}\leavevmode\nobreak\ and\leavevmode\nobreak\ I^{-\mathcal{G}}_{n}=(\mathcal{G}^{-T}_{n})\otimes I_{n},$$

(13)

where $\otimes$ denotes element-wise multiplication and $\mathcal{G}^{-T}_{n}=1-\mathcal{G}^{T}_{n}$. The masked image $I^{o}_{n}$ means our current detected co-salient objects of $I_{n}$ while image $I^{\mathcal{G}}_{n}$ and $I^{-\mathcal{G}}_{n}$ mean the real co-salient objects and non-common regions of $I_{n}$. Then we apply the perceptual extractor [100] $\phi$ to all masked images $\{I^{o}_{n},I^{\mathcal{G}}_{n},I_{n}^{-\mathcal{G}}\}^{N}_{n=1}$ and obtain their corresponding perceptual features $\{\phi(I^{o}_{n}),\phi(I^{\mathcal{G}}_{n}),\phi(I^{-\mathcal{G}}_{n})\}^{N}_{n=1}$. We apply triplet loss $L_{c}$ on each $I^{o}_{n}$ as the group-wise training objective as shown in Figure 4, formulated as:

$$\begin{split}L_{c}=\frac{1}{N-1}\sum_{m\neq n}\big{[}b&+max\ \ E(I^{o}_{n},I^{\mathcal{G}}_{m})\\ &-min\ \ E(I^{o}_{n},I^{-\mathcal{G}}_{m})\big{]}_{+},\end{split}$$

(14)

where $b$ is the margin and $E(\cdot,\cdot)$ denotes the Euclidean distance between two feature vectors. The group-wise training objective uses the hinge function $[b+\bullet]_{+}$ to force co-saliency result $M_{n}$ to be more similar to real co-saliency objects than non-common regions. In co-saliency task, it can be beneficial to pull together co-occurring objects as much as possible. For this purpose, it is possible to replace the hinge function by a smooth approximation using the softplus function: In$\big{(}1+exp(\bullet)\big{)}$. The softplus function has similar behavior to the hinge, but it decays exponentially instead of having a hard cut-off, we hence refer to it as the soft-margin formulation. So the total co-saliency loss can be written as:

$$L_{co}=L_{s}+L_{c}.$$

(15)

In this paper, we first propose MSRU to make our network insensitive to the input orders. A stable CoSOD network means different input orders can generate similar group features. This means we should pull close embedding spaces of different group features. So we design a cross-order contrastive loss (COCL) to achieve this purpose. However, existing contrastive losses (e.g. [101, 102, 103]) cannot be directly applied to CoSOD task, because their aim is to help the network distinguish between feature embeddings of different inputs. Data augmentations are used to embed one-view data into different spaces, which helps the network learn the distinctive feature embedding of one-view data. While in CoSOD task, we leverage contrastive loss to further pull close different group features and make our proposed model insensitive to the input orders. So we propose the cross-orders consistent mining, leveraging the high similarity of samples in one order to guide the learning process in another order. It excavates positive pairs across different orders according to the embedding similarity to promote knowledge exchange among different orders, then the size of hidden positive pairs in each order can be boosted and the extracted group features will contain different order knowledge, resulting in a more regular embedding space. Specifically, as can be seen in Figure.7, we randomly generate $q=10$ group features through MSRU from different orders of image group $\mathcal{I}$. Samples $z^{u}$ and $z^{v}$ are generated from the different orders of the same image group, and their corresponding memory banks are $\textbf{M}^{u}$ and $\textbf{M}^{v}$:

$$\textbf{M}^{u}=\{m_{i}^{u}\}_{i\in{N}},\textbf{M}^{v}=\{m_{i}^{v}\}_{i\in{N}},$$

(16)

where $N$ is the total number of different orders. The contrastive context of sample $z^{u}$ is the similarity set $S^{u}$ among $z^{u}$ and $\textbf{M}^{u}$, and we use relation miner $\Psi$ that generates $N_{+}$ positive samples:

$$\begin{split}\textbf{S}^{u}&=\{s_{i}^{u}\}_{i\in N}=\{z^{u}\cdot m_{i}^{u}\}_{i\in N}\\ (\textbf{S}^{u}_{+},N_{+})&=\Psi(\textbf{S}^{u})=TopK(\textbf{S}^{u})\end{split}$$

(17)

if we want to use the knowledge of order $u$ to guide order $v$ contrastive learning, it contains two aspects: 1): we select the most similar pairs (positive) in order $u$ as the positive sets in order $v$, i.e., $\textbf{S}^{v}$, $N_{+}^{u}\to\mathcal{C}(\textbf{S}^{v}|N_{+}^{u})$. Thus the sample $z^{v}$ shares the positive neighbors of $z^{u}$. 2): we do not directly pull close the group feature embeddings generated from different orders, which would cause the collapse problem. We pull close the similarity of different samples. If sample $x$ is the positive sample of $z^{u}$, so $x$ would also be the positive sample of $z^{v}$. And we can calculate the similarities between {$z^{u}$,$x$} and {$z^{v}$,$x$}. Then we pull close these two similarities. The overall loss is conducted as:

$$L_{u\rightarrow v}=-\log\frac{\sum_{i\in N_{+}^{u}}\exp\left(s_{i}^{u}s_{i}^{v}\right)/\tau}{\sum_{i\in N}\exp\left(s_{i}^{u}s_{i}^{v}\right)/\tau}.$$

(18)

Finally, the total contrastive loss can be written as:

$$L_{cocl}=L_{u\rightarrow v}+L_{v\rightarrow u}.$$

(19)

Noting that all parts of our proposed network are trained jointly, so the total loss function is written as:

$$L=L_{cocl}+L_{co}.$$

(20)

Most of the previous methods [21, 24, 19] use the VGG16 as the backbone. Therefore, for a fair comparison, we also choose the VGG16 network to extract the features of each image in the group, which is the dominant reason. Moreover, the CoSOD task deals with image group data, and the core problem in CoSOD is group-feature learning, which cannot be solved by simply replacing the stronger backbones. The same conclusion is reported in the work GLNet [19]. The training set is a subset of the COCO dataset [104] (9213 images) and saliency dataset DUTS [105], as suggested by [18]. All the images are resized to the same size of $352\times 352$ for easy processing. The model is optimized by the Adam algorithm with a weight decay of 5e-4 and an initial learning rate of 1e-4. During training, the batchsize is 32. For co-saliency detection, the training of our proposed network includes two stages:

Stage1. We first train our model using DUTS dataset [105] to focus on the salient areas. Note that when training, to match the size of input group, we augment the single salient image to $N=32$ different images as a group using affine transformation, horizontal flipping and left-right flipping.

Stage2. We further fine-tune our model using sub-coco dataset to better focus on the co-salient areas. All the parameter settings are the same as those in Stage1.

As described, since CoSEG dose not need to detect these objects belong to salient regions, we only use Stage2 when training CoSEG model.

Co-saliency Detection. We employ five challenging datasets for evaluation: CoCA [23], CoSOD3k [106], Cosal2015 [107], iCoseg [108] and MSRC [109]. Cosal2015 has 50 groups and a total of 2015 images. Cosal2015 suffers from various challenging factors such as complex environments, occlusion issues, target appearance variations and background clutters, etc. CoSOD3k [106] has 160 groups and a total of 3000 images. CoSOD3k is the largest-scale and most comprehensive benchmark, which has sufficient object diversity and the complexity for size and number for instances. CoCA has 80 groups and a total of 1297 images. CoCA is a challenging dataset, since the images typically contain other multiple objects in addition to the co-salient objects which are even smaller in size. iCoseg consists of 38 groups of total 643 images. MSRC contains 7 groups of total 240 images, and each group has $30\sim 53$ images.

To evaluate the performance of the proposed method, we adopted five widely used criteria: (1) Precision-Recall (PR) curve, which shows the tradeoff between precision and recall for different threshold (ranging from 0 to 255). (2) The F-measure ($F_{\beta}$), which denotes the harmonic mean of the precision and recall values obtained by a self-adaptive threshold $T=\mu+\sigma$ ($\mu$ and $\sigma$ are the mean value and standard deviation of co-saliency map):

$$F_{\beta}=\frac{(1+\beta^{2})\times Precision\times Recall}{\beta^{2}Precision+Recall},$$

(21)

where $\beta^{2}$ is typically set to 0.3 as suggested in [110, 111, 112]. In this paper, we use maximum F-measure $F_{\beta}$ to evluate the performance. (3) Structure Measure ($S_{m}$) is adopted to evaluate the spatial structure similarities of saliency maps based on both region-aware structural similarity $S_{r}$ and object-aware structural similarity $S_{o}$, defined as

$$S_{m}=\alpha\ast S_{r}+(1-\alpha)\ast S_{o},$$

(22)

where $\alpha=0.5$ [113]. (4) The E-measure [114] is a perceptual metric that evaluates both local and global similarity between the predicted map and ground-truth simultaneously. In this paper, we use maximum E-measure $E_{\xi}$. (5). Mean absolute error (MAE), which characterize the average 1-norm distance between ground truth maps and predictions. Evaluation toolbox:https://dpfan.net/CoSOD3K/.

Object Co-segmentation. We evaluate the proposed method and compare it with existing methods on three benchmarks for object co-segmentation, including the Internet dataset [70], the iCoseg dataset [108], and the PASCAL-VOC dataset [115]. These datasets are composed of real-world images with large intra-class variations, occlusions and background clutters. The Internet dataset contains images of three object categories including airplane, car and horse. Thousands of images in this dataset were collected from the Internet. Following the same setting of the previous work [77, 70, 116] , we use the same subset of the Internet dataset where 100 images per class are available. iCoseg consists of 38 groups of total 643 images which are challenging for object co-segmentation task because of the large variations of viewpoints and multiple co-occurring object instances. The PASCAL-VOC dataset contains total 1,037 images of 20 object classes from PASCAL-VOC 2010 dataset. The PASCAL-VOC dataset is more challenging and difficult than the Internet dataset due to extremely large intra-class variations and subtle figure-ground discrimination. Two widely used measures, precision ($\mathcal{P}$) and Jaccard index ($\mathcal{J}$), are adapted to evaluate the performance of object co-segmentation. Precision measures the percentage of correctly segmented pixels including both object and background pixels. Jaccard index is the ratio of the intersection area of the detected objects and the ground truth to their union area.

For CoSOD task, we compare our approach with 11 CoSOD models and 3 SOD models: CBCS [60], GWD [117], RCAN [14], CSMG [16], CoEG [8], GICD [23], ICNet [24], CoADNet [18], GCoNet [118], CADC [119], GLNet [19], EGNet [46], F3Net [38] and MINet [39]. For CoSEG task, we compare our approach with other 2 most SOTA methods: Li19 [120] and Li21 [11].

Quantitative Evaluation. From Table.I, we can see that compared to other state-of-the-art methods, our model (Ours(32)) outperforms all of other SOTA methods in all metrics. As reported in Table.I, our approach achieves good performance on different size groups (8, 16 and 32) and still consistently outperforms all the state-of-the-art methods. And the performance raises along with the group size, which emphasizes the importance of the group information completeness to robust co-saliency detection. In dataset CoCA, compared to the second ranked performance, the improvement of our proposed method reaches 2.1% for $E_{\xi}$, 7.4% for $S_{m}$, 12.4% for $F_{\beta}$ and 5.7% for MAE. On the challenging CoSOD3k and Cosal2015 datasets, our model capitalizes on our better consensus and significantly outperforms other methods. These results demonstrate the efficiency of the proposed framework. As shown in Fig.8, we can see that our method (the red line) achieves the highest precision on all datasets. Table.IV, Table.IV and Table.IV show the results on CoSEG datasets, it can be seen that our method can outperform all other two SOTA methods in most datasets. Finally, we show the parameter number and running time comparison with other SOTA methods, and the results are shown in the Table.V. Our proposed model runs at a competitive efficiency compared to other models. This is primarily due to the fact that cross-order contrastive loss is only used during training, and only the dummy orders mechanism (DOM) would slow the network down to some extent.

Qualitative Evaluation. Fig.9 shows the co-saliency maps generated by different methods for qualitative comparison. As can be seen, the SOD method F3N can only detect salient objects and fail to distinguish co-salient objects. The CoSOD methods perform better than the SOD methods because of considering group-wise relationships in designing the model. As can be seen in ”Baseball Group”, these CoSOD can suppress some non-co-salient regions. However, these CoSOD methods require constant input data, or only using simple adding operation to generate final group features. When facing complex real-world scenarios, they are unable to handle these challenging cases, like ”Hammer Group” and ”Axe Group”. While our proposed model can capture complete inter-saliency relations of an image-group, and make the training and inference process more stable, therefore performs much better on detecting co-salient objects. Fig.10 shows the co-segmentation maps generated by our proposed methods for qualitative comparison, which can further demonstrate the superiority of our model in CoSEG task.

In this section, we first conduct evaluation on CoSOD3K, Cosal2015 and PASCAL-VOC datasets to investigate the effectiveness of various components of the proposed model. We first do architecture ablation studies, and we set the baseline model by only using single feature learning branch and replacing the recurrent neural network with concatenation operation. The baseline model is only trained with co-saliency loss $L_{co}$ alone. Moreover, we set the input orders of different models the same, so the performance changes of different models are a result of different architectures. The results are shown in Table.VI. Secondly, we further evaluate the stability of our proposed network, and the results are shown in Table.VII.

Architecture ablation studies. As can be seen in No.1 of Table.VI, even though trained with co-saliency loss $L_{co}$, the baseline model can not well handle the co-saliency or co-segmentation task. When we replace the simple concatenation operation with typical recurrent unit, GRU or LSTM (No.2 and No.3), the performance is improved compared to baseline, which means the recurrent architecture is suitable for co-saliency detection task. Moreover, we use our previous RCAU to replace GRU or LSTM (No.4), the performance can be further improved, because proposed RCAU can generate more robust group features compared to GRU or LSTM. Then, we add our proposed MSRU (No.5), the performance has large improvement. This is because our proposed MSRU addresses the order-sensitive problem of RCAU and further improve the stability of proposed network. It should be noted that in this paper, we add GRU, LSTM, RCAU and MSRU on three levels of backbone network (Conv4-3, Conv5-3 and Conv6-3), which can help capture multi-level inter-saliency relations. Then, adding the NLFM (No.6) can further improve the performance. Finally, COCL (No.7) can further boost the performance because it can help pull close group feature embeddings generated from different orders. To further justify the denoising ability of our MSRU, we add a noise image which is randomly selected from COCO dataset to the testing groups in size 8, 16 and 32 (No.15, No.16 and No.17). As shown in results, although the noise data damages our performance a little, we still outperforms all the SOTA methods, which demonstrate the robustness of the proposed method.

We also investigate the effectiveness our proposed DOM, and NLCA or CFPM in RU. Compared No.9 to No.8, It can be seen that when adding DOM on RCAU, it can improve the performance. Compared No.10 to No.7, when removing the DOM from our proposed method, the performance would be declined. It shows the effectiveness our proposed DOM. As can be seen in No.13 and No.14, when we set window size $k=4$ or $k=5$ in DOM (Eq.3), the performance has no obvious change while the training/testing is increasing about 30%. So in this paper, we only set $k=3$. When we remove the reset gate $g_{d}$ from MSRU (No.11), the performance declines on three metrics especially on MAE. This indicates the reset gate $g_{d}$ is able to suppress the noise information in the group. When we remove the update gate $g_{z}$ from MSRU (No.12), the performance declines which indicates update gate $g_{z}$ can well retain group information in group feature and determine what new information should be updated from current image feature representation. These experiments verify the effectiveness of the different modules proposed in this paper.

Stability ablation studies. During testing of our proposed method (Ours(RU+DOM+$L_{cocl}$)), for each image group, we randomize 10 different orders. From Table.VII, it can be seen that performance has no obvious change. “Std” means the standard deviation of these 10 orders. This result verifies that our proposed MSRU and Contrastive Loss can let the proposed method less insensitive to the input order of group images. In the Introduction of this paper, we argue that existing sequential order modeling approaches make CoSOD networks unstable. So we do experiments on two typical methods, including RCAN [14] and ICNet [24], to verify the inferring procedure of these two methods are unstable. Because CoADNet [18] does not release their code, so we can not do experiments on CoADNet. During testing, for each image group, we randomize 10 different orders, and the results are shown in Table.VII. As can be seen in Table.VII, whether using RNN ( RCAN(Std) ), or applying some sophisticated modification on CNNs architectures ( ICNet(Std) ), can not let the CoSOD network be insensitive to the input order of group images, leading to an unstable inferring procedure. Because in sequential order modeling, both CNNs and RNNs have inherent deficiencies. In this paper, our proposed DOM and COCL can help eliminate the effects of different orders. Therefore, as can be seen in Table.VII, when we remove these two modules, our proposed network would be more sensitive to the order of images input. Through these experiments, we further verify that our proposed network greatly improves the stability of CoSOD. In Table.IV and Table.I, we test our network 10 times and select the medium performance in this paper. From the first row of the Table.VII, the worst performance of our proposed network remains SOTA. Note that the FPS of the models Ours(RU) and Ours(RU+DOM) are 44 and 27. Compared to the previous RCAN, our proposed RU is slower because we use non-local attention. Moreover, adding DOM mechanism further reduces the running speed. However, as shown in the Table.V, our proposed model still runs at a competitive efficiency compared to other SOTA models.

Training stages ablation studies. In this paper, we introduce two stage training processes for the CoSOD task. The stage1 training process helps the network capture the regions tended to be salient, and the stage2 training process helps the network learn the co-salient regions. In the Fig.11, we show some examples where the stage1 model fails but the stage2 model succeeds. In the “Apple Group” and “Baseball Group”, the stage1 model detects “salient noise” which is suppressed in the stage1 model. In the “Dumbbell Group” and “Train Group”, the object “person” is detected in the stage1, and the real co-salient objects are omitted. Because in the SOD DUTS training dataset, the person is always the salient object and the other objects are the background. After fine-tuning the stage1 model on the sub-coco dataset [104], the stage2 model would detect real co-salient objects. Quantitative comparison between the stage1 model and the stage2 model is in Table.VIII, and the stage2 model is obviously superior to the stage1 model.