A Unified Two-Stage Group Semantics Propagation and Contrastive Learning Network for Co-Saliency Detection

Fig. 1 illustrates the flowchart of the unified Two-stage grOup semantics PropagatIon and Contrastive learning NETwork (TopicNet). Given $M$ groups with $N$ images in each group $\{I_{1,n},I_{2,n},...,I_{M,n}\}_{n=1}^{N}$, firstly, the last three layers of a weight-shared encoder are adopted to generate feature map $\{x_{m,n}^{i}\}_{n=1}^{N}\in\mathbb{R}^{N\times C\times H\times W}$ for $M$ groups, where $i=3,4,5$ represents the layer serial number, $C$ is the channel number and $H\times W$ is the dimensional size. Secondly, $\{x_{m,n}^{i}\}_{n=1}^{N}$ is fed into the image-to-group propagation module (IGP) to preliminarily extract group semantics $\{r_{m,n}^{i}\}_{n=1}^{N}\in\mathbb{R}^{N\times C\times H\times W}$. Thirdly, $\{r_{m,n}^{i}\}_{n=1}^{N}$ is transferred to group-to-pixel propagation module (GPP) to distill the co-attention $\{e_{m,n}^{i}\}_{n=1}^{N}\in\mathbb{R}^{1\times C\times 1\times 1}$. Then, $\{x_{m,n}^{i}\}_{n=1}^{N}$ element-wise multiplied by $\{e_{m,n}^{i}\}_{n=1}^{N}$ explores the recalibrated group semantics $\{z_{m,n}^{i}\}_{n=1}^{N}\in\mathbb{R}^{N\times C\times H\times W}$. Fourthly, $\{z_{m,n}^{i}\}_{n=1}^{N}$ is fed into the contrastive learning module to enhance semantic consistency and suppress noise objects. Finally, the refined group semantics are aggregated into the decoder to modify the individuals to predict the salient maps $\{\mathcal{M}_{m,n}\}_{n=1}^{N}$.

Contrastive learning aims to pull close the positive samples and push away the negative samples at the same time. As mentioned in section 1, reasonably designing positive and negative samples can intuitively enhance the semantic consistency and suppress noise objects. Concretely, the relationship $\Psi^{+}$ between positive samples and the anchor is defined as.

$$\begin{split}F(z)&=x_{m,n}^{5}\odot\{e_{m,n}^{5}\}_{n=1}^{N},\\ \ F(z^{+})&=x_{m,n}^{i}\odot\{e_{m,n}^{i}\}_{n=1}^{N},i=3,4,\\ \Psi^{+}(F(z),F(z^{+}))&=exp(\frac{{F(z)}^{T}F(z^{+})}{\tau||F(z)||||F(z^{+})||}),\end{split}$$

(1)

where, $F(z)$ is the anchor and $F(z^{+})$ is the positive sample. The temperature parameter $\tau$ relaxes the element-wise multiplication and $\tau=0.07$ following previous method [10]. Why is it designed like this? Evidence [20] suggests that as the complexities of network depth, blocks and topology increase, the risk of insufficient representation learning is posed. An intuitive approach is to supervise the network. However, the encoder supervision will lead to performance degradation. Therefore, many researches [5, 6] add auxiliary supervision in the decoder. Unfortunately, CoSOD needs to calibrate the group semantics with encoder, which contradicts the previous supervision design. To alleviate this contradiction, we pull close to the last three layers to enhance the semantic consistency. Meanwhile, the unsupervised refined encoder will not reduce the generalization ability of the network. Why only the last three layers? Because the first two layers are too close to the input, pulling close to all layers will lead to the loss of key information in the shallow layer or the addition of useless information in the deep layer.

Fig. 1 lists three generation routes of negative samples. The first one $\Psi_{1}^{-}$ is that the inter-group objects are negative samples of each other, aiming to maximize the inter-group object separability.

$$\Psi_{1}^{-}(F(z),F_{1}^{l}(z^{-}))=exp(\frac{{F(z)}^{T}F_{1}^{l}(z^{-})}{\tau||F(z)||||F_{1}^{l}(z^{-})||}),$$

(2)

where, $l$ is another group anchor. The second one $\Psi_{2}^{-}$ is that the element-wise multiplication of non-corresponding $x_{j,n}^{i}$ and $e_{k,n}^{i}$ $(j\neq k)$ is defined as negative samples.

$$\begin{split}F_{2}(z^{-})&=x_{j,n}^{i}\odot e_{k,n}^{i},j\neq k,\\ \Psi_{2}^{-}(F(z),F_{2}(z^{-}))&=\sum_{j=1}^{M}\sum_{k=1}^{M}exp(\frac{{F(z)}^{T}F_{2}(z^{-})}{\tau||F(z)||||F_{2}(z^{-})||}).\end{split}$$

(3)

Different group attentions are equivalent to assigning the main attributes of other objects to the current object, which enhances the discriminative representations of the network. The last one $\Psi_{3}^{-}$ is the fusion of different attentions.

$$\begin{split}F_{3}(z^{-})&=x_{j,n}^{i}\odot(e_{j,n}^{i}+e_{k,n}^{i}),j\neq k,\\ \Psi_{3}^{-}(F(z),F_{3}(z^{-}))&=\sum_{j=1}^{M}\sum_{k=1}^{M}exp(\frac{{F(z)}^{T}F_{3}(z^{-})}{\tau||F(z)||||F_{3}(z^{-})||}).\end{split}$$

(4)

The fused attention represents aggregating the features of other groups semantics to the current group semantics, aiming at producing interference features to enhance the generalization ability of the network. Then the relationship $\Psi^{-}$ between the whole negative samples and the anchor can be summed as.

$$\Psi^{-}(F(z),F(z^{-}))=\sum_{h=1}^{H}\Psi_{h}^{-}(F(z),F_{h}(z^{-})),$$

(5)

where, if the number of groups is 2, then $H=3$, otherwise $H=2$ because negative samples are enough to support the learning of noise objects. Then the loss function of contrastive learning can be defined as.

$$\mathcal{L}_{cl}=-log(\frac{\Psi^{+}(F(z),F(z^{+}))}{\Psi^{+}(F(z),F(z^{+}))+\Psi^{-}(F(z),F(z^{-}))}).$$

(6)

Finally, the optimized positive samples and anchor are integrated into the decoder to calibrate individuals with saliency supervision. The saliency loss function is as follows.

$$\mathcal{L}_{s}=1-\frac{\sum_{(p,q)}{\mathcal{M}(p,q)\mathcal{T}(p,q)}}{\sum_{(p,q)}{(\mathcal{M}(p,q)+\mathcal{T}(p,q))}},$$

(7)

where, $\mathcal{M}$ is the predicted saliency map and $\mathcal{T}$ is the ground truth, $p$ and $q$ are the pixel coordinates. During training, the network is jointly optimized by $\mathcal{L}_{cl}$ and $\mathcal{L}_{s}$.

$$\mathcal{L}=\lambda_{1}\mathcal{L}_{cl}+\lambda_{2}\mathcal{L}_{s},$$

(8)

where, $\lambda_{1}$ and $\lambda_{2}$ are the parameters that balance the loss weights. We set $\lambda_{1}$ and $\lambda_{2}$ to 1 because they are equally important.

Following pervious methods [5, 6], the proposed TopicNet is evaluated on three prevailing datasets, including CoCA [6], CoSal2015 [14] and CoSOD3k [8]. CoCA has 80 categories with 1295 images. It is the most challenging dataset that contains at least one noise object in each group. CoSal2015 is also a challenging dataset that consists of 50 groups with 2015 images. CoSOD3k is widely used in CoSOD and contains 160 groups with 3316 images in total. Five evaluation criteria, including maximum F-measure ($F_{\mu}$), mean absolute error ($\epsilon$), average F-measure ($F_{\gamma}$), maximum E-measure ($E_{\mu}$) and S-measure ($S_{\alpha}$), are adopted to evaluate methods.

The proposed framework is implemented on Pytorch platform. The number of groups is set to 2 and 3 (TopicNet-2 and TopicNet-3) to quantitatively evaluate the results. TopicNet-2 and TopicNet-3 are equipped with a V100 GPU. Following previous researches [5, 6], TopicNet is trained end-to-end on the dataset [6] and the data augmentation skills such as normalization, random horizontal flipping and random rotation are adopted to decrease over-fitting optimization. Besides, the classic vgg16 and feature pyramid network are adopted as the baseline. During training, all the inputs are resized as $224\times 224$. Adam optimizer with default parameters is applied. The learning rate is set to 1e-4 during 100 epochs and the batch size is set to 16.

We compare the proposed TopicNet against the CoSOD methods with released codes. In visual comparison, only the representative TopicNet-2 is visualized for saving room.

Tab. 1 reports the performance statistics and the relative improvements between TopicNet and other methods. Obviously, our method outperforms other competitors across most datasets in terms of five evaluation metrics. It is worth mentioning that TopicNet-3 achieves better performance than TopicNet-2. However, TopicNet-3 takes up more video memories (16 G vs 9 G) during training. Note that more input groups only occupy more video memories during training, and their consumptions in inference are the same. Besides, our method can achieve real-time speed (around 50 fps), which greatly benefits various down-streaming methods. The visualized results in Fig. 2 show that TopicNet has superior visual performance, which verifies the effectiveness of the proposed method.

The effective experiments of each proposed module are shown in Tab. 2. Besides, we add the module GAM of recent method [5] as control group to verify the effectiveness. For a fair comparison, we design the ablation experiments with only TopicNet-2. We can observe that compared with GAM [5], the proposed IGP achieves more competitive results. The control groups (e-h) show that contrastive learning can effectively boost the performance. Ablation studies verify that the proposed each module makes the network learn more discriminative semantics representations. Besides, Tab. 3 reports the results of picking different layers. The last three layers can better maintain the semantic consistency. If all the layers are selected, the performances will be significantly reduced. Because the first two layers are close to the image, pulling close to these layers will lead to the loss of key features or the addition of useless features.