Inferring the Class Conditional Response Map
for Weakly Supervised Semantic Segmentation

For a class $c$, a CAM [53] is a feature map indicating the discriminative region of the image that influence the classifier to make the decision that this object belong to class $c$. Given $f_{k}(x,y)$, the activation of unit $k$ in the last convolutional layer at location $(x,y)$, and $w_{c}$, the weights from $f_{k}(x,y)$ via global average pooling, the CAM $M$ is a map of activation at locations $(x,y)$ as described in [53]:

$$M_{c}(x,y)=\Sigma_{k}w^{c}_{k}f_{k}(x,y).$$

(1)

Consider a network $h$ with parameters $\theta$. For input image $I$, suppose the output for class $c$, $h_{\theta}(c|I)>\tau$, where $\tau$ is the threshold probability. Recent high-performance networks (e.g., [22, 20]) trained on large datasets (e.g., ImageNet) yield highly accurate classification for well-represented classes. Hence, given positive classification for $c$, we can treat $M_{c}$ as an approximation to a class conditional response map. Hence, we define a class conditional response map for image $I$ using network parameters $\theta$ as:

	$\displaystyle R_{c}((x,y)|I;\theta)$	$\displaystyle=M_{c}((x,y)|c,I;\theta)$
		$\displaystyle\approx M_{c}((x,y)|I;\theta),$

where $M_{c}((x,y)|c,I;\theta)$ is the CAM for $I$ in which class $c$ appears, and $M_{c}((x,y)|I;\theta)$ is the CAM for image $I$, given that class $c$ appears with high probability in the image.

For networks with deep residual structures it would be difficult to trace back through the cascading and residual activation to find the exact pixels that have led to this suppression of CAM locations that may otherwise have positive features. Instead, we propose a class conditional inference-time approach to solve this issue. We aim to generate a more uniform distribution of activation across the visible object.

We assume an initially well-trained classification network $h_{\theta}$, which we do not re-train. Instead we propose class conditional inference, whereby we augment $I$ at inference time to remove regions that may suppress the response of other object parts to mine the class-conditional object activation. Concretely, we perform this by computing the class conditional response map $R_{c}(x,y|I^{\prime};\theta)$, where $I^{\prime}$ is an augmented image. By removing pixels in $I$ associated with high activation, we aim to explore other regions of being suppressed. A high response on $R_{c}(x,y|I^{\prime};\theta)$ for some location where the activation was not high for $R_{c}(x,y|I;\theta)$ is likely to indicate an appearance pattern that is strongly associated with the object, but was suppressed by the regions that visible in $I$, but not $I^{\prime}$. In this section we describe our implementations of this by class conditional Split $\&$ Unite Augmentation, and Iterative Inference.

Instead, we first apply conventional inference with the baseline classifier to obtain $R_{c}((x,y)|I;\theta)$. As we assume a well-trained model $\theta$, the initial high CAM activation will generally fall on the discriminative region of the target object. Then we calculate the centre of mass of the original CAM to obtain a center point about which we split the image into four patches, as shown in Fig. 3. We find the centre of mass generally falls on the class, in which case some part of the object appears in each of the four sub-images.

With the classifier fixed, we then compute $R_{c}((x,y)|I_{i};\theta)$ where $i$ indexes each split, Although the unequal distribution of the object activation in the baseline CAM leads to only highly discriminative regions being highlighted, by separate inference, object discrimination is computed individually in each patch with any suppressing elements in the other patches removed so we can locate more discriminative areas on different object parts. For example, see the first row of Fig. 3 showing the split for images with one object class. Without the highly activated sheep head in the patch #2, other parts of the sheep are highlighted in patch #1, #3 and #4. For images with two object classes, we refine our split strategy, as shown in second row of Fig. 3, we calculate centres of mass for both class’s CAM, then we use these two center points to obtain a rectangle inside the image (displayed as the red central area). We use the four corners of this rectangle as a split point to crop the original image into four patches. The central rectangle (red area) is retained in all four patches, as shown in the “Splits” column. Each split generally contains different parts of both object classes. Then we run inference on each patch in turn, and merge the four response maps, we take the max activation for the overlapping central rectangle area. For images with three or more classes, we split the images by the CAM mass center for each class separately.

Furthermore, our split $\&$ unite method could also be used as a common image augmentation method to assist re-training the classification model. Specifically, we feed each patch individually into the classification model to train the network with the same image-level ground truth as the original image. i.e. we use different object parts to train the classifier instead of the entire object, thus the classifier will naturally pay attention to more object parts during inference. We show further experimental results of in Sec. 4.3.

Refer to Fig. 4, we first feed the original image into the classifier and produce response map, revealing the highly discriminative areas. Then these highly discriminative areas are erased with mean colour value of the original image. The augmented image is then fed back to the classifier for next inference iteration. No training is performed as the pixels corresponding to the discriminative region are removed. With these object features absent, without suppression, weaker activation will be naturally driven to shift to high activation. We iterate this inference process and then add the newly generated activation map of every iteration together to obtain the final response map. As shown in Fig. 4, new object areas are activated progressively in each iteration without updating the network.

Saliency information is used by many approaches in image-level WSSS to refine object boundaries and obtain a background mask [37, 24, 27, 16, 17, 21, 23, 44, 41]. Commonly, a pre-trained off-the-shelf salient object detection model is adopted to generate class-agnostic saliency masks on the segmentation dataset. Then the saliency masks are used during pseudo label generation, normally they are multiplied by a manually chosen parameter and concatenated onto the activation maps as background. High activation pixels are then used to obtain pixel-wise pseudo labels.

However, we observe that since saliency detection models are usually trained by class-agnostic objects with center bias, the saliency maps may falsely detect non-object salient areas in the foreground, and tend to ignore non-salient objects in the background (see Fig. 7), Thus, it may introduce errors into pseudo-labels and harm segmentation training. Some desired object regions are ignored and some non-object regions are falsely detected in the saliency maps. However, saliency maps are still required by WSSS to support finding accurate foreground object boundaries. To address this issue, we propose a new method to leverage saliency information in WSSS, including a new saliency loss. The use of CAMs to obtain pseudo-labels without saliency maps is unchanged. Then we use the pseudo labels together with the provided saliency maps to train our semantic segmentation model. As shown in Fig. 6, we keep both labels visible to the network, so the saliency maps can refine foreground object boundaries, but we inhibit suppression of activated objects in the background. More formally, our segmentation loss is defined as:

$$L_{seg}=L_{seg}+\alpha(e^{\tau}-1)L_{sal}$$

(4)

The first term $L_{seg}$ is the cross entropy loss between segmentation prediction and our activation-generated pseudo labels. The second term $L_{sal}$ denotes the binary cross entropy loss between the background channels of the predictions and the saliency maps. We incorporate a modulating factor $\tau$ (called the conflict temperature) with tunable weight $\alpha>0$. Intuitively, as two noisy signals (pseudo labels and saliency maps) in the supervision pair have conflict areas as shown in Fig. 7, they will compete with each other. We use $\tau$ to control the competition between two supervision terms. $\tau$ is defined as the mean intersection-over-union (MIoU) between our pseudo label background channel and the saliency maps. If $\tau$ is low, it indicates the saliency map is not consistent with the pseudo background, and so the saliency maps will harm segmentation training, so the weight of the saliency loss $L_{sal}$ is diminished. Contrarily, if MIoU is high, then the saliency loss is enhanced to refine the object boundaries. In summary, our activation-aware mask refinement loss proposes an adaptive mechanism to refine segmentation training. Saliency information is fully explored to refine the foreground object boundaries but not harm activated objects in the background.

In this section, we introduce implementation details of the proposed method and the following procedures to generate semantic segmentation results. We evaluate our approach on the PASCAL VOC 2012 dataset [12] with the background and 20 foreground object classes. The official dataset consists of 1446 training images, 1449 validation and 1456 test. We follow common practice, augmenting the training set by adding images from the SBD dataset [19], to form a total of 10582 images.

In Table 1, we show initial response map (CAM) performance, using best mean IoU, i.e. the best match between the response map and segmentation ground truth under all different background thresholds. We also report pseudo-label results after applying the random walk refinement (CAM + RW). Note that our results are obtained with the baseline classifier without any re-training, all improvements come from our proposed class conditional inference. As shown in Table 1, our initial response maps are significantly improved over baseline [2] on both training and validation sets. We also compare response maps generated by recent state-of-the-art methods [4, 5], and observe a a clear margin.

Our better initial response maps lead to better performance of downstream tasks: generating pixel-wise pseudo-labels and final segmentation results. After refining the response map into pseudo labels that are used to train a semantic segmentation model, we also achieve significant improvement over the baseline and outperform competing methods, as shown in “CAM+RW” column in Table 1. This validates that, by direct inference without fine-tuning, our method can substantially improve object activation and generate better response maps than competing methods with re-trained classification models. In Fig. 8, we show qualitative examples compared with baseline CAMs, showing that ours can activate substantially more object parts and uniformly cover larger regions of the objects.

In Table 2, we show an ablation study, how each of our modules improves the initial response maps. The improvement mainly stems from more dense coverage of objects, we test each module independently. Our split $\&$ unite augmentation improves the baseline CAM by 1.9%. The iterative inference module has an improvement of 2.2% compared with baseline. It validates that both our modules provide manifest improvement over the baseline CAM. Integrating them together, we achieve a significant improvement over the baseline by 4.2% on our initial response map.

Finally, we report performance improvements for different numbers of classes appearing in an image in Table 3. As shown, we achieve consistent improvements over competing methods on images with all class numbers, demonstrating the effectiveness and generality of our method.

Our method can be directly used as an add-on inference module with any existing re-trained classifier to obtain a better initial response map. First, as discussed in Sec. 3.2, our split $\&$ unite augmentation can be used in the training stage as a data augmentation method. We follow the methods described in Sec. 3.2 to augment the training set to feed different parts of the object into the classification model to train the network, so the network will update its weights to pay attention to more object parts. Refer to Table 4: fine-tuning the classifier with our split $\&$ unite augmentation further improves the initial response map performance.

In addition, we perform our inference-time data augmentation on the re-trained CAM of (SC-CAM [5]) to refine their produced CAM. SC-CAM [5] introduced a sub-category clustering method to force the network to activate in more categories, leading to enlarged activation regions. We then perform our inference-time approach on their re-trained classifier. As shown in Table 4, we (Ours + SC-CAM) improve their performance significantly by 2.4%. This validates that our inference method can be used as an add-on solution to integrate with existing re-trained classifiers to further refine the object activation maps. .

In Table 1, we show the refined pseudo labels generated by our initial response maps. We then use the pseudo labels and our activation aware mask refinement loss to train a segmentation network on PASCAL VOC dataset. As per common practice, we use the densely connected CRF [26] to refine the semantic segmentation predictions as post processing. We show final predictions in Fig. 9, clearly showing the effectiveness of our approach. In Table 5, we compare our method with recent work. We report performance on both the validation and test set of the PASCAL VOC 2012 dataset [12]. As shown, our method outperforms all others on the test set, and is comparable with state-of-the-art on the validation set. Note that most other methods require additional re-training steps to obtain better object activation, our method is easier and faster to implement. Further, we remove the saliency guidance in Fig. 6 from our framework, leading to the cheapest model (no re-training, no saliency), and achieve mIoU(%) on PASCAL VOC validation set and testing set as 66.2 and 66.3 respectively, which is comparable with re-training based model RRM [48] and slightly worse than AdvCAM [28].

We argue that current WSSS multi-training schemes are complicated and inelegant, so we propose to improve WSSS performance without further increasing computation time. First, our saliency loss is a subsidiary supervision calculating the binary cross entropy loss during segmentation training, the increased time complexity is negligible, and most SOTA approaches already incorporate saliency to generate pseudo labels. Second, although re-training time is saved, our class conditional inference module requires once-off additional inference-time computation to generate pseudo-labels, we give a quantitative analysis here. To obtain activation maps on the PASCAL VOC train set (1464 images) our method takes 40 minutes compared to 13 (factor of 3, e.g., an extra 3.5 hours for the 10582 image Augmented dataset). On the other hand, re-training the baseline classifier as performed by most others requires 6 hours on the same GPU settings (note that competing methods have additional augmented images, and/or network enhancements meaning their re-training takes at least this long). Thus, our inference-time method still greatly saves overall time to obtain high-quality activation maps. As future work, we will refine our method to further reduce the inference time.