Causal Intervention for Weakly-Supervised
Semantic Segmentation

We formulate the causalities among pixel-level image $X$, context prior $C$, and image-level labels $Y$, with a Structural Causal Model (SCM) [41]. As illustrated in Figure 3 (a), the direct links denote the causalities between the two nodes: cause $\rightarrow$ effect. Note that the newly added nodes and links other than $X\rightarrow Y$³³3Some studies [51] show that label causes image ($X\leftarrow Y$). We believe that such anti-causal assumption only holds when the label is as simple as the disentangled causal mechanisms [40, 56] (e.g., 10-digit in MNIST). are not deliberately imposed on the original image-level classification; in contrast, they are the ever-overlooked causalities. Now we detail the high-level rationale behind the SCM and defer its implementation in Section 3.2.

$\boldsymbol{C\to X}$. Context prior $C$ determines what to picture in image $X$. By “context prior”, we adopt the general meaning in vision: the relationships among objects in a visual scene [38]. Therefore, $C$ tells us where to put “car”, “road”, and “building” in an image. Although building a generative model for $C\rightarrow X$ is extremely challenging for complex scenes [24], fortunately, as we will introduce later in Section 3.2, we can avoid it in causal intervention.

$\boldsymbol{C\rightarrow M\leftarrow X}$. $M$ is an image-specific representation using the contextual templates from $C$. For example, a car image can be delineated by using a “car” context template filled with detailed attributes, where the template is the prototypical shape and location of “car” (foreground) in a scene (background). Note that this assumption is not ad hoc in our model, in fact, it underpins almost every concept learning method from the classic Deformable Part Models [15] to modern CNNs [17], whose cognitive evidence can be found in [29]. A plausible realization of $M$ and $C$ used in Section 3.2 is illustrated in Figure 3 (c).

$\boldsymbol{X\rightarrow Y\leftarrow M}$. A general $C$ cannot directly affect the labels $Y$ of an image. Therefore, besides the conventional classification model $X\rightarrow Y$, $Y$ is also the effect of the $X$-specific mediation $M$. $M\rightarrow Y$ denotes an obvious causality: the contextual constitution of an image affects the image labels. It is worth noting that even if we do not explicitly take $M$ as an input for the classification model, $M\rightarrow Y$ still holds. The evidence lies in the fact that visual contexts will emerge in higher-level layers of CNN when training image classifiers [72, 74], which essentially serve as a feature map backbone for modern visual detection that highly relies on contexts, such as Fast R-CNN [16] and SSD [36]. To think conversely, if $M\not\rightarrow Y$ in Figure 3 (a), the only path left from $C$ to $Y$: $C\rightarrow X\rightarrow Y$, is cut off conditional on $X$, then no contexts are allowed to contribute to the labels by training $P(Y|X)$, and thus we would never uncover the context, e.g., the seed areas. So, WSSS would be impossible.

So far, we have pinpointed the role of context $C$ played in the causal graph of image-level classification in Figure 3 (a). Thanks to the graph, we can clearly see how $C$ confounds $X$ and $Y$ via the backdoor path $X\leftarrow C\rightarrow M\rightarrow Y$: even if some pixels in $X$ have nothing to do with $Y$, the backdoor path can still help to correlate $X$ and $Y$, resulting the problematic pseudo-masks in Figure 2. Next, we propose a causal intervention method to remove the confounding effect.

We propose to use causal intervention: $P(Y|do(X))$, as the new image-level classifier, which removes the confounder $C$ and pursues the true causality from $X$ to $Y$ so as to generate better CAM seed areas. As the “physical” intervention — collecting objects in any context — is impossible, we apply the backdoor adjustment [44] to “virtually” achieve $P(Y|do(X))$. The key idea is to 1) cut off the link $C\rightarrow X$ in Figure 3 (b), and 2) stratify $C$ into pieces $C=\{c\}$. Formally, we have:

where $f(\cdot)$ is a function defined later in Eq. (3). As $C$ is no longer correlated with $X$, the causal intervention makes $X$ have a fair opportunity to incorporate every context $c$ into $Y$’s prediction, subject to a prior $P(c)$.

However, $C$ is not observable in WSSS, let alone stratifying it. To this end, as illustrated in Figure 3 (c), we use the class-specific average mask in our proposed Context Adjustment (CONTA) to approximate the confounder set $C=\{{c}_{1},{c}_{2},...,{c}_{n}\}$, where $n$ is the class size in dataset and $c\in\mathbb{R}^{h\times w}$ corresponds to the $h\times w$ average mask of the $i$-th class images. $M$ is the ${X}$-specific mask which can be viewed as a linear combination of $\{c\}$. Note that the rationale behind our $C$’s implementation is based on the definition of context: the relationships among the objects [38], and thus each stratification is about one class of object interacting with others (i.e., the background). So far, how do we obtain the unobserved masks? In CONTA, we propose an iterative procedure to establish the unobserved $C$.

Figure 4 illustrates the overview of CONTA. The input is training images with only class labels (Training Data 1, $t=0$), the output is a segmentation model ($t=T$), which is trained on CONTA generated pseudo-masks (Training Data 2). Before we delve into the steps below, we highlight that CONTA is essentially an EM algorithm [66], if you view Eq. (1) as an objective function (where we omit the model parameter $\Theta$) of observed data $X$ and missing data $C$. Thus, its convergence is theoretically guaranteed. As you may realize soon, the E-step is to calculate the expectation ($\sum\nolimits_{c}$ in Eq. (1)) over the estimated masks in $C|(X,\Theta_{t})$ (Step 2, 3, 4); and the M-step is to maximize Eq. (1) for $\Theta_{t+1}$ (Step 1).

Step 1. Image Classification. We aim to maximize $P(Y|do(X))$ for learning the multi-label classification model, whereby the subsequent CAM will yield better seed areas. Our implementation for Eq. (1) is:

where $\mathds{1}$ is 1/0 indicator, $s_{i}=f(X,M_{t};\theta^{i}_{t})$ is the $i$-th class score function, consisting of a class-shared convolutional network on the channel-wise concatenated feature maps $[X,M_{t}]$, followed by a class-specific fully-connected network (the last layer is based on a global average pooling [34]). Overall, Eq. (2) is a joint probability over all the $n$ classes that encourages the ground-truth labels $i\in Y$ and penalizes the opposite $i\notin Y$. In fact, the negative log-likelihood loss of Eq. (2) is also known as the multi-label soft-margin loss [49]. Note that the expectation $\sum\nolimits_{c}$ is absorbed in $M_{t}$, which will be detailed in Step 4.

Step 2. Pseudo-Mask Generation. For each image, we can calculate a set of class-specific CAMs [74] using the trained classifier above. Then, we follow the conventional two post-processing steps: 1) We select hot CAM areas (subject to a threshold) for seed areas [2, 63]; and 2) We expand them to be the final pseudo-masks [1, 26].

Step 3. Segmentation Model Training. Each pseudo-mask is used as the pseudo ground-truth for training any standard supervised semantic segmentation model. If $t=T$, this is the model for delivery; otherwise, its segmentation mask can be considered as an additional post-processing step for pseudo-mask smoothing. For fair comparisons with other WSSS methods, we adopt the classic DeepLab-v2 [9] as the supervised semantic segmentation model. Performance boost is expected if you adopt more advanced ones [32].

Step 4. Computing $\boldsymbol{M_{t+1}}$. We first collect the predicted segmentation mask $X_{m}$ of every training image from the above trained segmentation model. Then, each class-specific entry $c$ in the confounder set $C$ is the averaged mask of $X_{m}$ within the corresponding class and is reshaped into a $hw\times 1$ vector. So far, we are ready to calculate Eq. (1). However, the cost of the network forward pass for all the $n$ classes is expensive. Fortunately, under practical assumptions (see Appendix 2), we can adopt the Normalized Weighted Geometric Mean [68] to move the outer sum $\sum_{c}P(\cdot)$ into the feature level: $\sum_{c}P(Y|X,M)P(c)\approx P(Y|X,M=\sum_{c}f(X,c)P(c))$, thus, we only need to feed-forward the network once. We have:

where $\alpha_{i}$ is the normalized similarity (softmax over $n$ similarities) between $X_{m}$ and the $i$-th entry $c_{i}$ in the confounder set $C$. To make CONTA beyond the dataset statistics per se, $P(c_{i})$ is set as the uniform $1/n$. $\textbf{W}_{1},\textbf{W}_{2}\in\mathbb{R}^{n\times hw}$ are two learnable projection matrices, which are used to project $X_{m}$ and $c_{i}$ into a joint space. $\sqrt{n}$ is a constant scaling factor that is used as for feature normalization as in [62].

Datasets. PASCAL VOC 2012 [14] contains 21 classes (one background class) which includes 1,464, 1,449 and 1,456 images for training, validation (val) and test, respectively. As the common practice in [1, 63], in our experiments, we used an enlarged training set with 10,582 images, where the extra images and labels are from [19]. MS-COCO [35] contains 81 classes (one background class), 80k, and 40k images for training and val. Although pixel-level labels are provided in these benchmarks, we only used image-level class labels in the training process.

Evaluation Metric. We evaluated three types of masks: CAM seed area mask, pseudo-mask, and segmentation mask, compared with the ground-truth mask. The standard mean Intersection over Union (mIoU) was used on the training set for evaluating CAM seed area mask and pseudo-mask, and on the val and test sets for evaluating segmentation mask.

Baseline Models. To demonstrate the applicability of CONTA, we deployed it on four popular WSSS models including one seed area generation model: SEAM [63], and three seed area expansion models: IRNet [1], DSRG [22], and SEC [26]. Specially, DSRG requires the extra saliency mask [23] as the supervision. General architecture components include a multi-label image classification model, a pseudo-mask generation model, and a segmentation model: DeepLab-v2 [9]. Since the experimental settings of them are different, for fair comparison, we adopted the same settings as reported in the official codes. The detailed implementations of each baseline + CONTA are given in Appendix 3.

Our ablation studies aim to answer the following questions. Q1: Does CONTA merely take the advantage of the mask refinement? Is $M_{t}$ indispensable? We validated these by concatenating the segmentation mask (which is more refined compared to the pseudo-mask) with the backbone feature map, fed into classifiers. Then, we compared the newly generated results with the baseline ones. Q2: How many rounds? We recorded the performances of CONTA in each round. Q3: Where to concatenate $M_{t}$? We adopted the channel-wise feature map concatenation $[X,M_{t}]$ on different blocks of the backbone feature maps and tested which block has the most improvement. Q4: What is in the confounder set? We compared the effectiveness of using the pseudo-mask and the segmentation mask to construct the confounder set $C$.

Due to page limit, we only showed ablation studies on the state-of-the-art WSSS model: SEAM [63], and the commonly used dataset – PASCAL VOC 2012; other methods on MS-COCO are given in Appendix 4. We treated the performance of the fully-supervised DeepLab-v2 [9] as the upperbound.

A1: Results in Table 1 (Q1) show that using the segmentation mask instead of the proposed $M_{t}$ (concatenated to block-5) is even worse than the baseline. Therefore, the superiority of CONTA is not merely from better (smoothed) segmentation masks and $M_{t}$ is empirically indispensable.

A2: Here, $[X,M_{t}]$ was applied to block-5, and the segmentation masks were used to establish the confounder set $C$. From Table 1 (Q2), we can observe that the performance starts to saturated at round 3. In particular, when round $=3$, CONTA can achieve the unanimously best mIoU on CAM, pseudo-mask, and segmentation mask. Therefore, we set #round $=3$ in the following CONTA experiments. We also visualized some qualitative results of the pseudo-masks in Figure 5. We can observe that CONTA can gradually segment clearer boundaries when compared to the baseline results, e.g., person’s leg vs. horse, person’s body vs. sofa, chair’s leg vs. background, and horse’s leg vs. background.

A3: In addition to $[X,M_{t}]$ on various backbone blocks, we also reported a dense result, i.e., $[X,M_{t}]$ on block-2 to block-5. In particular, $[X,M_{t}]$ was concatenated to the last layer of each block. Before the feature map concatenation, the map size of $M_{t}$ should be down-sampled to match the corresponding block. Results in Table 1 (Q3) show that the performance at block-2/-3 are similar, and block-4/-5 are slightly higher. In particular, when compared to the baseline, block-5 has the most mIoU gain by 1.1% on CAM, 2.3% on pseudo-mask, and 1.8% on segmentation mask. One possible reason is that feature maps at block-5 contain higher-level contexts (e.g., bigger parts, and more complete boundaries), which are more consistent with $M_{t}$, which are essential contexts. Therefore, we applied $[X,M_{t}]$ on block-5.

A4: From Table 1 (Q4), we can observe that using both of the pseudo-mask and the segmentation mask established $C$ ($C_{\textrm{Pseudo-Mask}}$ and $C_{\textrm{Seg. Mask}}$) can boost the performance when compared to the baseline. In particular, the segmentation mask has a larger gain. The reason may be that the trained segmentation model can smooth the pseudo-mask and thus using higher-quality masks to approximate the unobserved confounder set is better.

To demonstrate the applicability of CONTA, in addition to SEAM [63], we also deployed CONTA on IRNet [1], DSRG [22], and SEC [26]. In particular, the round was set to $3$ for SEAM, IRNet and SEC, and was set to $2$ for DSRG. Experimental results on PASCAL VOC 2012 are shown in Table 2. We can observe that deploying CONTA on different WSSS models improve all their performances. There are the averaged mIoU improvements of 0.9% on CAM, 2.0% on pseudo-mask, and 2.0% on segmentation mask. In particular, CONTA deployed on SEAM can achieve the best performance of 56.2% on CAM and 66.1% on segmentation mask. Besides, CONTA deployed on IRNet can achieve the best performance of 67.9% on the pseudo-mask. The above results demonstrate the applicability and effectiveness of CONTA.

Table 3 lists the overall WSSS performances. On PASCAL VOC 2012, we can observe that CONTA deployed on IRNet with ResNet-50 [21] achieves the very competitive 65.3% and 66.1% mIoU on the val set and the test set. Based on a stronger backbone ResNet-38 [67] (with fewer layers but wider channels), CONTA on SEAM achieves state-of-the-art 66.1% and 66.7% mIoU on the val set and the test set, which surpasses the previous best model 1.2% and 1.0%, respectively. On MS-COCO, CONTA deployed on SEC with VGG-16 [54] achieves 23.7% mIoU on the val set, which surpasses the previous best model by 1.3% mIoU. Besides, on stronger backbones and WSSS models, CONTA can also boost the performance by 0.9% mIoU on average.

Figure 6 shows the qualitative segmentation mask comparisons between SEAM+CONTA and SEAM [63]. From the first four columns, we can observe that CONTA can make more accurate predictions on object location and boundary, e.g., person’s leg, dog, car, and cow’s leg. Besides, we also show two failure cases of SEAM+CONTA in the last two columns, where bicycle and plant can not be well predicted. One possible explanation is that the segmentation mask is directly obtained from the $8\times$ down-sampled feature maps, so some complex-contour objects can not be accurately delineated. This problem may be alleviated by using the encoder-decoder segmentation model, e.g., SegNet [4], and U-Net [46]. More visualization results are given in Appendix 5.

Table A1, Table A2, and Table A3 show ablation results of IRNet+CONTA, DSRG+CONTA, and SEC+CONTA on PASCAL VOC 2012, respectively. We can observe that IRNet+CONTA and SEC+CONTA can achieve the best performance at round$=3$, and DSRG+CONTA can achieve the best mIoU score at round$=2$. In addition to results of SEAM+CONTA in our main paper, we can see that IRNet+CONTA can achieve the second best mIoU results: $48.8\%$ on CAM, $67.9$% on pseudo-mask, and $65.3$% on segmentation mask.

Table A4, Table A5, Table A6, and Table A7 show the respective ablation results of SEAM+CONTA, IRNet+CONTA, DSRG+CONTA, and SEC+CONTA on MS-COCO. We can see that SEAM+CONTA, IRNet+CONTA and, SEC+CONTA can achieve the top mIoU at round$=3$, and DSRG+CONTA can achieve the best performance at round$=2$. In particular, we see that the mIoU scores of IRNet+CONTA are the best on MS-COCO as respectively $28.7\%$ on CAM, $35.2\%$ on pseudo-mask, and $33.4\%$ on segmentation mask.