Using Unreliable Pseudo-Labels for Label-Efficient Semantic Segmentation

The positive sample is the same for all anchors from the same class. It is the prototype of class $c$ defined in Eq. (15):

We define a binary variable $n_{ij}(c)$ to identify whether the $j$-th pixel of image $i$ is qualified to be negative samples of class $c$.

where $n_{ij}^{l}(c)$ and $n_{ij}^{u}(c)$ are indicators of whether the $j$-th pixel of labeled and unlabeled image $i$ is qualified to be negative samples of class $c$ respectively.

For $i$-th labeled image, a qualified negative sample for class $c$ should be: (a) not belonging to class $c$; (b) difficult to distinguish between class $c$ and its ground-truth category. Therefore, we introduce the pixel-level category order $\mathcal{O}_{ij}=\texttt{argsort}(\mathbf{p}_{ij})$. Obviously, we have $\mathcal{O}_{ij}(\arg\max\mathbf{p}_{ij})=0$ and $\mathcal{O}_{ij}(\arg\min\mathbf{p}_{ij})=C-1$.

where $r_{l}$ is the low-rank threshold. Here, a small $r_{l}$ represents an aggressive strategy, which tries to make full use of unreliable predictions but may introduce too much noise. We set to it $3$. Two indicators reflect (a) and (b), respectively.

For $i$-th unlabeled image, a qualified negative sample for class $c$ should: (a) be unreliable; (b) probably not belongs to class $c$; (c) not belongs to most unlikely classes. Similarly, we also use $\mathcal{O}_{ij}$ to define $n_{ij}^{u}(c)$:

where $r_{h}$ is the high-rank threshold and is set to $20$. Finally, the set of negative samples of class $c$ is

Due to the long tail phenomenon of the dataset, negative candidates in some particular categories are extremely limited in a mini-batch. In order to maintain a stable number of negative samples, we use category-wise memory bank $\mathcal{Q}_{c}$ (FIFO queue) to store the negative samples for class $c$. We simply follow MoCo (He et al., , 2020), setting the queue size to $65,536$. A large queue size contributes diverse negative samples while introducing extra computational costs.

Finally, the whole process to use unreliable pseudo-labels is shown in Algorithm 1. All features of anchors are attached to the gradient, and come from the student hence, while features of positive and negative samples are from the teacher.

This time, as segmentation maps of the whole dataset $\mathcal{D}$ are inaccessible, we cannot sample anchors using ground-truth labels as Eq. (21) does. Instead, we simply regard those pixels with a value of CAM larger than $\beta$ as candidates:

Similarly, positive keys for each query $\mathbf{q}_{c}\in\mathcal{A}_{c}$ are the prototype of class $c$, i.e., $\mathbf{z}_{c}^{+}=\mathbf{z}_{c}^{\mathrm{proto}}$. Similarly, the prototypes are momentum updated described in Eq. (18), but category-wise centroids of the current mini-batch $\mathbf{z}_{c}^{\prime\mathrm{proto}}$ are computed by

Determining negative keys is much easier under the weakly supervised setting because a set of image-level labels is given for each image. Concretely, given a image $\mathbf{x}_{i}$, its image-level label is $\mathbf{y}_{i}\in\{0,1\}^{C}$, where $y_{i}^{c}=1$ indicates that class $c$ exists in this image and vise versa. Therefore, the indicator $n_{ij}(c)$ representing whether pixel $j$ from the $i$-th image is a qualified negative key for class $c$ natually becomes

where the first term $\mathbbm{1}[y_{i}^{c}=0]$ means this image does not contain category $c$ at all, while the second term $\mathbbm{1}[\mathbf{M}^{c}_{ij}<\beta]$ indicates this prediction is unreliable to determine the pixel belongs to category $c$. The prediction becomes a qualified negative key when either of these two conditions is met.

Tab. 1 compares our method with the other state-of-the-art methods on classic PASCAL VOC 2012 Dataset. U${}^{\text{2}}$PL+ outperforms the supervised baseline by $+23.52\%$, $+18.48\%$, $+9.15\%$, $+5.40\%$, and $+7.02\%$ under $1/16$, $1/8$, $1/4$, $1/2$, and “full” partition protocols respectively, indicating that semi-supervised learning fully mines the inherent information of unlabeled images. When comparing to state-of-the-arts, our U${}^{\text{2}}$PL+ outperforms PC²Seg under all partition protocols by $+12.29\%$, $+7.12\%$, $+5.25\%$, $+4.04\%$, and $+5.37\%$ respectively. Note that when labeled data is extremely limited, e.g., when we only have $92$ labeled data, our U${}^{\text{2}}$PL+ outperforms previous methods by a large margin ($+12.29\%$ under $1/16$ split for classic PASCAL VOC 2012), proofing the efficiency of using unreliable pseudo-labels. Furthermore, by introducing extra strategies, U${}^{\text{2}}$PL+ can outperform its previous version U${}^{\text{2}}$PL by $+1.31\%$, $+4.25\%$, $+1.37\%$, $+0.93\%$, and $+0.03\%$ respectively. The fewer labeled images we have, the larger improvement it brings, indicating that U${}^{\text{2}}$PL+ is more capable of dealing with training noise than U${}^{\text{2}}$PL.

Tab. 2 shows the comparison results on blender PASCAL VOC 2012 Dataset. Our method U${}^{\text{2}}$PL+ outperforms all the other methods under various partition protocols. Compared with the supervised baseline, U${}^{\text{2}}$PL+ achieves improvements of $+9.36\%$, $+7.80\%$, $+4.41\%$ and $+3.65\%$ under $1/16$, $1/8$, $1/4$ and $1/2$ partition protocols respectively. Compared with the existing state-of-the-art methods, U${}^{\text{2}}$PL+ surpasses them under all partition protocols. Especially under $1/8$ protocol and $1/4$ protocol, U${}^{\text{2}}$PL+ outperforms AEL by $+1.78\%$ and $+2.15\%$. When comparing U${}^{\text{2}}$PL+ to its previous version U${}^{\text{2}}$PL, it brings improvements of $+0.02\%$, $+0.34\%$, $+0.91\%$, and $+0.28\%$ respectively. Compared to the classic VOC counterpart, it seems that U${}^{\text{2}}$PL+ may struggle to bring significant improvements when we access adequate labeled images (i.e., the number of labeled data is extended from $1,464$ to $10,582$).

Tab. 2 illustrates the comparison results on the Cityscapes val set. U${}^{\text{2}}$PL+ improves the supervised only baseline by $+10.35\%$, $+5.47\%$, $+4.59\%$ and $+1.79\%$ under $1/16$, $1/8$, $1/4$, and $1/2$ partition protocols, and outperforms existing state-of-the-art methods by a notable margin. In particular, U${}^{\text{2}}$PL+ outperforms AEL by $+1.64\%$, $+2.45\%$, $+1.54\%$ and $+0.61\%$ under $1/16$, $1/8$, $1/4$, and $1/2$ partition protocols Compared to its previous version U${}^{\text{2}}$PL, U${}^{\text{2}}$PL+ achieves improvements of $+1.19\%$, $+1.52\%$, $+0.51\%$, and $+0.50\%$ respectively.

To prove our core insight, i.e., using unreliable pseudo-labels promotes semi-supervised semantic segmentation, we conduct experiments about selecting negative candidates (which is described in Sec. 3.5) with different reliability, i.e., whether to regard only unreliable pseudo-labels to be negative samples of a particular query.

Tab. 3 demonstrates the mIoU results on PASCAL VOC 2012 val set and Cityscapes val set, respectively. Containing “U” in negative keys outperforms other options, proving using unreliable pseudo-labels does help, and our U${}^{\text{2}}$PL+ fully mines the information of all pixels, especially those unreliable ones. Note that “R” in Tab. 3 indicates negative keys in memory banks are reliable. Containing both “U” and “R” means that all features are stored in memory banks without filtering. It is worth noticing that containing features from labeled set $\mathcal{D}_{l}$ brings marginal improvements, but significant improvements are brought because of the introduction of unreliable predictions, i.e., “U” in Tab. 3.

We conduct experiments in Tab. 4 to ablate each component of U${}^{\text{2}}$PL+ step by step. For a fair comparison, all the ablations are under 1/4 partition protocol on the blender PASCAL VOC 2012 Dataset.

Above all, we use no $\mathcal{L}_{c}$ trained model as our baseline, achieving mIoU of $77.33\%$ (CutMix in Tab. 2). We first ablate components in contrastive learning, including $\mathcal{Q}_{c}$, DPA, PRT, and Un. Simply adding vanilla $\mathcal{L}_{c}$ even contributes to the performance degradation by $-0.27\%$. Category-wise memory bank $\mathcal{Q}_{c}$, along with PRT and high entropy filtering brings an improvement by $+1.41\%$ to vanilla $\mathcal{L}_{c}$. Dynamic Partition Adjustment (DPA) together with high entropy filtering, brings an improvement by $+1.99\%$ to vanilla $\mathcal{L}_{c}$. Note that DPA is a linear adjustment without tuning (refer to Eq. (13)), which is simple yet efficient. For Probability Rank Threshold (PRT) component, we set the corresponding parameter according to Tab. 8. Without high entropy filtering, the improvement decreased significantly to $+0.49\%$.

Then, we ablate components in noisy label learning and its denoising strategies, including PD, MP, and SCE. Introducing Prototypical Denoising (PD) to pseudo-labels improves the performance by $+0.60\%$ to CutMix. Using Symmetric Corss-Entropy (SCE) in computing unsupervised loss, improves the performance by $+0.36\%$ to CutMix. Adding them together brings an improvement by $+0.68\%$. Adopting an extra Momentum Prototype (MP) on the basis of the above two techniques brings an improvement by $+0.91\%$.

Finally, when adding all the contributions together, our method achieves state-of-the-art results under $1/4$ partition protocol with mIoU of $80.21\%$. Following this result, we apply these components and corresponding parameters in all experiments on Tab. 1 and Tab. 2.

To further see the impact of prototypical denoising (PD), we conduct experiments in Tab. 5 to find out how PD affects the performance under different partition protocols of blender PASCAL VOC 2012. From Tab. 5, we can tell that PD brings improvements of $+0.14\%$, $+0.25\%$, $+0.73\%$, and $+0.35\%$ under $1/16$, $1/8$, $1/4$, and $1/2$ partition protocols respectively, indicating that PD manages to enhance the quality of pseudo-labels by weighted averaging predictions from similar feature representations.

We conduct experiments in Tab. 6 to find out the best $\beta$ for momentum prototype (MP) on $1/8$ and $1/4$ partition protocols on blender PASCAL VOC 2012, and $1/4$ and $1/2$ partition protocols on Cityscapes, respectively $\beta=0.999$ yields slightly better than other settings, indicating that our U${}^{\text{2}}$PL+ is quite robust against different $\beta$.

To find the optimal $\xi_{1}$ and $\xi_{2}$ for symmetric cross-entropy (SCE) loss, we conduct experiments in Tab. 7 under $1/8$ and $1/4$ partition protocols on both blender PASCAL VOC 2012 and Cityscapes. When $\xi=1$ and $\xi_{2}=0.1$, it achieves the best performance, better than standard cross-entropy loss (i.e., when $\xi_{1}=1$ and $\xi_{2}=0$) by $+0.29\%$ and $+0.53\%$ on $1/8$ and $1/4$ partition protocols respectively. Note that the performances drop heavily when using only the reserve CE loss (i.e., $\xi_{1}=0$ and $\xi_{2}=1$).

Sec. 3.5 proposes to use a probability rank threshold to balance informativeness and confusion caused by unreliable pixels. Tab. 8 provides a verification that such balance promotes performance. $r_{l}=3$ and $r_{h}=20$ outperform other options by a large margin. When $r_{l}=1$, false negative candidates would not be filtered out, causing the intra-class features of pixels incorrectly distinguished by $\mathcal{L}_{c}$. When $r_{l}=10$, negative candidates tend to become irrelevant with corresponding anchor pixels in semantics, making such discrimination less informative.

Tab. 9 studies the impact of different initial reliable-unreliable partition $\alpha_{0}$. $\alpha_{0}$ has a certain impact on performance. We find $\alpha_{0}=20\%$ achieves the best performance. Small $\alpha_{0}$ will introduce incorrect pseudo labels for supervision, and large $\alpha_{0}$ will make the information of some high-confidence samples underutilized.

The impact of the base learning rate is shown in Tab. 10. Results are based on the blender VOC PASCAL 2012 dataset. We find that 0.001 outperforms other alternatives.

Tab. 10 gives a study on the effect of temperature $\tau$. Temperature $\tau$ plays an important role to adjust the importance of hard samples When $\tau=0.5$, our U${}^{\text{2}}$PL+ achieves the best results. Too large or too small of $\tau$ will have an adverse effect on overall performance.

To further verify the generalization of U${}^{\text{2}}$PL+ towards different network architectures, We apply the network architecture of DAFormer (Hoyer et al., , 2022) on semi-supervised benchmarks in Tab. 11. We train models for 40k iterations instead of 200 epochs for efficient evaluation. U${}^{\text{2}}$PL+ manages to bring significant improvements when using transformer-based models.

To further demonstrate the generalization of U${}^{\text{2}}$PL+, we extend our U${}^{\text{2}}$PL+ to semi-supervised image classification. We implement our U${}^{\text{2}}$PL+ on the basis of FixMatch (Sohn et al., , 2020) and FreeMatch (Wang et al., 2023d, ), and almost no modifications are needed when adapting our U${}^{\text{2}}$PL+ from segmentation to classification. Similarly, we tend to make full use of unreliable image-level pseudo labels under the pipeline of U${}^{\text{2}}$PL+. Our implementation is based on USB (Wang et al., 2022a, ). From Tab. 12, we can tell that incorporating U${}^{\text{2}}$PL+ brings significant improvements.