Confidence-Weighted Boundary-Aware Learning for Semi-Supervised Semantic Segmentation

We address the problem of semantic segmentation in a semi-supervised setting, where a labeled dataset $\mathcal{D}_{L}=\{(x_{i},y_{i})\}_{i=1}^{N_{L}}$ and an unlabeled dataset $\mathcal{D}_{U}=\{x_{i}\}_{i=1}^{N_{U}}$ are available, with $N_{L}$ and $N_{U}$ representing the number of samples in the labeled and unlabeled datasets, respectively.

We train a teacher model on $\mathcal{D}_{L}$, and use it to generate high-quality pseudo-labels for $\mathcal{D}_{U}$ and train a student model that improves performance on both datasets.

For each unlabeled image $x_{i}\in\mathcal{D}_{U}$, the teacher model, ${f}_{T}$ generates logits, $z_{i}$:

From these logits, we derive pseudo-labels, $s_{i}$ and pixel-wise confidence scores, $p_{i}$:

where $s_{i,j}$ is the pseudo-label for pixel $j$ in image $x_{i}$;
$p_{i,j}\in[0,1]$ represents the confidence score for pixel $j$; and
$k$ is the index of the possible classes.

The loss consists of labeled and unlabeled components:

Confidence-Weighted Cross-Entropy Loss. To address the label noise introduced by inaccurate pseudo-labels from the teacher, we propose a confidence-weighted loss that weights the contribution of each pixel to the overall loss based on its confidence score. The pixel-wise confidence $p_{i}$ is raised to a power $\gamma$ to emphasize high-confidence predictions:

The loss for an unlabeled image, $x_{i}$ is defined as:

where $N_{i}$ is the total number of pixels in image $x_{i}$;
$z^{\prime}_{i}=f_{S}(x_{i})$ are the logits predicted by the student model $f_{S}$; and $\gamma\geq 0$ is the hyperparameter controlling emphasis on high-confidence predictions.

Loss for Labeled Data. For labeled data, we use the standard cross-entropy loss:

where $y_{i,j}$ is the ground truth label for pixel $j$ in image $x_{i}$.

Total Loss. The total loss is a combination of the losses from labeled and unlabeled data:

where $\lambda$ is a balancing parameter between the supervised and unsupervised losses.

We introduce a dynamic threshold mechanism $T$ to adaptively learn and adjust the confidence threshold for filtering low-confidence pseudo-labels, based on the model’s performance during training.

Average Confidence Calculation. For each batch, we calculate the average confidence score $\bar{p}$:

where $\bar{p}$ is the average confidence.

Dynamic Threshold Adjustment. The base threshold $T$ is then adjusted using a logistic function based on the average confidence:

where $T_{0}$ is the initial confidence threshold, $\bar{p}$ is the initial average confidence and $\beta$ is a hyperparameter controlling the sensitivity of the threshold adjustment.

Retaining Pseudo-labels. We retain pseudo-labels where the confidence score exceeds the threshold:

To further reduce the influence of consistently low-confidence pixels, we introduce a confidence decay strategy that progressively reduces the influence of low-confidence pseudo-labels over time:

For low-confidence pixels, we update their confidence scores as:

where $p_{i,j}^{(t)}$ is the confidence score at epoch $t$ and $\alpha\in(0,1)$ is the decay factor.

After the pseudo-labels are refined and filtered through confidence weighting and thresholding, we introduce boundary-aware learning to focus on the more intricate task of boundary delineation and overall improve segmentation performance.

Boundary Detection. We apply Sobel filters to the pseudo-labels $\hat{y}_{i}$ to detect edges [30]. Let $G_{x}$ and $G_{y}$ be the horizontal and vertical gradients, respectively. The gradient magnitude at pixel, $(h,w)$ in image, $n$ is:

A binary boundary mask is then defined as:

where $E_{n,h,w}$ is the gradient magnitude at pixel $(h,w)$; $G_{x}^{n,h,w},G_{y}^{n,h,w}$ are the pixel gradients at $(h,w)$ derived via Sobel filters and $\text{BoundaryMask}_{n,h,w}$ is the binary mask indicating boundaries (if $E_{n,h,w}>0$, else $0$).

Boundary-Aware Loss. To emphasize correct predictions at boundaries, we scale the cross-entropy loss by the boundary mask to encourage the model to focus more on object contours:

where $\text{CE}_{i,j}$ is the pixel-wise cross-entropy at pixel $(i,j)$ and $N_{i}$ is the total number of pixels in image $x_{i}$.

The final loss function combines all components:

Student Model. The student model $f_{S}$ is trained using $\mathcal{D}_{U}$ and the refined pseudo-labeled data $(x_{i},\hat{y}_{i})$:

Teacher Model. Once $f_{S}^{(t)}$ is trained, we set:

This dataset consists of 1,464 training images and 1,449 validation images, annotated with 21 semantic classes including the background.

Cityscapes contains 2,975 training images and 500 validation images, with high-resolution street scenes annotated into 19 classes.

We use the DeepLabV3+ model with ResNet-50 [38] as the backbone for all experiments, initialized with weights pre-trained on ImageNet [39].

For optimization, we employ stochastic gradient descent (SGD) with a momentum of 0.9 and a weight decay of $1\times 10^{-4}$. The learning rate (LR) and scheduler are as follows: Pascal VOC: 0.001, and for Cityscapes: 0.004 due to larger image sizes and batch adjustments.
A polynomial learning rate decay is used, defined as: $\text{LR}_{\text{current}}=\text{LR}_{\text{initial}}\times\left(1-\frac{\text{iter}}{\text{total\_iters}}\right)^{0.9}$.
Batch size for Pascal VOC is set to 16, and for Cityscapes, due to higher resolution, is set to 8. The model is trained for 80 epochs on Pascal and 240 epochs on Cityscapes. It is also important to note, for fair comparisons with existing methods and to reduce computational costs, that our method does not use advanced optimization strategies such as OHEM, auxiliary supervision [18], or SyncBN, which are commonly used in other studies.

We apply similar semi-supervised training settings as most state-of-the-art-methods to ensure fair comparisons. Data augmentation techniques include random horizontal flipping and random scaling (ranging from 0.5 to 2.0) [9]. We use reduced cropping sizes during training compared to CPS [18], ie. 321 × 321 pixels on Pascal VOC and 721 × 721 pixels on Cityscapes to save memory. On unlabeled images, we also utilize color jitter with the same intensity as [25], grayscale conversion, Gaussian blur as stated in [26], and Cutout with randomly filled values. All unlabeled images undergo test-time augmentation as well [18].

We use the standard mean Intersection over Union (mIoU) metric to evaluate segmentation performance. No ensemble techniques where used for all evaluations.

In our experiments, we use specific hyperparameters with default settings, and their impact is further analyzed in the ablation studies. Confidence Weighting ($\gamma$) is set to a default value of 1.0, to balance the contribution of high- and low-confidence pseudo-labels in the confidence-weighted loss. For Dynamic Thresholding ($T$), the base threshold ($T_{0}$) is 0.6, allowing the model to learn filter pseudo-labels dynamically based on its performance, thereby mitigating confirmation bias. Additionally, a sensitivity parameter ($\beta$) with a default value of 0.5 ensures stability by preventing the thresholds from becoming excessively lenient or overly strict. Lastly, the Confidence Decay Factor ($\alpha$) is set to 0.9, to gradually reduces the influence of low-confidence pseudo-labels over time and refine the learning process.

Figure 5 illustrates the convergence behavior and performance of CW-BASS compared to the self-training method ST++ [9] on the Cityscapes dataset during the initial 20 epochs, utilizing the 1/16 partition protocol. CW-BASS achieves faster stabilization due to its confidence-weighted loss, which prioritizes high-confidence predictions and mitigates the influence of noisy pseudo-labels. Furthermore, its boundary-aware learning enhances segmentation accuracy near object edges, contributing to more rapid convergence. In contrast, ST++ depends on iterative retraining after selecting reliable images post pseudo-labeling. Overall, CW-BASS integrates swift initial convergence with sustained performance improvements throughout the training process.

Table II summarises data utilisation and training overhead for various approaches, highlighting differences in the number of networks trained, unlabeled data processed per epoch, and total training epochs. CPS has a dual network design and treats the labeled set, $\mathcal{D}_{L}$ without the ground truth as additional unlabeled set, increasing the volume of training data. On the other hand, PS-MT implements consistency learning with perturbations to both teacher and student networks. CPS trains two networks, processing 10.5k and 2.9k unlabeled images per epoch on Pascal VOC and Cityscapes, respectively. PS-MT employs three networks with aggressive augmentations, often doubling or tripling unlabeled data (e.g., 19.8k samples per epoch for the 1/16 Pascal partition) and requires 320–550 epochs on Cityscapes. ST++ trains four networks with slightly fewer unlabeled samples per epoch compared to PS-MT (14.8k on Pascal and 4.1k on Cityscapes, both below 1/16). In contrast, CW-BASS trains two networks with fewer augmentations, using the least unlabeled data (up to 9.9k on Pascal and 2.7k on Cityscapes) and converging in just 80 epochs on Pascal and 240 epochs on Cityscapes, regardless of partition. This balanced approach achieves competitive performance while avoiding the overheads of intensive augmentations or large network ensembles.

Our proposed CW-BASS framework achieves state-of-the-art segmentation performance on both Pascal VOC 2012 and Cityscapes, as shown in Table I.

Results on Pascal VOC. Table I compares performance on the Pascal VOC 2012 dataset using various state-of-the-art (SOTA) semi-supervised semantic segmentation methods. Unlike approaches that use multiple teacher networks and various perturbations, our method relies on a single model for self-training. To highlight our performance, we also specifically compare our results with ST++, an advanced and simple self-training approach.

Using only 1/8 (12.5%) of the labeled training data, our method produces a mean Intersection over Union (mIoU) of 75.81%, outperforming existing state-of-the-art approaches. This also exceeds the performance of the fully supervised baseline by a remarkable 11.2%, demonstrating the effectiveness of our semi-supervised learning method in limited label settings.

Results on Cityscapes. Our method also outperforms state-of-the-art methods on the Cityscapes dataset as shown in Table I. Cityscapes presents more complex, high-resolution urban environments. Under an extremely limited and rarely used setting, where only 1/30 (3.3%) of the training data (100 labeled images) is annotated, our method achieves a remarkable mIoU of 65.87%. This surpasses the supervised baseline by 10.77%, showcasing our model’s efficiency in extremely limited label scenarios and ability to handle complex scene layouts, fine-grained object boundaries, and diverse urban visuals.

Confidence-Weighted Loss ($L_{labeled}$ + $L_{weighted}$) Confidence-Weighting serves as the main component in our method ($L_{\text{labeled}}$) and ($L_{\text{weighted}}$) and achieves a mean Intersection over Union (mIoU) of 73.43%. This result is much higher than the baseline and some previous SOTA methods, underscoring the importance of the weighted loss in prioritizing high-confidence pseudo-labels and effectively mitigate the impact of label noise on model performance.

Boundary-Aware Loss ($L_{boundary}$). Incorporating boundary-aware loss into the confidence-weighting, significantly improves segmentation performance, particularly near object boundaries. This addition results in an mIoU of 74.67%, marking a notable improvement of 6.36%. The findings demonstrate the effectiveness of addressing boundary blur through specialized loss functions.

Dynamic Thresholding Mechanism ($T$). The integration of a dynamic thresholding mechanism enhances the model’s ability to filter out low-confidence pseudo-labels. Using this mechanism in conjunction with $L_{\text{labeled}}$ and $L_{\text{weighted}}$ yields an mIoU of 69.01%, highlighting its role in dynamically refining pseudo-label quality by discarding unreliable predictions.

Confidence Decay Strategy ($\alpha$). The confidence decay strategy gradually reduces the influence of low-confidence pseudo-labels during training. This approach stabilizes learning by improving pseudo-label reliability over time. The configuration ($L_{\text{labeled}}+L_{\text{weighted}}+\alpha$) produces modest improvements, showcasing its ability to complement other strategies by enhancing pseudo-label robustness.

Discussion. The ablation study highlights the Confidence-Weighted Loss and Boundary-Aware Loss as the primary contributors to performance gains. The significant improvement observed with boundary-aware loss highlights the effectiveness of integrating boundary-delineation techniques in semi- supervised frameworks. The Confidence Decay Strategy and Dynamic Thresholding Mechanism address the challenges of noisy pseudo-labels and confirmation bias.

To investigate the impact of our hyperparameter on model performance, we perform experiments using two groups of settings: our standard & default setting (i.e., $\gamma=1.0,\beta=0.5,\alpha=0.85$), which we use for all our experiments, and a more conservative setting ($\gamma=0.5,\beta=1.0,\alpha=1.0$), as shown in Table IV. The standard settings batch yield the best model performance because the parameters are balanced, whereas the conservative settings yield an mIoU 4.41% lower than the standard settings.

For Confidence weighting, a $\gamma$ value of 0.5 reduces the influence of these labels, making the model more conservative to minimize the risk of overfitting to incorrect labels. On the other hand, setting ($\gamma$) to 2.0 increases the emphasis on high-confidence pseudo-labels, encouraging the model to reinforce confident predictions more aggressively.

Dynamic thresholding ($T$) involves a base threshold ($T_{0}$) set at 0.6, which is adjusted using a parameter $\beta$. A higher $\beta$ value (e.g., 1.0) imposes stricter filtering, discarding more low-confidence pseudo-labels. To maintain stability, thresholds are constrained within a range of 0.3 to 0.8, ensuring they do not become excessively lenient or overly strict.

Finally, the confidence decay factor ($\alpha$), with a default value of 0.9, can be adjusted for specific training needs. Lowering $\beta$ to 0.85 accelerates the decay, which is particularly useful when a more aggressive reduction of noisy pseudo-labels is required.