Deep Self-cleansing for Medical Image Segmentation with Noisy Labels

The proposed framework can use existing segmentation networks as its backbone. Given the U-Net architecture’s effectiveness in combining high- and low-level features for medical image segmentation [29], we employ an enhanced version of U-Net with residual connections as the segmentation backbone in this paper. Our backbone comprises three down-sampling stages and three up-sampling stages to extract multi-level and multi-scale features progressively. The specific network structure is detailed in Table 1. To ensure that the backbone network can extract effective features and be prepared for subsequent noisy label cleansing, we initially warm up the backbone network for 50 epochs.

Existing pixel-level cleansing methods generate pseudo-labels for all samples. However, in real-world scenarios, annotated samples often contain varying levels of label noise. Modifying all labels indiscriminately can introduce new noise into clean and low-noise labels, thereby impacting the model’s stability and performance. Therefore, the first step in label self-cleansing is to filter out noisy labels for cleansing while preserving the clean labels.

Gaussian Mixture Model for Label Classification: It has been observed that neural networks tend to first fit clean labels during the early training stage [1, 6]. During this period, the network’s predictions usually result in smaller losses for clean labels, whereas the losses for noisy labels are typically larger. Therefore, our LFM leverages the loss characteristic to filter out noisy samples. Specifically, we use the Dice Similarity Coefficient (DSC) loss [22], which is the most commonly used metric in segmentation, as an example to illustrate this process.

Let $X={\left\{x_{i}\right\}}_{i=1}^{N}$ denote the input images (where $N$ is the number of training samples), $Y={\left\{y_{i}\right\}}_{i=1}^{N}$ denote the manual labels, and $\hat{Y}={\left\{\hat{y}_{i}\right\}}_{i=1}^{N}$ denote the network predictions, the DSC loss between $\hat{Y}$ and $Y$ is formulated as:

Given the flexibility of the Gaussian Mixture Model (GMM) in fitting distributions with varying sharpness [15], we employ a two-component GMM to model per-sample loss distributions and to classify the training labels into Clean (to be preserved) and Noisy (to be cleansed) categories. After normalizing $L_{DSC}$ into the range of [0,1], we use the Exception Maximization algorithm [23] to fit a two-component GMM to the distribution of $L_{DSC}$. This GMM allows us to compute the posterior probabilities of each label $y_{i}$ being classified as either Clean or Noisy. These posterior probabilities are denoted as $w_{Clean}^{(i)}$ and $w_{Noisy}^{(i)}$, respectively:

where $g_{small}$ is the gaussian component with the smaller mean value, and $g_{large}$ is the gaussian component with the larger mean value. We classify each label $y_{i}$ as Clean if $w_{Clean}^{(i)}>w_{Noisy}^{(i)}$; otherwise, it is classified as Noisy.

Cascaded Label Filtering: In this paper, our network is devised to filter and cleanse noisy labels through pseudo-labeling every $k$ epochs, making the training stage an iterative cleansing process. To ensure efficient label cleansing and minimize potential noise introduced by pseudo-labeling, we retain both Clean manual labels and Clean previous pseudo-labels (generated in the previous iteration) and modify only the remaining labels. To achieve this, we propose a cascaded label filtering algorithm using two GMMs to filter the labels that require cleansing.

Let $Y^{\prime}={\left\{y^{\prime}_{i}\right\}}_{i=1}^{N}$ denote the pseudo label set, initially containing all-zero images, and let $L_{DSC}^{\prime}={\{{l^{\prime}_{i}\}}}_{i=1}^{N}$ denote losses between $\hat{Y}$ and $Y^{\prime}$. We first fit a GMM (termed GMM-1 in Fig. 2) to $L_{DSC}$ to classify $Y$ into Noisy and Clean categories. Subsequently, we fit another GMM (termed GMM-2 in Fig. 2) to $L^{\prime}_{DSC}$ to categorize $Y^{\prime}$ into Noisy and Clean categories. Finally, we cleanse only those samples whose both manual label and pseudo label are identified as Noisy. The new filtering criteria is as follows:

• -

  If a sample $x_{i}$ has a Clean manual label $y_{i}$, its pseudo label $y^{\prime}_{i}$ is kept the same as $y_{i}$, thereby preserving the Clean manual label;

• -

  If a sample $x_{i}$ has a Noisy manual label and a Clean pseudo label, $y^{\prime}_{i}$ remains unchanged, thus preserving Clean previous pseudo-label;

• -

  If both the manual label and pseudo label of $x_{i}$ are Noisy, the sample should be filtered out. And its corresponding $y^{\prime}_{i}$ will be cleansed by the LCM with newly generated pseudo labels.

After each cleansing iteration, $Y^{\prime}$ consists of three parts as shown in Fig. 2: 1) preserved Clean manual labels, 2) preserved Clean pseudo labels generated in the previous iteration, and 3) newly generated pseudo-labels. By maintaining a portion of low-noise manual labels, $Y^{\prime}$ ensures the stability of the training process and mitigates cumulative errors to a certain extent.

Sec. 2.2 introduces how LFM preserves clean labels and filters out noisy labels to be cleansed. Here, we elaborate on the details of LCM based on pseudo-labeling. Known pseudo-labeling methods typically measure feature similarity between pixels and target prototypes to compute new classification scores. Prototypes are often calculated as either the average feature of the suspected region (network predictions) or as several feature vectors of discrete representative pixels. However, these methods struggle to accurately represent local characteristics of targets, leading to inaccuracies in pseudo-label generation. To address this issue, LCM refines the process by selecting the most representative region from the suspected area as prototypes, while discarding low-confidence regions. The features from this refined representative region are then used as prototype features for calculating pseudo-labels.

Prototype Construction: For an input image $x_{i}$ (abbreviated as $x$), its feature map, i.e., the output of "Residual_block_6" in Table 1, is denoted as $f$, and the suspected target region is denoted as $R$. We construct prototypes based on representative features from $f$. Specifically, we divide $f$ into $K\times K$ grids ($K=64$ in experiments) and select the grids most representative of $R$. The features of these representative grids are then used as prototypes. Note that we process feature maps based on grids instead of pixels, as calculating the representation degree for each pixel sequentially is highly memory-intensive.

Let $G={\{g_{j}\}}_{j=1}^{K^{2}}$ denote the grids, the feature of a grid $g_{j}$ is an aggregated feature calculated as follows:

where $p$ is a pixel position in $g_{j}$ and $f(p)$ is the feature vector of $p$. The representation degree $D$ of each grid can be measured by the following equation:

where $\cos\ (*,*)$ denotes the cosine similarity. $D(g_{j})$ reflects the similarity between $g_{j}$ and $R$. A larger $D(g_{j})$ indicates that $g_{j}$ is more representative of target areas. Fig. 3 provides examples of grid-based representation maps. After normalizing $D$ to the range [0,1], we set a threshold $\sigma$ (empirically set to 0.7) to extract grids with $D>\sigma$ as representative grids. Finally, the features of these representative grids, calculated using Eq. 4, are used as prototypes, denoted as $PROTO={\{proto_{m}\}}_{m=1}^{M}(M<K^{2})$.

Pseudo Labeling: Subsequently, LCM generates a pseudo label for $x$ by updating the classification score of each pixel. For a pixel $q\in x$, its new foreground probability $P_{fg}(q)$ is calculated by measuring feature similarity between $q$ and prototypes:

The final pseudo label can be obtained by converting the soft probability map into a binary mask with a threshold $\mu$ (empirically set to 0.7).

After obtaining the pseudo-labels, we further refine them through post-processing, which involves two steps: 1) Removing connected foreground regions with small areas. Since the generated pseudo-labels may contain fragmented regions, we empirically remove any connected foreground regions whose area is smaller than 10% of the total foreground area; 2) Filling small holes within the foreground. Similarly, we fill any holes with an area smaller than 10% of the total foreground area.

Loss Function: We devise a hybrid loss $L_{\text{hybrid}}$ to optimize the proposed network, which can be expressed as:

where $L_{DSC}$ measures the difference between network outputs and manual labels, while $L^{\prime}_{DSC}$ reflects the difference between outputs and pseudo labels. The coefficients $\lambda_{1}(e)$ and $\lambda_{2}(e)$ are two epoch-dependent parameters, which are defined as:

where $e$ represents the training epoch, $E$ is the total training epochs ($E=500$), $E_{1}$ is the warm-up epochs ($E_{1}=E/10$), and $E_{2}=E/2$.

Training Pipeline: The entire training pipeline is illustrated in Fig. 4. In the early training stage, our network relies only on manual labels to learn features. After a warm-up period of 50 epochs, the network begins to update pseudo-labels every $k$ epochs (set to 5 in our experiments), with the contribution of pseudo-labels gradually increasing. As the training becomes stable, the network is fine-tuned using only pseudo-label set. It is important to note that the pseudo-label set always includes some Clean manual labels to ensure that the training process remains controlled.

Our method is implemented by PyTorch 1.5.0 [26] and deployed on an NVIDIA GTX 2080 Ti GPU (12 GB). We use the SGD optimizer [3] with a mini-batch size of 16. The initial learning rate is set to $4\times 10^{-4}$ and is divided by 10 every 50 epochs.

Two datasets, an in-house CT dataset of liver tumors (HCC dataset) and a public MRI dataset for automatic cardiac diagnosis (ACDC dataset), are used to verify the effectiveness of the proposed method.

The HCC dataset [38] is an in-house liver tumor dataset collected from the First Affiliated Hospital of Zhejiang University School of Medicine, consisting of 231 cases of CT images (30,828 CT slices in total) of hepatocellular carcinoma. Each case has an intra-slice resolution of $512\times 512$, a slice thickness of 0.5 mm, and an inter-plane resolution varying from $\rm 0.56\times 0.56mm^{2}$ to $\rm 0.85\times 0.85mm^{2}$. The tumor regions within each sample in the dataset is delineated by experienced radiologists. In our study, all slices are resized to $256\times 256$ and truncated into the range of [-70, 180] Hu to eliminate irrelevant tissues. The dataset is randomly divided into three parts: 139 CT scans for training, 46 CT scans for validation, and 46 CT scans for testing.

The ACDC dataset [2] is a publicly available dataset for automated cardiac diagnosis, acquired at the University Hospital of Dijon (France). It consists of 200 MRI scans from 100 patients, obtained using two MRI scanners with magnetic field strengths of 1.5T and 3.0T. The scans have in-plane resolutions ranging from $\rm 0.70\times 0.70\ mm^{2}$ to $\rm 1.92\times 1.92\ mm^{2}$, and through-plane resolutions ranging from 5 mm to 10 mm. All data were collected at the End-Diastolic and End-Systolic stages, with annotations provided for the left ventricle (LV), right ventricle (RV), and myocardium (Myo). Following  [21], we extracted $128\times 128$ patches centered on cardiac regions for our experiments. These patches were further split into 1488 patches for training, 414 patches for validation and 414 patches for testing.

To comprehensively evaluate our method under disturbance from varying levels of label noise, we augment the training set by adding label noise of different proportions and amplitudes.

For the HCC dataset, we randomly select $\alpha_{1}$ ($\alpha_{1}\in\{30\%,50\%,70\%\}$) proportion of images from the training set and randomly add missing-label noise or boundary noises. Specifically, missing-label noise is introduced by removing annotations, while boundary noise is added by randomly dilating or eroding tumor masks (labels) by $\beta_{1}\pm 3$ ($\beta_{1}\in\{5,10\}$) pixels. This augmentation process results in seven different training sets (including the original set), each corrupted by varying levels of label noise.

Likewise, we generate multiple training sets with different levels of label noise for the ACDC dataset. We randomly select $\alpha_{2}$ ($\alpha_{2}\in{30\%,50\%,70\%}$) of the images from the training set and introduce either missing-label noise or boundary noise. Missing-label noise is added by removing the organ masks, while boundary noise is applied by randomly dilating or eroding the organ masks by $\beta_{2}\pm 3$ pixels ($\beta_{2}\in\{5,10\}$).

Competing Methods: To evaluate the effectiveness of the proposed method, we first compare it with the Baseline method, i.e., the backbone network without any noise reduction strategy. Besides, we compare our method with four relevant noise-resistance segmentation methods:

• 1.

  Pick-and-learn (PL) [43], an image-level cleansing method that assesses image-level quality of labels in a training batch and assigns lower weights for noisy labels when computing losses.

• 2.

  Confident-learning (CL) [36], a pixel-level cleansing method that identifies noisy regions within labels and generates pseudo-labels using Confident Learning [24].

• 3.

  Joint Co-Regularization (JoCoR) [33],a pixel-level cleansing method that uses two networks to select confident pixels (regions) for cross-training.

• 4.

  Superpixel-guided iterative learning (SP) [16], a pixel-level cleansing method that uses superpixel representations to guide noisy label correction.

To ensure the fairness of the comparison, all methods adopt the same segmentation backbone (as described in Sec. 2.1) and are trained by 500 epochs. Besides, to thoroughly assess the noise-resistance capability of each method, we evaluate the performance of both the B-model and L-model for each method. The B-model represents the best checkpoint selected using the validation set, reflecting the actual segmentation performance on the testing set. Besides, the L-model corresponds to the final checkpoint obtained at the $500^{th}$ epoch, reflecting the stability and sustained noise-resistance performance over time.

Evalutation Metrics: Following PL [43] and CL [36], we use the Dice Similarity Coefficient (DSC) metric to quantify segmentation performance by measuring the similarity between the network outputs and the ground truth labels. A higher DSC score indicates better segmentation results and greater robustness to label noise.

In this section, we analyze the comparative experimental results between our method and competing methods on the HCC dataset, both qualitatively and quantitatively. First, we analyze the qualitative results from the visual perspective. Fig. 5 shows several examples of the segmentation results (produced by B-model with $\alpha_{1}=70\%$, $\beta_{1}=10$), where difference maps highlight the discrepancies between network-produced results and ground truths. It is evident that the Baseline network is significantly impacted by noisy samples, leading to obvious false negatives (e.g., Case #2 and Case #3). While the four competing methods exhibit some noise-resistance ability, there still remains a notable gap between their segmentation results and the ground truths. The issue with PL is that it addresses only image-level label noise by reducing the influence of noisy samples but fails to account for disturbances from local noisy regions within each sample. In contrast, the three pixel-level methods (CL, JoCoR, SP) identify pixel-level noise within all samples indiscriminately and rectify the training process. These methods may either introduce extra noise into clean labels or lead to decreased network efficiency and stability. Particularly, CL performs label cleansing in a one-shot manner, meaning the generated pseudo-labels are fixed and cannot be further refined as training progresses. In contrast, our method demonstrates superior performance in noise reduction and achieves optimal segmentation results. By combining the advantages of both image-level and pixel-level methods, our approach preserves clean samples while effectively cleansing noisy ones during training. Furthermore, our method is an iterative method that updates pseudo-labels to reduce noise throughout the training process. As training progresses, these pseudo-labels are continuously refined, enabling the network to better learn robust target features.

Further, we present quantitative results for different methods in Table 2. The Baseline method, trained on the noise-free training set, provides the upper bound for performance. The B-model, representing the best checkpoint selected using the validation set, reflects the actual segmentation performance of the network. The L-model, corresponding to the last checkpoint obtained at the $500^{th}$ epoch, indicates network stability and sustained noise-resistance ability during training. From Table 2, it is evident that PL demonstrates robustness to label noise in the B-model and shows some sustained noise resistance in the L-model. CL exhibits noise resistance in the B-model but shows poor stability and sustained noise robustness (as measured by the L-model). As the level of noise increases, (with larger $\alpha_{1}$ and $\beta_{1}$), both methods experience a more significant decline in stability and noise resistance. JoCoR and SP show improvements in stability (as reflected in the L-model), but the results are still not ideal. In contrast, our method achieves the best results in both the B-model and L-model. More importantly, as the noise level increases, our method shows increasingly effective noise-resistance performance. Based on both quantitative and qualitative analyses, it can be concluded that our method outperforms competing methods in terms of noise-resistance capability and stability.

To assess the generalization ability of the proposed method across different datasets and segmentation tasks, we conduct additional validation experiments on the ACDC dataset. Specifically, we focus on segmenting three different organs—left ventricle (LV), right ventricle (RV), and myocardium (Myo)—from MRI images. Fig. 6 provides visual examples of segmentation results produced by different methods (produced by B-model with $\alpha_{2}=70\%$, $\beta_{2}=10$). It is shown that our method produces results that are most similar to the ground truths. Table 3 presents the quantitative results, including the analysis of both the B-model and L-model for each method. The results demonstrate that the proposed method outperforms competing methods across the three-organ segmentation tasks. Besides, the improvement of the stability and sustained noise-resistance ability of the network is particularly evident compared with other four methods. These findings on the ACDC dataset demonstrate the generalization capability of our method, implying its noise-resistance ability across different datasets and segmentation tasks.

To verify the effectiveness of each component of the proposed method, we conduct an ablation study on the HCC dataset (with $\alpha_{1}=70\%$, $\beta_{1}=10$). We start from the baseline network (the segmentation backbone without any noise-cleansing strategy) and incrementally add the proposed components. Table. 4 summarizes the results of ablation study. It is observed that the B-model of the baseline achieves the segmentation performance of 59.71% (in DSC) on the test set. However, as training proceeds, the network gradually overfits to noisy samples, the segmentation performance eventually falls back to 51.19% (L-model).

Effectiveness of LCM: We first add LCM to the baseline to assess its effectiveness, whose specific structure is described in Section 2.3. After the warm-up training, LCM indiscriminately corrects all samples every 5 epochs without preserving clean samples. Fig. 7 illustrates several examples of cleansed labels produced by the LCM, demonstrating its effectiveness in correcting both missing labels (Case #1) and boundary noise (Cases #2 and #3). Table 4 provides the corresponding quantitative results, which shows that the LCM improves the segmentation performance of the B-model by +1.36% (in DSC) and also enhances the stability of the baseline, resulting in a +2.53% (in DSC) improvement in the performance of the L-model.

Meanwhile, we compare the pseudo-labeling method used in LCM with two other pseudo-labeling methods: 1) LCM (pred), which directly employs network predictions as pseudo labels; 2) LCM (avg), which yields pseudo labels based on averaged prototypes (the averaged features of suspected target regions). The experimental results of the comparison are shown in Table 4. The results indicate that LCM based on representative prototypes produces more accurate pseudo-labels as it generates better representative features of the target regions.

Effectiveness of LFM: Although LCM effectively improves label quality, indiscriminately modifying all labels can corrupt clean and low-noise manual labels, as the generated pseudo-labels still contain some noise. Such noise can accumulate as training progresses, which potentially limits the performance and stability of the network.

To address this issue, we introduce LFM to allow the network to preserve clean samples and correct only noisy samples in each cleansing iteration, thereby reducing the potential noise introduced by pseudo-labeling. LFM achieves this by modeling per-sample loss distributions and classify labels into Clean and Noisy categories. Incorporating LFM ensures that the network is consistently supervised by low-noise manual labels, which stabilizes the training process and prevents error accumulation. As shown in Table 4, LFM significantly boosts performance by +5.95% in the B-model and +9.83% in the L-model, which demonstrates its effectiveness.