Noise-Tolerant Hybrid Prototypical Learning with Noisy Web Data

We aim to learn an unbiased few-shot classifier with the aid of additional large-scale web data with noisy labels. Generally, there are two stages in the few-shot classification.

The first stage is the representation learning phase. In this stage, we are given a large clean labeled dataset $\mathcal{D}_{base}=\{\mathcal{D}_{base}^{1},\mathcal{D}_{base}^{2},\ldots,\mathcal{D}_{base}^{L}\}$, where $L$ is the number of categories in the base set. $\mathcal{D}_{base}^{l}$ denotes the set of images for base class $l$. $\mathcal{D}_{base}^{l}=\{(x_{i}^{l},y_{i}^{l})|i=1,...,n\}$, where $n$ is the number of training examples. The base set $\mathcal{D}_{base}$ is leveraged to learn a strong feature extractor $\Phi:x\mapsto\Phi(x)\in\mathbb{R}^{d}$. Here, $d$ denotes the dimension of the feature.

In the second stage, we receive a novel dataset $\mathcal{D}_{novel}$ with few clean labeled examples. The goal is to adapt the learned feature extractor to the novel dataset and train a robust classifier given only few clean examples. We define $\mathcal{D}_{novel}=\{\mathcal{D}_{novel}^{1},\mathcal{D}_{novel}^{2},\ldots,\mathcal{D}_{novel}^{C}\}$ and $C$ is the number of classes in the novel set. For each novel class $c$, $\mathcal{D}_{novel}^{c}=\{(x_{i}^{c},y_{i}^{c})|i=1,...,k\}$, where $k$ is the number of novel examples and $k\ll n$, usually called the $k$-shot setting. Note that the base classes in $\mathcal{D}_{base}$ have no overlaps with the novel classes in $\mathcal{D}_{novel}$.

In this paper, we focus on the second stage of few-shot learning and tackle this challenge with an additional large-scale noisy dataset $\mathcal{D}_{noisy}$, particularly from the Internet. The problem is formulated as learning an unbiased classifier from many freely accessible web images with noisy labels given only few clean labeled images. Combining both clean labeled data and large amounts of noisy ones, we obtain the whole dataset $\widetilde{\mathcal{D}}_{novel}^{c}=\mathcal{D}_{novel}^{c}\cup\mathcal{D}_{noisy}^{c}$ for each novel class $c$. Note that $\mathcal{D}_{novel}^{c}$ is the image set with few clean labeled examples and $\mathcal{D}_{noisy}^{c}$ consists of a large number of images with noisy labels. We expect to identify the relevant images from the noisy set. Training with these relevant images can potentially enhance the generalization of the few-shot classifier.

We extract the features of examples in $\widetilde{\mathcal{D}}_{novel}^{c}$ using the learned feature extractor $\Phi$. The clean and noisy features are represented by the matrix $V^{c}=[\mathbf{v}_{1}^{c},...,\mathbf{v}_{k}^{c},...,\mathbf{v}_{N}^{c}]\in\mathbb{R}^{d\times N}$, where $\mathbf{v}_{i}^{c}=\Phi(x_{i}^{c})\in\mathbb{R}^{d}$, $x_{i}^{c}$ is a training example from $\widetilde{\mathcal{D}}_{novel}^{c}$ and $N$ is the number of training examples in $\widetilde{\mathcal{D}}_{novel}^{c}$. We assume the first $k$ features are from the clean set and the remaining are from the noisy set. For convenience, we ignore the superscript $c$ if it can be inferred from the context.

Our work extends the GCN cleaning framework (Iscen et al., 2020) by directly bridging the optimization between the relevance scores and noise-tolerant hybrid prototypes generation in an end-to-end manner. Firstly, we review the relation modeling with GCN.

Iscen et al. (Iscen et al., 2020) introduced a noise cleaning framework to tackle the few clean and many noisy problem. Their approach is divided into two stages: a graph cleaning and a classifier learning stage.

First, they applied a two-layered GCN to perform offline cleaning by predicting a class relevance score $r$ for each noisy example in the $\mathcal{D}_{noisy}^{c}$. The class relevance scores $\mathbf{r}\in\mathbb{R}^{N}$ are learned independently for each novel class $c$:

where $\Theta=\{\Theta_{1},\Theta_{2}\}$ is the GCN parameters, $[\cdot]_{+}=ReLU(\cdot)$ and $\widetilde{A}$ is the normalized affinity matrix. The training process is constrained by a binary classification loss $L_{\Theta}$:

where $\lambda$ is a hyperparameter for balancing. This loss aims to classify the clean labeled images as positive examples and treat all the images with noisy labels as negative examples.

In general, a class prototype $\mathbf{p}_{c}$ for category $c$ is defined as below:

where the normalization term $r(\widetilde{\mathcal{D}}_{novel}^{c})=\sum_{i=1}^{N}{r_{i}^{c}}$. The prototype classifier $\mathbf{P}$ consists of $C$ prototypes, i.e. $\mathbf{P}=[\mathbf{p}_{1},...,\mathbf{p}_{C}]\in\mathbb{R}^{d\times C}$.

In prototypical learning, each class would produce a single prototype to serve as a representative feature for discriminative classification. In the few clean and many noisy classification, the unified prototype could be learned from the combination of a clean prototype and a noise-tolerant prototype based on Eq. 3:

where

Second, a few-shot classifier is learned over the novel classes. Particularly, they used a cosine classifier to minimize the loss function $L(\widetilde{\mathcal{D}}_{novel};\mathbf{W})$:

Herein, $\boldsymbol{\sigma}(\cdot)$ is the softmax function, $s$ is the temperature parameter. Both the feature vectors $\mathbf{v}\in\mathbb{R}^{d}$ and the classifier $\mathbf{W}\in\mathbb{R}^{d\times C}$ are transformed into $\ell_{2}$-normalized form, denoted as $\mathbf{\hat{v}}$ and $\mathbf{\hat{W}}$.

Although this approach achieved promising results over those methods that only deal with clean data, one of the drawbacks is that noisy data are often diverse and one noise-tolerant prototype cannot represent the whole set. Meanwhile, the binary classification loss introduces a discrepancy between the graph cleaning process and the subsequent prototype-based classifier learning. There is no guarantee that the learned $\mathbf{p}_{noise}$ using the class relevance score $r$ would produce a meaningful prototype for classification.

The diversity of the noisy images motivates us to treat the noisy images differently. In the noisy image set, there might be a few closely relevant images that are visually similar to the clean examples. In the meantime, it is expected that there are a large number of noisy images that are irrelevant. It can be imagined that the noisy features are scattered in the embedding space. If we use one noise-tolerant prototype to represent the noisy set, it can incur a large intra-class variance. Based on this hypothesis, we propose to divide the noisy images into multiple groups and then generate multiple noise-tolerant prototypes. We define the noise-tolerant prototypes generation procedure as $G$. Take the noisy “feature-relevance score” pairs as input, $G$ produces $T$ noise-tolerant prototypes based on some criteria,

Each noise-tolerant prototype $\mathbf{p}_{noise}^{t}$ is the representative “cluster” or “center” of the corresponding noise group. Each group also shares similar properties.

Following the above paradigm, we consider two types of noise-tolerant prototypes generation procedure $G$.

Feature based clustering. One of the methods to split the noisy images is to perform clustering on the visual features, e.g. k-means, where the noisy images could be grouped into $T$ clusters $\{\mathbf{C}_{t}\}_{t=1}^{T}$. The noise-tolerant prototype for each cluster $\mathbf{C}_{t}$ can be expressed as:

Since the noisy features are pre-computed in our setting, clustering can be done offline before the noise cleaning. One of the disadvantages is that the group is fixed and cannot be further adjusted during the graph cleaning stage.

Relevance score based separation. In this paper, we introduce a general strategy to divide noisy images into multiple groups. We group the noisy images based on the class relevance scores $\mathbf{r}$ (Eq. 1) generated by the same graph convolutional network $F_{\Theta}(\widetilde{A},V)$. For each group $t\in[1,\ldots,T]$, its relevance score window ${w}_{t}$ is denoted as:

where $r_{max}$ and $r_{min}$ denote the maximum relevance score and the minimum relevance score, respectively. In each noisy window $w_{t}$, we define the corresponding noise-tolerant prototype as the weighted average of the features of all noisy images. The noise-tolerant prototype for window $w_{t}$ (Eq. 10) is denoted as:

The generated noise-tolerant prototype $\mathbf{p}_{noise}^{t}$ can be regarded as the representative prototype for all noisy images in window $w_{t}$. Because the relevance scores are learned from the relations between the noisy images and clean examples, it takes more factors into consideration to produce better separation results. Note that the groups are dynamically changed during the training. We adopt this strategy in most of our experiments.

The introduction of multiple noise-tolerant prototypes provides a separation for different noisy examples, which enables the modeling of the complex noisy web data in a finer manner.

Given noise-tolerant hybrid prototypes consisted of $T$ diverse noise-tolerant prototypes and one clean prototype $\{\mathbf{p}_{noise}^{1},...,\mathbf{p}_{noise}^{t},...,\mathbf{p}_{noise}^{T},\mathbf{p}_{clean}\}$, our objective is to yield a compact and discriminative prototype $\mathbf{p}_{c}$ for few-shot image classification. The expected prototype is supposed to be represented as the sum of the noise-tolerant prototypes and the clean prototype. However, computed from scarce clean examples and large-scale noisy examples, it is nearly hard to get an unbiased prototype to support the class decision boundary if there are no constraints between the noise-tolerant prototypes and the clean prototype. A discriminative prototype is expected to exhibit low intra-class variance in the few clean and many noisy scenarios. Since the clean examples are more reliable than the noisy ones, it is much more reasonable to require the expected prototype biased towards the clean prototype.

We introduce a similarity maximization loss to pull the noise-tolerant prototypes to be closer to the clean prototype. The cosine distance is used to measure the similarity between two prototypes $\mathbf{p}_{1}$ and $\mathbf{p}_{2}$:

where ${\left\lVert{\cdot}\right\rVert_{2}}$ is $\ell_{2}$-norm. Our goal is to minimize the negative similarity between the two prototypes. The distance minimization enforces the learned noise-tolerant prototype to be relevant to the specific category. It can overcome the discrepancy between the relevance score generation process and the prototype-based classification stage, so as to learn a discriminative unified prototype from the clean prototype and the noise-tolerant prototypes in an end-to-end manner.

It is worth emphasizing that the noise-tolerant prototypes might hold different degrees of bias against the clean prototype. Specifically, we define the SimNoiPro loss based on Eq. 12 as below:

where $\mathbf{p}_{clean}$ is the clean prototype in Eq. 5, $\mathbf{p}_{noise}$ is the global noise-tolerant prototype in Eq. 6, $\alpha_{t}$ is a scaling factor to control the relative weight of the similarity between the clean prototype and each local noise-tolerant prototype and $\beta$ is a hyperparameter. In general, a noise-tolerant prototype that exhibits a higher degree of similarity to the clean prototype should be assigned a relatively more substantial proportion in the loss function.

Discussion. Iscen et al. (Iscen et al., 2020) used a binary classification loss during training to classify the noisy examples as negative instances and treat the clean examples as positive instances. It can assign low relevance scores for those relevant images in the noisy set. The relevance scores and the class prototype generation are separated into two stages. No constraints are imposed to maintain the quality of the prototype, which results in a suboptimal class prototype. In contrast, our SimNoiPro is easy to implement and it directly optimizes the relevance scores and the class prototype generation in an end-to-end manner, which could potentially produce better relevance scores in a more plausible way. In the experiments, we find SimNoiPro works better in the low-data regime, but comparably in the 1-shot setting. The clean prototype formed from one clean image might incur large intra-class variance, e.g. if the clean image is a photo of cucumbers growing on vines, it might be difficult to identify noisy images of cucumber slices in salad as relevant ones.

We evaluate our method on two benchmarks: Low-shot Places365 (Zhou et al., 2017) and Low-shot ImageNet (Hariharan and Girshick, 2017).

Low-shot Places365 (Zhou et al., 2017) is divided into 183 test and 182 validation classes by (Iscen et al., 2020). Note that we treat all classes in Places365 as novel categories.

Low-shot ImageNet (Hariharan and Girshick, 2017) is created from ImageNet. The total 1000 classes from the ImageNet dataset are divided into 389 base classes and 611 novel classes. For the purpose of cross-validation, this benchmark is split into two disjoint sets where the test set contains 196 base classes and 311 novel classes and the remaining are in the validation set.

Noisy data statistics. In our experiments, the above two datasets are considered as the clean sets and are both extended by large-scale noisy images from the YFCC100M dataset (Thomee et al., 2016). YFCC100M consists of around 100M images collected from the Flickr where each image is attached to a text description. Refer to (Iscen et al., 2020), noisy images are selected if their text annotations contain the name of the novel class. Towards the end, the Low-shot Places365 and the Low-shot ImageNet datasets are supplied by extra 9,720,957 and 3,744,994 noisy images, respectively.

Following the standard evaluation protocol used in the few-shot setting (Hariharan and Girshick, 2017; Iscen et al., 2020), we perform 5 trials under different $k\in\{1,2,5,10,20\}$ shot setup. For each trial, we sample $k$ clean images per class from the clean set and combine them with all noisy data as the training dataset. We report the average top-5 accuracy over 5-trials on the novel class in the test set.

In our experiments, features are extracted from ResNet-10 and ResNet-50 by (Iscen et al., 2020). The feature extractor is trained on the base class from the Low-shot ImageNet. The dimension $d$ of the input features is 512 for ResNet-10 and 256 for ResNet-50 (after PCA), respectively. For graph cleaning stage, similar to (Iscen et al., 2020), we use Adam with a weight decay of 0.0005 as our optimizer. The initial learning rate is set to 0.1 for 100 iterations and decays by 0.1 every 30 iterations. We divide $T=5$ groups for all shot setups. The hyperparameters $\alpha$ and $\beta$ are tuned on the validation set. We cross-validate the possible values of $\alpha$ and $\beta$ in the interval $[0.01,1.0]$. The step is 0.01 for $[0.01,0.1]$ and 0.1 otherwise. For classifier learning, we initialize the cosine classifier with the class prototype. The cosine classifier is trained with a batch size of 512 and optimized with Adam for 50 epochs. The learning rate starts from 0.1 and is finally reduced to 0.001 with cosine annealing (Loshchilov and Hutter, 2016). We set the temperature $s=15$.

We compare our method with several baselines. These baselines include: (1) Class proto. (Gidaris and Komodakis, 2018): the class prototype is computed as the mean of the clean feature embeddings. (2) ProtoNets (Snell et al., 2017): a meta-learning approach for few-shot classification. (3) $\beta$-weighting (Iscen et al., 2020): the relevance score is set as $\beta$. (4) Label Propagation (Iscen et al., 2020): the relevance scores are obtained by solving a linear system. (5) MLP (Iscen et al., 2020): this model learns a nonlinear mapping for assigning relevance scores. (6) Similarity (Iscen et al., 2020): the relevance score is calculated as the cosine similarity between the data point and the class prototype. (7) GJS (Englesson and Azizpour, 2021): a generalized jensen-shannon divergence loss for learning with noisy labels. (8) NCR (Iscen et al., 2022): a neighbor consistency loss for combating with noisy labels. (9) Iscen et al. (Iscen et al., 2020): a recently proposed method based on the graph convolutional network for learning with few clean and many noisy labels. Please refer to (Iscen et al., 2020) for more details.

We report the top-5 accuracy performance on Low-shot Places365 in Table 1. $\beta$-weighting takes effect when clean images are limited. It performs worse than class proto if enough clean samples are provided. GJS and NCR get better results by constraining the network to output consistent predictions for noisy labels, but still lag behind our method by 9.4% and 7.9%, respectively, in the 5-shot setting. By measuring the relations between noisy and clean data, Label Propagation and MLP improve a lot in the low-shot settings. In the 20-shot setting, the improvement upon the class prototype is marginal. We observe that our method consistently outperforms other methods, especially in the 2/5/10-shot setup. The performance gaps relative to the Iscen et al. are +1.7%, +4.3% and +1.6%, respectively. In the 5-shot setting, our SimNoiPro provides 58.60% accuracy over the Iscen et al. 54.26%. It indicates we can learn a better few-shot classifier with the relevance scores generated by our consistent learning objective.

Table 2 presents the top-5 accuracy performance of different methods on Low-shot ImageNet. First, we can find that learning with additional noisy data brings significant improvement in the few-shot setting, especially 1-shot case, where we can gain more than 20% performance improvement. Even a simple baseline Similarity can improve 4% accuracy after using noisy data. This confirms that noisy data are potentially useful to facilitate the learning of the few-shot classifier if we can mine those relevant images and measure their similarities with clean ones. Second, compared to other methods that also use additional noisy data to enhance the classifier learning, our method achieves better performance, mostly in the 5-shot setup, where our model can gain the average top-5 accuracy up to 78.83% with the backbone of ResNet-10, while the Iscen et al. method (Iscen et al., 2020) can only reach 76.07%. Our SimNoiPro directly optimizes the class relevance scores for noise-tolerant prototypes generation so that it can overcome the inconsistent optimization issues in the baseline method and produce a more discriminative prototype for classification. Note our SimNoiPro can even offer comparable performance in the 5-shot setting to the Iscen et al. (78.78%) where 10 clean images are given. This is very beneficial and efficient when deployed to the real-world scenario because it indicates that few well-annotated clean data are good enough to identify relevant images from a noisy set and improve the generalization of the few-shot learner. We also notice that our method achieves comparable results under the $k\in\{1,20\}$ shot settings. The comparable performance on the 1-shot setting might be due to the lack of a representative clean prototype. Given only one clean image, noise cleaning can be much more biased. For the 20-shot setting, the highly discriminative capacity of the prototype composed of 20 clean images might lead to limited improvement. Last, if we use a stronger backbone such as ResNet-50 in our experiment, more discriminative features are extracted, which contributes to better performance. Our approach yields an enhancement of 3.5% and 2.1% in relative improvement over the strong Iscen et al. baseline in the 5-shot and 10-shot settings, respectively. These results suggest that our SimNoiPro works better by introducing multiple noise-tolerant prototypes to model the diversity of the noisy web image set and incorporating the class prototype generation into the noise cleaning procedure to produce more plausible relevance scores. The noise-tolerant prototype can be much more compact and discriminative with lower intra-class variance. Training with the cosine classifier can boost performance.

We directly leverage the class prototype to perform the classification. The test image is classified by the nearest matching. As seen in Table 3, our method outperforms the baseline method on both Low-shot Places365 and Low-shot ImageNet. The improvement is 3.4% and 1.6%, respectively. Our SimNoiPro enables better relevance score generation, which results in more discriminative and compact prototypes.

We investigate the influence of the number of noise groups on the performance in our few-shot setting. Here, we evaluate our method under the 5-shot setting. We divide the noisy data into $T\in\{1,2,3,4,5,6\}$ groups, and each group is assigned an equal weight, i.e., $\alpha_{t}$ is set to $1.0$ for all groups. The results on Low-shot ImageNet are shown in Figure 4. We observe that the top-5 classification accuracy improves when we increase the number of noise groups but reaches saturation when $T=4$. This indicates that SimNoiPro is less sensitive to the number of noise groups when the number of noise groups is larger than 4. Compared to the case when $T=1$, more noise-tolerant prototypes bring the improvement of the performance, showing that the introduction of multiple noise-tolerant prototypes plays an important role in the modeling of the diverse noisy set.

We study the effect of the hyperparameter by setting $\alpha_{t}=0$ or $\beta=0$ individually. The results are shown in Table 4. We find that removing any of them can degrade the performance. The accuracy drops by 8.8% when $\beta=0$ and 2.7% when $\alpha_{t}=0$, respectively. Besides, we perform the ablation analysis on the setup of the sequence $\{\alpha_{t}\}$. We compare three types: (1) Decreasing $\alpha_{t}$. (2) Equal $\alpha_{t}$. (3) Increasing $\alpha_{t}$. Among them, Increasing $\alpha_{t}$ achieves the best result. It confirms our hypothesis that a noise-tolerant prototype should be assigned a relatively more substantial proportion if it exhibits a higher degree of similarity to the clean prototype, as discussed in Section 3.2.2.

In this ablation, we consider two types of noise-tolerant prototypes generation: feature based clustering and relevance score based separation. For feature based clustering, k-means is applied to cluster the pre-computed noisy features into 5 groups. We adopt similar hyperparameter configurations as the relevance score based method, and ensure that the noise-tolerant prototype closer to the clean prototype is assigned a higher weight. Table 4 shows the comparison between two ways of noise-tolerant prototypes generation on Low-shot ImageNet. Clustering based method shows worse performance than the relevance score based method. When there are more clean images, the gap tends to be smaller. As discussed in Section 3.2.1, clustering is performed offline and the resulting noise groups are fixed in the graph cleaning stage. The separation is determined by the geometry of the feature space. It cannot embrace the benefit of the learned relations by graph convolutions. This leads to performance degradation. On the contrary, the relevance score based generation exhibits the advantages of adaptively producing the customized noise groups, which are much more robust to the influence of the irrelevant images. Therefore, we apply the relevance score based noise-tolerant prototypes generation in our experiments.

If the noisy set is pretty diverse and composed of many closely relevant images, pulling the noise-tolerant prototypes closer to the less representative clean prototype might result in overfitting issues. We investigate the effect of replacing the similarity maximization by minimization. The similarity minimization loss tries to push the noise-tolerant prototypes away from the clean prototype. Therefore, the final combined prototype is prevented from being too close to the clean prototype. We validate the similarity minimization loss with the prototype classifier under different shot settings. The top-5 accuracy results on Low-shot ImageNet are presented in Table 5. We observe that the performance drops a lot for each shot setting. For 2-shot case, the similarity minimization method is 52% behind the similarity maximization method. When given more and more clean images, the gap narrows. This suggests that more irrelevant images are in the noisy set and our similarity maximum loss can well select those relevant images and assign higher relevance scores to build a more discriminative classifier. Meanwhile, our learned relevance scores can measure the clean-noisy relations better, which can help the generation of a more compact prototype.

Instead of using binary classification loss to treat the clean data as positive instances and noisy data as negative ones (Iscen et al., 2020), our SimNoiPro directly optimizes the relevance scores for the noise-tolerant prototypes generation in an end-to-end manner. We compare the distribution of the class relevance $r$ generated by our SimNoiPro and the baseline method (Iscen et al., 2020). In this experiment, we divide the interval $r_{max}-r_{min}$ into 10 groups equally and then count the number of noisy data for each group in the first trial under the 5-shot setting.

The visualization results for Low-shot Places365 and Low-shot ImageNet are illustrated in Figure 5 and 6, respectively. In Figure 5, we select three typical classes with higher accuracy obtained by our method on Low-shot Places365. It can be seen that for the baseline method (Iscen et al., 2020), most noisy data fall into the 0-10% interval. This indicates that the binary classification loss used in (Iscen et al., 2020) simply pushes the class relevance scores of the noisy data to be close to 0. There is no guarantee to produce a discriminative unified prototype for classification. However, our SimNoiPro regards each noisy data as one of the components of the noise-tolerant prototypes. As a result, the output of our model reveals the relative importance of noisy features and directly contributes to the unified prototype used in the following classification stage. Specifically, we observe that the distribution of $r$ produced by our method is quite different from the baseline in class 2. In Figure 6, more visualization results on Low-shot ImageNet are presented. We also find that the distribution of the relevance scores for the baseline model is more centralized while our method produces a more diverse distribution.

We show the representative noisy images with the corresponding class relevance scores in the different noise groups produced by our SimNoiPro. The results on Low-shot Places365 and Low-shot ImageNet are presented in Figure 7 and 8. Figure 7 depicts the results for class “wind farm” on Low-shot Places365. In Figure 8, we present the noise groups for class “afghan hound”. Here, noise groups are divided by the class relevance scores. Note that our relevance score can represent the relative importance. For each noise group, we randomly sample several representative noisy images. We notice that each group mostly shares similar visual semantics except in noise group 1 which contains many irrelevant images. Our method can assign higher scores to those that look very similar to the given clean image. These results support our basic motivation in Section 3.2.1. As the noisy set is pretty diverse, using one coarse noise-tolerant prototype can fail to represent the complex noisy data collections. Our method can well model the noise data by introducing multiple prototypes. The generated relevance scores are more plausible.