Uncertainty Herding: One Active Learning Method for All Label Budgets

We first define a measure of how much uncertainty a set of data points covers.

Here, the uncertainty function $U$ is based on a model $f$ which is updated as the model trains, while $k_{\sigma}$ uses a fixed feature extractor $g$. UCoverage weights the GCoverage of (1) with a choice of uncertainty measure $U(\mathbf{x};f)$; choosing $U(\mathbf{x};f)=1$ immediately recovers GCoverage.

In Figure 1(a), we visualize differences in assessing coverage. ProbCover (left) treats all data points within a $\sigma$-radius ball around selected points equally, assigning uniform weight to each. GCover (middle) introduces a smooth proximity measure by applying a kernel function, such as the RBF kernel, to weigh nearby points more effectively. UCover (right) additionally incorporates uncertainty. The green circles represent uncertainty, with their size proportional to each point’s uncertainty. Ultimately, UCover evaluates how well the selected data points account for uncertainty across the space.

We now show $\widehat{\mathrm{UC}}_{k_{\sigma}}$ is a good estimator of $\mathrm{UC}_{k_{\sigma}}$. All proofs are in Appendix A. {restatable}theoremuniconv Let $U(\mathbf{x};f)\in[0,U_{\max}]$, $k_{\sigma}(\mathbf{x},\mathbf{x}^{\prime};g)=\tilde{k}_{\sigma}(g(\mathbf{x}),g(\mathbf{x}^{\prime}))\in[0,1]$, $\{g(\mathbf{x}):\mathbf{x}\in\mathcal{U}\}\subseteq\{\mathbf{t}\in\mathbb{R}^{d}:\left\lVert\mathbf{t}\right\rVert\leq R\}$, and $\left\lvert\tilde{k}_{\sigma}(\mathbf{t},\mathbf{t}^{\prime})-\tilde{k}_{\sigma}(\mathbf{t},\mathbf{t}^{\prime\prime})\right\rvert\leq L_{\sigma}\left\lVert\mathbf{t}^{\prime}-\mathbf{t}^{\prime\prime}\right\rVert$. Let $\mathcal{L}\subseteq\mathcal{X}$ be arbitrary and fixed. Assume $B/N<16R^{2}$. Then, with probability at least $1-\delta$ over the choice of the $N$ iid data points in $\mathcal{U}\subseteq\mathcal{X}$ used to estimate $\widehat{\mathrm{UC}}_{k_{\sigma}}$, all size-$B$ sets $\mathcal{S}$ (not only subsets of $\,\mathcal{U}$) have low error:

Since typically $B\ll N$ – we query up to perhaps a few hundred points at a time, out of a dataset of at least tens of thousands but perhaps millions – even optimizing our noisy estimate of the uncertainty coverage does not introduce substantial error.⁴⁴4It is worth emphasizing that, although Bae et al. (2024) mentioned a simple Hoeffding bound for $\widehat{\mathrm{C}}_{k_{\sigma}}$, this bound only applies to fixed $\mathcal{S}$ independent of $\mathcal{U}$ – while in reality $\mathcal{S}\subseteq\mathcal{U}$. Ignoring this problem and taking a union bound over all $\binom{N}{B}$ subsets of $\mathcal{U}$ would yield a uniform convergence bound of $U_{\max}\sqrt{\frac{1}{2N}\log\left(\binom{N}{B}\frac{2}{\delta}\right)}\leq U_{\max}\sqrt{\frac{B}{2N}\left[\log\left(e\frac{N}{B}\right)+\frac{1}{B}\log\frac{2}{\delta}\right]}$. The rate of Definition 1 is very similar, with the advantage of being correct. If $U$ is given by the margin between probabilistic predictions, then $U_{\max}\leq 1$; other notions can also be easily bounded. Assuming $k_{\sigma}\leq 1$ is for convenience, but a different upper bound can simply be absorbed into $U_{\max}$. The bound is indeed 1 for the Gaussian kernel on $g$ used by Bae et al. (2024) (as well as by us); this kernel also has $L_{\sigma}=\sqrt{\frac{2}{e}}\cdot\frac{1}{\sigma}$. The kernel which corresponds to the probability coverage of Yehuda et al. (2022), however, is not Lipschitz, suggesting why it is so sensitive to $\sigma$. Self-supervised representations in $g$ are often normalized to $R=1$, and usually have $d$ at most a few hundred.

We wish to choose parameters of UCoverage such that it smoothly changes from behaving like generalized coverage in low-budget regimes to behaving like uncertainty in high-budget regimes.

To obtain an actual active learning method, we still need an algorithm to maximize $\widehat{\mathrm{UC}}_{k_{\sigma}}$. We could select a batch by finding $\bar{\mathcal{S}}\in\operatorname*{arg\,max}_{\mathcal{S}\subseteq\mathcal{U},\lvert\mathcal{S}\rvert=B}\widehat{\mathrm{UC}}_{k_{\sigma}}(\mathcal{L}\cup\mathcal{S})$. This is equivalent to the weighted kernel $k$-medoids objective, with weights determined by uncertainty $U(\mathbf{x};f)$; thus, we could use Partitioning Around Medoids (PAM) (Kaufman & Rousseeuw, 2009) to try to find $\bar{\mathcal{S}}$.

UHerding improves uncertainty measures by accounting for both the uncertainty of a selected point and its influence on reducing nearby uncertainty. It improves on MaxHerding by incorporating uncertainty, putting less weight on covering already-certain points.

corollarycoveroptim In the setting of Definition 1, let $\hat{\mathcal{S}}\subseteq\mathcal{U}$ be the result of UHerding for $B$ steps to add to $\mathcal{L}$, and $\mathrm{UC}^{*}=\max_{{\mathcal{S}\subseteq\mathcal{U},\left\lvert\mathcal{S}\right\rvert=B}}\mathrm{UC}_{k_{\sigma}}(\mathcal{L}\cup\mathcal{S})$ the optimal coverage obtainable among $\mathcal{U}$. Then

Figure 2 visually compares the next selected data points (in red) by Margin, MaxHerding, and UHerding (using Margin uncertainty) on a two-class half-moon dataset represented by $\mathbin{\vbox{\hbox{\scalebox{1.7}{$\circ$}}}}$ and $\mathbin{\vbox{\hbox{\scalebox{1.0}{$\bm{\vartriangle}$}}}}$. A logistic regression model with fifth-order polynomial features is trained at each iteration, with the decision boundary after training on 12 previously selected points (in green) shown as dashed lines. As expected, Margin selects points near the decision boundary, which can lead to suboptimal models if the predicted boundary deviates from the true one. MaxHerding selects the most representative points based on prior selections but ignores model predictions; this gives quick generalization with few labeled points, but performance saturates over time as it neglects points near the boundary. UHerding balances these approaches by initially selecting representative points and gradually focusing on uncertain points near the boundary, improving performance over time.

UHerding is closely connected to existing hybrid active learning methods. As mentioned in Section 2, we replace $k$-means and $k$-means++ in various algorithms with greedy kernel $k$-medoids (MaxHerding) to simplify our arguments; this does not harm their effectiveness. We also apply a self-supervised feature extractor $g$ instead of a feature extractor from a classifier $f$, since feature embeddings from $g$ are more informative when $f$ is trained on a small labeled set. Finally, we often assume that $k_{\sigma}$ is in fact a positive-definite (RKHS) kernel; this is true for the kernels we use.

[Weighted $k$-means of Zhdanov 2019]propositionwkmeans Define an uncertainty measure $U(\mathbf{x};f)$ from another uncertainty measure $U(\mathbf{x};f)$ as $U(\mathbf{x};f)\vcentcolon=U^{\prime}(\mathbf{x};f)\cdot\mathbbm{1}[U^{\prime}(\tilde{\mathbf{x}};f)\geq\nu]$, where $\nu\geq 0$ satisfies $\sum_{n=1}^{N}\mathbbm{1}[U^{\prime}(\mathbf{x}_{n};f)\geq\nu]=M$, a pre-defined number. Then weighted $k$-means with uncertainty $U^{\prime}$, changed to use greedy kernel $k$-medoids, is UHerding with uncertainty $U$ and the same kernel.

As we shall see in Section 4, UHerding significantly outperforms weighted $k$-means, indicating that it is crucial to (a) convert $k$-means into MaxHerding and (b) apply parameter adaptation.

[ALFA-Mix of Parvaneh et al. 2022]propositionalfamix Let $\hat{y}(\cdot;f)$ be the predicted label of an input under $f$. Define an uncertainty measure

where $\bar{g}^{j}$ is the mean of feature representations belonging to class $j$ and $\alpha_{j}(\mathbf{x})\in[0,1)$ is the same parameter as determined by ALFA-Mix. Then ALFA-Mix, with clustering replaced by greedy kernel $k$-medoids, is UHerding with uncertainty $U$ and the same kernel.

The equivalence of weighted $k$-means and ALFA-Mix to UHerding with the right choice of uncertainty measure implies that Sections 3.2 and 3.2 also apply to them. There is a weaker connection to BADGE; UHerding and BADGE are not equivalent with any choice of uncertainty measure. However, a variant of BADGE with greedy kernel $k$-medoids uses the kernel,

where $q(\mathbf{x})=\hat{y}(\mathbf{x};f)-\hat{p}(\mathbf{x};f)$ with $f$ being a classifier. This does satisfy the following statement, showing properties similar to Sections 3.2 and 3.2 hold. {restatable}[BADGE, Ash et al., 2020]propositionbadge

If $\forall\mathbf{x}\in\mathcal{U},\,\hat{p}(\mathbf{x};f)\rightarrow\frac{1}{K}\vec{1}$, then this BADGE approaches a slightly modified MaxHerding: $\left(\mathbbm{1}\left[\hat{y}(\mathbf{x}_{n};f)=\hat{y}(\mathbf{x}^{\prime};f)\right]-\frac{1}{K}\right)k_{\sigma}(\mathbf{x}_{n},\mathbf{x}^{\prime};g)$ instead of $k_{\sigma}(\mathbf{x}_{n},\mathbf{x}^{\prime};g)$. If $\sigma\rightarrow 0$, it approaches to the uncertainty-based method where uncertainty is defined as $U^{\prime\prime}(\tilde{\mathbf{x}})\vcentcolon=\min_{\mathbf{x}^{\prime}\in\mathcal{L}\cup\{\tilde{\mathbf{x}}\}}\left\lVert\hat{y}(\mathbf{x}^{\prime};f)-\hat{p}(\mathbf{x}^{\prime};f)\right\rVert_{2}^{2}$. With our parameter adaptation heuristics, hybrid methods can also smoothly interpolate between MaxHerding and uncertainty; Figure 6(b) shows this improves BADGE. Although selecting data points maximizing $U^{\prime\prime}(\tilde{\mathbf{x}})$ is counter-intuitive, lowering $\sigma$ still helps, as it stays away from zero.

We first aim to verify that UHerding effectively interpolates between MaxHerding and uncertainty measures across budget regimes. We train a randomly-initialized ResNet18 (He et al., 2016) on CIFAR10 using 5 random seeds, gradually increasing the size of the labeled set, as shown in Figure 3. We consider UHerding based on Margin (Mar.), Entropy (Ent.), and Confidence (Conf.). The y-axis of Figure 3 represents $\Delta$Acc, indicating the performance difference from Random selection.

UHerding performs slightly better or comparably to MaxHerding in the low-budget regime, with a growing performance gap in the mid-budget regime, and substantial improvements in the high-budget regime. The opposite is true with uncertainty measures: UHerding is substantially better with low budgets, while they eventually catch up and tie UHerding with high budgets.

The choice of uncertainty measure has little impact on performance. We thus only use Margin as the UHerding uncertainty measure in the future unless otherwise specified.

We now compare UHerding with state-of-the-art active learning methods, particularly hybrids, to assess their robustness across different budget regimes. Figure 4 compares CIFAR100 and TinyImagenet, using 3 runs of a randomly initialized ResNet18; CIFAR10 results are in Appendix B.

Overall trends are similar to before: representation-based methods are good with low budgets but lose as the budget increases; uncertainty-based and hybrid methods are largely the opposite. UHerding with Margin uncertainty (UHerding Mar.) wins convincingly: no competitor ever outperforms UHerding, and for each competitor there is some budget where UHerding wins substantially.

Fine-tuning foundation models to new tasks or datasets is of increasing importance. Inspired by ActiveFT (Xie et al., 2023), we compare UHerding to leading approaches for active transfer learning.

We use DeiT (Touvron et al., 2021) pre-trained on ImageNet (Deng et al., 2009), following Parvaneh et al. (2022); Xie et al. (2023). We fine-tune the entire model, using DeiT Small for CIFAR-100 and DeiT Base for DomainNet. Figure 5 compares UHerding with various active learning methods, including ActiveFT, which consistently underperforms the other methods likely due to its design being optimized for single-iteration data selection. On CIFAR100, UHerding is comparable to representation-based methods in the low-budget regime but surpasses them by about $2\%$ in the high-budget; it substantially outperforms uncertainty and hybrid methods in the low-budget regime and ties or wins with high-budgets. On DomainNet, UHerding outperforms MaxHerding by $1.5$–$2\%$ even in the low-budget. Compared to the best-performing hybrid method ALFA-Mix, UHerding wins by up to $13\%$ with low budgets and performs similarly with high budgets.

It is also common to fine-tune only the last few layers, especially in meta-learning (Wang et al., 2019; Chen et al., 2019; Goldblum et al., 2020) and self-supervised learning (Chen et al., 2020; Caron et al., 2021). Similarly to Bae et al. (2024), we use DINO (Caron et al., 2021) features fixed through fine-tuning; we train a head of three fully connected ReLU layers on ImageNet. Figure 6(a) shows similar results, with UHerding consistently outperforming other methods across various budget regimes.

Table 1 summarizes the results of Sections 4.2, 4.3 and B, reporting the mean improvement/degradation over Random for each budget regime.⁵⁵5ActiveFT is reported only for DomainNet (Dom.) and ImageNet (ImN.), as it is designed for fine-tuning. UHerding wins across all budget regimes, while other methods have a significant range of budgets where they are worse than Random.

Figure 6(b) gradually modifies BADGE to be similar to UHerding with Margin uncertainty on CIFAR-100, replacing $k$-means++ with MaxHerding, then adding parameter adaption. Each step improves; so does the final step to UHerding, which changes the choice of uncertainty.

In Appendix C, we analyze the contributions of each parameter adaptation component in UHerding. The baseline with fixed parameters performs slightly worse than MaxHerding at low- and mid-budgets, and worse than Margin at high-budgets. Adding temperature scaling results in notable improvements across all budget regimes, with further gains from incorporating radius adaptation.