Streaming Active Learning with Deep Neural Networks

Active learning considers a supervised learning situation where unlabeled data are abundant, but acquiring labels is expensive (Settles, 2010; Dasgupta, 2011). One example of this might be classifying underlying disorders from histological images, where obtaining labels involves querying medical experts. Another might be predicting drug efficacy, where labels corresponding to candidate molecules could require clinical trials or intensive computational experiments. In these settings, we typically want to carefully consider what samples to request labels for, and to obtain labels for data that are maximally useful for progressing the performance of the model.

Active learning is a classic problem in machine learning, with traditional approaches typically considering the convex and well-specified regime (Settles, 2010; Dasgupta, 2011; Hanneke, 2014a). Much recent interest in active learning has turned to the neural network case, which requires some special considerations. One such consideration is the expense associated with fitting these neural architectures — when used in conjunction with a sequentially growing training set, as one has in active learning, the model cannot be initialized from the previous round of optimization without damaging generalization performance. Instead, practitioners typically re-initialize model parameters each time new data are acquired and train the model from scratch (Ash & Adams, 2020). This structure has repositioned active learning to focus on the batch domain, where we are interested in simultaneously labeling a batch of $k$ samples to be integrated into the training set. The model is typically retrained only after the entire batch has been labeled.

In the convex case, where a model can easily be updated to accommodate for a single sample, active learning algorithms have tended to focus on uncertainty or sensitivity. That is, a label for a given sample should be requested if the model is highly uncertain about its corresponding label, or if incorporating this sample into the training set will greatly reduce the set of plausible model weights. In contrast, a high-performing, batch-mode active learning algorithm must also consider diversity. If two samples are relatively similar to each other, it is inefficient to include them both in the batch, regardless of the model’s uncertainty about their labels; having only one such sample labeled and integrated into the current hypothesis may be enough to resolve the model’s uncertainty on the other.

Popular approaches for batch active learning rely on samplers that require all unlabeled data to be simultaneously available. This reliance poses several major concerns for the deployment of these algorithms. For one, the run time of these methods is conditioned on the number of unlabeled samples in a way that makes them unusable for extremely large datasets. To exacerbate the issue, it is unclear how to deploy these algorithms on modern databases, where samples might be stored in a fractured manner and cannot easily be made available in their entirety.

It is especially unclear how to perform active learning in a streaming setting, where data are not all simultaneously available, and we do not know how many samples will be encountered. Here we might instead prefer to specify an acceptable labeling rate rather than a fixed acceptable batch size. In this streaming setup, it is further desirable to commit to a decision about whether to include an unlabeled sample in the batch as soon as it is encountered, rather than only after the stream has terminated. As a concrete example, consider an HCI application where a user interacts with the world while wearing an assistive or diagnostic device (Bohus et al., 2022; Singh et al., 2016). The software on the device might involve a classifier that detects objects being interacted with or the activity being performed by the user. Requesting a label to update the model to better classify these phenomena can only be done in the moment; it would be cumbersome to ask the user to provide a label corresponding to an event that occurred far in the past. How can we efficiently identify samples from a data stream for neural networks while respecting a maximum query rate?

We propose a simple active learning algorithm, Volume Sampling for Streaming Active Learning (VeSSAL)¹¹1Code for the implementation of VeSSAL can be found at https://github.com/asaran/VeSSAL.git, that addresses the concerns mentioned above. VeSSAL is made to accommodate the streaming setting, and as such it only needs to see each unlabeled point once in order to arrive at a decision about whether it should be labeled. This makes VeSSAL attractive even for fixed datasets that might be extremely large or fractured, as these are often interacted with using streaming, distributed database frameworks. VeSSAL is a natural choice for “committal” situations, when labeling decisions need to be made on the fly. On non-sequential datasets, where more conventional active learning algorithms could be exercised, VeSSAL can be significantly faster, especially for large batch sizes.

Despite its simplicity and flexibility, VeSSAL is surprisingly high performing. We show that VeSSAL produces models with predictive capabilities on par with state-of-the-art approaches, even though they are not restricted to this streaming, committal setting. We further demonstrate this to be the case in adversarial situations, where VeSSAL is presented with data that have been sorted to induce domain drift. VeSSAL is hyperparameter free, making it a powerful candidate for a wide range of active learning scenarios.

The paper proceeds as follows. We overview related work in Section 2. In Section 3, we present the mathematical formulation for the streaming active learning setting, along with details of our proposed algorithm. We give empirical support for our proposed approach in Section 4 via experiments on three benchmark datasets and one real-world dataset, in conjunction with two neural network architectures and three different batch sizes. We conclude with a discussion in Section 5.

This section situates VeSSAL with respect to prior work on streaming active learning (Sec. 2.1) as well as batch active learning strategies for training neural networks (Sec. 2.2).

Neural active learning algorithms that incorporate diversity can largely be thought of as making two design decisions. The first decision is how unlabeled candidate samples should be represented. Common choices include using the penultimate layer representation of the current state of the network (Sener & Savarese, 2018) or using a hypothetical gradient that might be induced by a given sample (Ash et al., 2020). In either case, once data are in this space, the second decision is regarding how unlabeled points should be selected in order to encourage batch diversity.

In VeSSAL, these decisions are made to produce a high-performing active learner that is amenable to the streaming, committal setting. Specifically, we assume that each candidate $x_{t}$ is seen only once, and that we must make a decision about whether or not to include it in the batch $B$ as soon as it is encountered. Once the labeling budget $k$ is allocated, we retrain the model and repeat the process.

VeSSAL performs approximate volume sampling over unlabeled candidate points in a gradient space computed with respect to the last layer of the neural network. For a neural network $f$ with parameters $\theta$, last-layer parameters $\theta_{L}\in\theta$ and cross-entropy loss function $\ell$, the gradient representation for a sample $x_{t}$ is

where $\hat{y}$ denotes the most likely label according to the current state of the model, i.e. $\hat{y_{t}}=\operatorname*{arg\,max}f(x_{t};\theta)$.

A typical way of doing volume sampling is to select a batch of points with probability proportional to the determinant of their gram or covariance matrix (Kulesza et al., 2012). In the latter case, this would mean the probability mass on a batch of samples $B$ is proportional to

where $|B|=k$, the pre-specified labeling budget.

There are several reasons to favor the covariance matrix version over the gram matrix. From a theoretical point of view, when used in an outer product, $g(x)$ could be thought of as a rank-1 approximation of the Fisher information matrix, $I(x;\theta):=\mathbb{E}_{y\sim p_{\theta}(\cdot\mid x,\theta)}\nabla^{2}\ell(x,y;\theta)$ (Ash et al., 2021). As such, this construction is reminiscent of a classic goal in active learning, which is to maximize the determinant for the Fisher (MacKay, 1992). This objective is attractive because, in the realizable setting, it selects samples that maximize the information gained by model parameters after labeling.

From a more practical point of view, the covariance matrix will stay fixed in dimensionality even as the batch size $k$ is changed. In the gram matrix alternative, which is suggested in Ash et al. (2020), the size of the matrix grows with $k$, potentially becoming intractable for larger batch sizes.

A process that samples from a distribution characterized by a determinant like this is referred to as a determinantal point process (DPP). Sampling from a DPP is usually done via Markov Chain Monte Carlo, and making this procedure efficient is an active area of research with wide-ranging statistical applications (Bardenet et al., 2017). Still, the mixing times associated with these algorithms generally makes them too inefficient to be used in conjunction with modern active learning algorithms. For example, BADGE (Ash et al., 2020) suggests using kmeans++ as a surrogate for DPP sampling, and Coreset (Sener & Savarese, 2018) uses a furthest-first traversal approach (though not in gradient space).

These sampling approaches are workable surrogates for true volume sampling, but they require all data to be simultaneously accessible. Our algorithm demonstrates that this is not necessary and that near state-of-the-art performance can be obtained by a sampler that (1) sees samples in a streaming fashion, such that each point is only observed once and that (2) commits to a labeling decision as soon as a sample is encountered.

VeSSAL choses a sample $x_{t}$ for labeling with probability $p_{t}$ proportional to the determinantal contribution of its gradient when considering other items that have already been chosen,

Using the matrix determinant lemma (Greub, 2012; Strang, 2006), the expression for $p_{t}$ reduces to

Here $\hat{\Sigma}_{t}$ is the covariance over samples that have already been selected for inclusion in the batch, $\sum_{x\in B}g(x)g(x)^{\top}$.

To compute a $p_{t}$ in practice, $g(x_{t})^{\top}\hat{\Sigma}_{t}^{-1}g(x_{t})$ must be scaled by some value $z_{t}$, which reflects the labeling budget available to the algorithm. Because the amount of data in the stream might not be known, we consider tuning $z_{t}$ to reflect a desired labeling frequency $q$. In HCI applications (Bohus et al., 2022; Singh et al., 2016; Wang et al., 2021), for example, we might not know how long a user will interact with a device (the total number of candidate samples), but we might instead have some sense of an acceptable frequency with which labels can be queried. If we instead do know the total number of samples, then we could consider $q$ to be the ratio between the labeling budget and the size of the candidate set. Specifically, we desire to find some scalar $z_{t}$ such that

How should this $z_{t}$ be chosen? Because the statistics of gradient representations $g(x)$ vary with the state of $f$, one fixed value is unlikely to work well across model architectures, datasets, batch sizes, and rounds of data selection. Instead, we aim to find an adaptive strategy, such that we both obtain the desired sampling frequency $q$ and that we do so in a way that does not allocate probability mass disproportionately across temporal regions of the stream.

One option for adaptively adjusting $z_{t}$ as selection progresses might be a multiplicative weights approach, where we select a scaling parameter from a distribution over a number of $z_{t}$ values, and constantly update this distribution to reflect whatever choice is giving us the best rate. Another option is to use gradient descent, making $z_{t}$ larger or smaller at each step in the service of minimizing error between the current and target sampling rate $q$. Unfortunately, these approaches are unlikely to work well, because the underlying distribution given by the determinantal contribution changes with each selected point. Specifically, every time a sample is added to the batch, and $\hat{\Sigma}_{t}$ is updated, the distribution of responses $g(x_{t})^{\top}\hat{\Sigma}_{t}^{-1}g(x_{t})$ can change drastically. This domain shift precludes adaptive solutions from efficiently finding a suitable $z_{t}$ like those mentioned above, because they assume the underlying distribution is stationary.

To circumvent this, we simply rewrite the expectation to disentangle $\hat{\Sigma}^{-1}_{t}$ from statistics relating to $g(x)$:

Here, our ability to find a suitable $z_{t}$ relies only on our ability to estimate the covariance of $g(x)$, and is not affected by the frequently changing $\hat{\Sigma}_{t}^{-1}$. If we approximate $\mathbb{E}_{x}\Big{[}g(x)g(x)^{\top}\Big{]}$ as $\frac{1}{t}\sum_{i=1}^{t}g(x_{i})g(x_{i})^{\top}$, this immediately suggests a way to compute $p_{t}$ that is amenable to the streaming setting:

Empirically, we find that this auto-tuning of the probability mass on each sample to be far more effective than using a fixed value for $z_{t}$. In Figure 1, we demonstrate that this approach not only consistently matches the desired labeling frequency $q$, but it also distributes our labeling budget equitably across the data stream. This is evident from the identity line between proportion of data seen and budget consumed. In contrast, fixed values of $z_{t}$ can drastically oversample or undersample, by a degree that varies with each round of active learning. Further, because of the nature of the determinant, these fixed-$z_{t}$ versions often sample far more aggressively in the beginning of the stream than the end.

The complete VeSSAL approach is presented as Algorithm 1. In it, the estimated covariance over all samples is denoted as $A$, which is initialized to all zeros. We increment $\hat{\Sigma}$ efficiently using a Woodbury update on each chosen sample (Woodbury, 1950).

One interesting note is that our estimate for $z_{t}$ is only as good as our estimate of $\mathbb{E}_{x}\big{[}g(x)g(x)^{\top}\big{]}$, which we obtain as a simple average of outer products of the $g(x_{t})$ vectors observed in the stream. A consequence of this is that the estimate will be biased if data are ordered in the stream in a non-I.I.D. fashion. In the following section, we empirically demonstrate that this appears to not be an issue — VeSSAL performs on-par with state-of-the-art, non-streaming algorithms regardless of how data are ordered.

We evaluate the performance of VeSSAL against several baselines on three academic benchmark datasets and one real-world dataset. In addition to measuring performance for a variety of architectures and batch sizes, we evaluate our approach in terms of robustness to feature drift in the data stream, and in terms of its fidelity to the predefined query rate.

Baselines. We compare our method against the following set of baselines. Most of these are non-streaming, meaning they have access to all unlabeled data when selecting samples to query. We also introduce two streaming baselines.

• •

  BADGE: A recent, hyperparameter-free approach that incorporates both uncertainty and diversity in sampling using k-means++ in the hallucinated gradient space (Eq. 1) (Ash et al., 2020) (non-streaming).

• •

  rand: A naive random sampling baseline (non-streaming).

• •

  conf: An uncertainty-based method that selects samples with smallest probability $p_{\hat{y}}$ of the top predicted class: $p_{\hat{y}}=\max f(x,\theta)$ (Wang & Shang, 2014) (non-streaming).

• •

  coreset: A diversity-based method that uses a greedy approximation to the k-center problem on representations from the model’s penultimate layer (Sener & Savarese, 2018) (non-streaming).

• •

  VeSSAL-pen: This baseline is similar to VeSSAL but uses the penultimate layer embeddings instead of hallucinated gradients of the last layer, making it a purely diversity-based hyperparameter-free approach (streaming).

• •

  stream-uniform: A naive baseline for the streaming setting where data points are sampled at a fixed frequency as they arrive (streaming).

At each round of active learning, streaming algorithms are only permitted to see each unlabeled example once, at whatever time it is presented. Further, they must commit to a labeling decision as soon as a sample is encountered, and are unable to refine their decisions as more data arrive. This puts the streaming approaches at a marked disadvantage in comparison to their non-streaming peers.

We presented VeSSAL, a new approach for batch active learning with deep neural networks in a streaming setting. Unlike prior pool-based active learning approaches for deep neural networks, our method can commit to queries as soon as samples are made available to the model from a data stream. Even in fixed-data settings, VeSSAL performs roughly as well as state-of-the-art methods — despite the fact that they are not hindered by streaming constraints. We envision several potential benefits of the proposed approach, expanding the applicability of neural networks for real-world, interaction-centric applications. Our algorithm can be run in settings that are inherently streaming, committal, or on datasets too large to be entirely stored in one place.

Our work also opens up exciting directions for future research. Currently, VeSSAL assumes that the number of label classes is known a priori. In real-world applications, where an interaction environment continues to evolve, the number of label classes may grow over time. Moreover, some queries can be costlier than others. For example, asking a user for an object label in their peripheral vision could distract them from performing the current task. Addressing these challenges are exciting topics for further work.

The authors would like to thank Sean Andrist, Dan Bohus, and the Platform for Situated Intelligence (PSI) team at Microsoft Research Redmond for providing inspiration on the problem statement, feedback on our approach, and access to the CLOW dataset.