Weak Supervision with Incremental Source Accuracy Estimation

Weak supervision approaches obtain labels for unlabeled training data using noiser or higher level sources than traditional supervision [1]. These sources may be heuristic functions, off-the-shelf models, knowledge-base-lookups, etc. [2]. By combining multiple supervision sources and modeling their dependency structure we may infer the true labels based on the outputs of the supervision sources.

Varma et. al. [3] and Ratner, et. al. [4] model the joint distribution of $\lambda_{1},\dots,\lambda_{m},Y$ in the classification setting as a Markov Random Field

associated with graph $G=(V,E)$ where $\theta_{i,j}\ 1\leq i,j\leq m+1$ denote the canonical parameters associated with the supervision sources and $Y$, and $Z$ is a partition function [here $V=\{\lambda_{1},\dots,\lambda_{m}\}\cup\{Y\}$]. If $\lambda_{i}$ is not independent of $\lambda_{j}$ conditional on $Y$ and all sources $\lambda_{k},\ k\in\{1,\dots,m\}\setminus\{i,j\}$, then $(\lambda_{i},\lambda_{j})$ is an edge in $E$.

Let $\Sigma$ denote the covariance matrix of the supervision sources and $Y$. To learn $G$ from the labels

and without the ground truth labels, Varma et. al. assume that $G$ is sparse and therefore that the inverse covariance matrix $\Sigma^{-1}$ associated with $\lambda_{1},\dots,\lambda_{m},Y$ is graph-structured. Since $Y$ is a latent variable the full covariance matrix $\Sigma$ is unobserved. We may write the covariance matrix in block-matrix form as follows:

Inverting $\Sigma$, we write

$\Sigma_{O}$ may be estimated empirically:

where $\Lambda=[\mathbf{\lambda}_{1}\mathbf{\lambda}_{2},\dots,\mathbf{\lambda}_{n}]$ denotes the $m\times n$ matrix of labels generates by the sources and $\nu=\hat{E}[O]\in\mathbb{R}^{m}$ denotes the observed labeling rates.

Using the block-matrix inversion formula, Varma et. al. show that

where $c=(\Sigma_{S}-\Sigma_{OS}^{T}\Sigma_{O}^{-1}\Sigma_{OS})^{-1}\in\mathbb{R}^{+}$. Letting $z=\sqrt{c}\Sigma_{O}^{-1}\Sigma_{OS}$, they write

where $K_{O}$ is sparse and $zz^{T}$ is low-rank positive semi definite. Because $\Sigma_{O}^{-1}$ is the sum of a sparse matrix and a low-rank matrix we may use Robust Principal Components Analysis [5] to solve the following:

Varma et. al. then show that we may learn the structure of $G$ from $K_{O}$ and we may learn the accuracies of the sources from $z$ using the following algorithm:

Note that $\hat{L}=zz^{T}$.

Ratner, et. al. [4] show that we may estimate the source accuracies $\hat{\mu}$ from $z$ and they propose a simpler algorithm for estimating $z$ if the graph structure is already known: If $E$ is already known we may construct a dependency mask $\Omega=\{(i,j):(\lambda_{i},\lambda_{j})\not\in E\}$. They use this in the following algorithm:

Snorkel, an open-source Python package, provides an implementation of algorithm 2 [6].

Although the algorithm proposed by Varma et. al. may be used determine the source dependency structure and source accuracy, it requires a robust principal components decomposition of the matrix $\hat{\Sigma}_{O}$ which is equivalent to a convex Principal Components Pursuit (PCP) problem [5]. Using the current state-of-the-art solvers such problems have time complexity $O(\epsilon^{-2})$ where $\epsilon$ denotes the solver convergence tolerance [5]. For reasonable choices of $\epsilon$ this may be a very expensive calculation.

In the single-task classification setting, algorithm 2 may be solved by least-squares and is therefore much less expensive to compute than algorithm 1. Both algorithms, however, require the observed labeling rates and covariance estimates of the supervision sources over the entire dataset and therefore cannot be used in an on-line setting.

We therefore develop an on-line approach which estimates the structure of $G$ using algorithm 1 on an initial ”minibatch” of unlabeled examples and then iteratively updates the source accuracy estimate $\hat{\mu}$ using using a modified implementation of algorithm 2.

Given an initial batch $b_{1}$ of unlabeled examples $X_{b_{1}}=\{x_{1},\dots,x_{k}\}$ we estimate $G$ by first soliciting labels $\mathbf{\lambda}_{1},\dots,\mathbf{\lambda}_{k}$ for $X_{b_{1}}$ from the sources. We then calculate estimated labeling rates $\hat{E}[O]$ and covariances $\hat{\Sigma}_{Ob_{1}}$ which we then input to algorithm 1, yielding $\hat{G}=(V,\hat{E})$ and $\hat{L}$. From $\hat{E}$ we create the dependency mask $\hat{\Omega}=\{(i,j):(\lambda_{1},\lambda_{j})\not\in\hat{E}\}$ which we will use with future data batches. Using the fact that $\hat{L}=zz^{T}$ we recover $\hat{z}$ by first calculating

We then break the symmetry using the method in [4]. Note that if a source $\lambda_{i}$ is conditionally independent of the others then the sign of $z_{i}$ determines the sign of all other elements of $z$.

Using $\hat{z},\ \hat{E}[O],\ \hat{\Sigma}_{Ob_{1}}$, class balance prior $\hat{E}[Y]$ and class variance prior $\hat{\Sigma}_{S}$ we calculate $\hat{\mu}$, an estimate of the source accuracies [if we have no prior beliefs about the class distribution then we simply substitute uninformative priors for $\hat{E}[O]$ and $\hat{\Sigma}_{Ob_{1}}$].

For each following batch $b_{p}$ of unlabeled examples $X_{b_{p}}$ we estimate $\Sigma_{Ob_{p}}$ and $E[O]_{b_{p}}$. Using these along with $\hat{E}[O]$ and $\hat{\Sigma}_{Ob_{1}}$ we calculate $\hat{\mu}_{b_{p}}$, an estimate of the source accuracies over the batch. We then update $\hat{\mu}$ using the following update rule:

where $\alpha\in[0,1]$ denotes the mixing parameter. Our method thus models the source accuracies using an exponentially-weighted moving average of the estimated per-batch source accuracies.

Using the estimated source accuracies and dependency structure we may estimate $p(y,\mathbf{\lambda})$ which we may then use to estimate $p(y|\mathbf{\lambda})$ by (1).

Our tests demonstrate the following:

1. 1.

   Our model generates labels which are more accurate than those generated by the baseline [when averaged over all 5 tests].

2. 2.

   Both our method and the baseline generate labels which are more accurate than those generated by each of the supervision sources.

3. 3.

   Our tests of the accuracy of labels generated by our method using different values of $\alpha$ yields an optimal values $\alpha=0.05$ and shows convexity over the values tested.

Theses tests show that the average accuracy of the incremental model qualitatively appears to increase as the number of samples seen grows. This result is not surprising as we would expect our source accuracy estimate approaches the true accuracy $\hat{\mu}\xrightarrow{}{\mu}$ as the number of examples seen increases. This implies that the incremental approach we propose generates more accurate labels as a function of the number of examples seen, unlike the supervision sources which are pre-trained and therefore do not generate more accurate labels as the number of labeled examples grows.

These tests also suggest that an optimal value for $\alpha$ for this problem is approximately $0.05$ which is in the interior of the set of values tested for $\alpha$. Since we used $100$ minibatches in each test of the incremental model this implies that choosing an $\alpha$ which places greater weight on more recent examples yields better performance, although more tests are necessary to make any stronger claims.

Finally, we note that none of the models here tested are in themselves highly-accurate as classification models. This is not unexpected as the supervision sources were intentionally chosen to be ”off-the-shelf” models and no feature engineering was performed on the underlying text data, neither for the datasets used in pre-training the two classifier supervision sources nor for the test set [besides Tf-idf vectorization]. The intention in this test was to compare the relative accuracies of the two generative methods, not to design an accurate discriminative model.

We develop an incremental approach for estimating weak supervision source accuracies. We show that our method generates labels for unlabeled data which are more accurate than those generated by pre-existing non-incremental approaches. We frame our specific test case in which we use pre-trained models and heuristic functions as supervision sources as a transfer learning problem and we show that our method generates labels which are more accurate than those generated by the supervision sources themselves.