Semi-Supervised Class Discovery

There is a recent wave in approaches for self-supervised and semi-supervised learning (Berthelot et al., 2019), which are relevant because our method is initialized in a semi-supervised fashion before it continues in a self-supervised fashion. Our method, however, differs dramatically from regular self-supervised learning where all classes are present in the training dataset and the model merely learns to slot unlabeled datapoints into those pre-existing classes.

In regular semi-supervised learning, the data set $X=(x_{i})_{i}\in[n]$ can be divided into two parts: the points $X_{l}:=(x_{1},...,x_{l})$ for which labels $Y_{l}:=(y_{1},...,y_{l})$ are provided, and the points $X_{u}:=(x_{l+1}...,x_{l+u})$ that are in the same domain as $X_{l}$ but the labels of which are not known.

In contrast with conventional semi-supervised learning, we consider $X_{u}\in\textit{OOD}:=(x_{l+1}\in\textit{OOD},...,x_{l+u}\in\textit{OOD})$ for which the data is not necessarily in the same domain as the $X_{l}$, and a correct existing class label is not available to the model.

We employ deep metric learning, using data to learn a similarity measure on which we cluster. Clustering in deep networks has occurred in other contexts. Learning Discrete Representations via Information Maximizing Self-Augmented Training (IMSAT) uses deep clustering to learn a discrete representation that is invariant to perturbations (Hu et al., 2017). Deep Clustering for Unsupervised Learning of Visual Features improves representation quality by creating labels for existing data points and training on said labels, though they do not train on datapoints outside the original training distribution. Their work offered initial proof that representations can be improved by training on classes discovered via clustering (Caron et al., 2018).

Deep Metric Learning via Facility Location (Song et al., 2016) introduced Normalized Mutual Information, a measure for comparing discovered classes to existing class labels.

Our system is related to the closed-world assumption (Scheirer et al., 2012), where our classifier does not make the assumption that all classes which appear in the test data must have appeared during training. Our system stretches this to a case where the initial training distribution does not include all classes, but where classes are introduced to the training distribution after being created by our model. Unseen Class Discovery in Open World Classification unmakes this assumption but focuses on finding the correct number of outstanding classes, rather than finding the data points which belong to those classes. (Shu et al., 2018) Towards Open World Recognition (Bendale & Boult, ) sets the frame for engineering open world systems but does not discover new object categories itself, leaving that to human labeling. iCaRL: Incremental Classification and Representation Learning (Rebuffi et al., ) focuses on incremental concept discovery, but does not discover the classes itself. Simultaneous Class Discovery and Classification of Microarray Data Using Spectral Analysis (Qiu & Plevritis, 2009) both discovers classes (with a spectral analysis technique) and classifies those classes. No deep metric learning is used to represent the data, but the application to microarray data is an example of the value of the technique.

The problem of generating new labels has been touched on in Bayesian Non-parametrics, including Distance Dependent Chinese Restaurant Process (Blei, 2011) and Latent Dirichlet Allocation (LDA) (Blei, 2003). We compare against LDA while applying our method to the Clnic OOS dataset.

We present a continual learning and lifelong learning system, which is influenced by the continual learning frontier. For example, Continual Unsupervised Representation Learning (Rao et al., 2019) aims to create an algorithm that discovers new concepts over its lifetime, using a generative model of past classes to avoid catastrophic forgetting. We show the advantage of leveraging a combination of human-labeled data and algorithm-labeled data in a similar setting.

Self-Supervised learning has seen lots of recent success, especially in language (Radford et al., 2019) (Devlin et al., 2018). Extensions to vision have mainly focused on pretext tasks like prediction context (Doersch et al., 2015), image rotation (Gidaris et al., 2018), or architecture design (Kolesnikov et al., 2019).

Our work is inspired by the transition to the pre-training with infinite training data regime achieved by these models, though it creates a task by using a system of machine learning algorithms (embedding + clustering) to generate the label rather than taking it directly from the training data. This differentiates of from other mergers of self-supervised and semi-supervised learning, such as Self-Supervised Semi-Supervised Learning (Zhai et al., 2019).

Metric-based semi-supervised clustering also focused on using some labeled data to aid unsupervised learning. There was rich activity in the field in the early 2000s (Ex., (Bilenko et al., 2004)). Rather than learning new classes, the goal is to fit existing classes well and learn new clusters around them. Metrics were often learned with an SVM and deep metric learning was rare.

There is a thread in intrinsic motivation research where an agent is trained to generate new tasks that it is capable of performing in an environment and then learn on those tasks (Held et al., 2018), (Schmidhuber, 1991), (Schmidhuber, 2013), (Clune, 2019). Our system has similar properties and motivations, though we focus on the multi-class classification task.

Our representation learner is trained to classify over all of the current training data. It will create embeddings of our OOD data, whether the domain is images or language. The representation learners employed include a small CNN and Resnet50 (He et al., 2016) for image data, and the TextCNN (Kim, 2014) for text data.

We take advantage of (Hendrycks & Gimpel, 2016)’s work in out of distribution detection, which evaluates a given data points as being whether out-of-distribution by thresholding its predictive confidence (i.e., the maximum predictive probability) with a pre-computed cut-off value. Specifically, this method computes the cut-off value as the 95% quantiles of predictive confidence of the training data, and using that confidence level to evaluate new data as being out of distribution. For example, our model may show that 95% of training datapoints have a class that is predicted with a probability of 80% or above. We would then treat new datapoints whose top class is predicted with 80% or less as out of distribution. In general, progress in OOD Detection (Ex. (Liang et al., 2017; Lee et al., 2018)) will lead to more effective class discovery systems, as more data points will correctly be placed into the data store from which new classes are created.

We use the clustering method K-means to transform a high dimensional set of embeddings into new classes to be evaluated. K-means gives a higher quality of candidate classes, as measured by correspondence to the original class set in comparison to other clustering methods like Hierarchical Agglomerative Clustering. Hierarchical clustering returns a hierarchy with multiple levels of class candidates, which allows for mode discovery and a choice of what level at which to define a label set. For us, this was not worth the drop in the quality of discovered clusters.

We initialize k-means with a k-means++ initialization, focusing the initial clusters on regions that are far from one another.

We run k-means 10 times with different centroid seeds, using the result that performs best on a measure of inertia. Inertia is the sum of squared distances of samples to their closest cluster center.

Carefully choosing which new class candidates are worth adding to the training dataset is important to preserving the quality of the training set. We explore three major heuristics. The first is class learnability, which asks whether a model trained on the candidate classes can effectively learn to predict whether a datapoint is from the candidate class. Here we ask if a class, when added to the training dataset, leads to an improvement in the ability of the representation learner trained on that dataset to quickly adapt to new tasks. The third is a measure of cluster density, assuming that denser clusters are more likely to contain datapoints that are similar to each other and so will have a higher quality. This led to a negative result, where denser clusters were not related to cluster accuracy. One important realization is that all three are features of clusters. The can be used in tandem or be put through a machine learning model that predicts a cluster’s value.

We use two major measures to evaluate class discovery performance in this paper.

The first major evaluation measure is cluster accuracy. We measure the amount of overlap that a given out-of-distribution cluster has with the removed class that it has the most overlap with. For example, an MNIST class with 6 ’9’ values and 4 ’7’ values would be mapped to class 9 and given a cluster accuracy of 60% (6/10). Multiple clusters are allowed to be mapped to the same unseen original class label.

Second, we introduce the dataset label reconstruction accuracy to measure the efficacy of our class discovery system. Dataset reconstruction asks what the overall overlap of the original class labels are with the labels discovered and assigned during class discovery. This overlap score is determined by maximum overlap. For example, if a cluster is discovered, the entire cluster is assigned to a new label. The check against the original dataset comes from comparing with the ground truth labels of the datapoints for each class. The class with the plurality of labels represented has every datapoint in the new cluster assigned to it.

One very important advantage of the dataset reconstruction score over cluster accuracy is its generality. It’s easy to imagine class discovery systems being created, for example, via raw discrete latent variable models. In that case, comparing discrete latent variable models to this technique would require a measure that wasn’t centered on clustering.

Let training labels $Y_{\ell}:=(y_{1},...,y_{\ell})$ where $\ell$ is equal to the number of training datapoints. Let OOD labels $Y_{o}:=(y_{t},...,y_{t}+o)$ where $o$ is equal to the number of OOD data points whose correct class label is not available to the model. and which correspond to the labels that the system discovered itself. Let the normalized cluster size $\boldsymbol{w}$ be the weighting of each discovered cluster based on the number of datapoints in the cluster and $\boldsymbol{a}$ be the accuracy of those clusters, which is the maximum overlap against the true label set. We compute the dataset reconstruction accuracy through an indicator function for whether a discovered label matches the true label distribution or not, which is

In our case, we use clustering to compute the datset reconstruction accuracy over K clusters, giving us:

We evaluate the method on both image and language modalities. We consider MNIST (LeCun, ), Fashion-MNIST (Xiao et al., 2017), CIFAR 10 (Krizhevsky & Hinton, 2010) for image, and CLINC out-of-scope (OOS) intent detection benchmark (Larson et al., 2019) for language.

There are a few settings for experiments, where our datasets are split according to the experiment. In the default setting, half of the classes of (MNIST, Fashion-MNIST, CFAIR10) have the labels stripped from their datapoints. In self-supervised style, a supervised training set is composed of the other 5 classes. An oracle correctly determines the split between in distribution data and out-of-distribution data.

In the second setting, an OOD detector determines whether data points are slotted into existing classes or pooled with other out of distribution data points for assignment to new class labels. This is required for the dynamic model because if it creates two classes which are have an identical ground truth label it will struggle to differentiate them from one another and training will stall.

On the CLINC OOS dataset, 120 classes are used during training and 30 classes are used for class discovery evaluation, as well as an out-of-scope class which contains random user utterances that do not match any of the known intents.

This important experiment shows that if you are able to successfully label new classes, your ability to discover new classes improves. The difference between training on 2 classes and training on 5 classes led to an increase from 43% accuracy to 71% accuracy in the small CNN case on MNIST (See Table 1). This dramatic impact from the addition of training data shows the potential of effective class discovery systems that dynamically feed back on themselves as they become more capable of discovering classes with each new class they discover.

The representation learner used in our experiments is either Resnet50 (He et al., 2016) or a small CNN. The small CNN has a single convolutional layer with 32 filters and a Relu activation, followed by a flattening and a dense layer with 128 neurons and a relu activation.

We apply an Adam optimizer with a learning rate of $0.001$, beta 1 of $0.9$, beta 2 of $0.999$, and epsilon of $1e-0.7$. Our error metric is the cross entropy loss function.

Our OOD baseline detector has its threshold set to 95% confidence on the training data.

The number of clusters $k$ in all vision experiments is 15. This hyperparameter has a fairly strong effect on both cluster accuracy (higher accuracy with higher $k$) and on dataset reconstruction score, so it’s kept constant through baseline comparisons.

We use Tensorflow 2.0 for all of our neural network models (Abadi et al., 2016). For our linear models and clustering we use Scikit-Learn (Pedregosa et al., 2011).

An optimial hyperparameter sweep (ex., for the number of clusters or hyperparameters for the representation learner) would optimize for the dataset reconstruction accuracy in the setting where you’re limited to creating a number of classes that is equal to the number of classes removed from the dataset.

All models were trained on a single GPU or CPU at a time. Our main compute infrastructure was a set of Nvidia p100 GPUs.

We compare 3 methods for generating new classes via clustering, assuming an oracle allows you to correctly determine which datapoints are not from the existing class label set. These results are displayed in Table 3.

’Random embedding’ uses an untrained single convolutional layer followed by a single feedforward layer to embed the image. ’Semi-Supervised’ trains that same model on the first 5 classes from each dataset. ’Dynamic’ trains on the first 5 classes for an epoch, discovers and adds the most learnable cluster, and then trains on the entire dataset until 5 new classes have been added.

One simple approach to class discovery is to create classes over all the out of distribution datapoints at once, assigning every OOD datapoint to a new label. That approach can be made dynamic by choosing (for example) one new class at a time. Once that class is chosen, a re-training process integrates that class into the existing representation learner. That retrained model is used to generate and select another class, in an ongoing process.

We show results from both setups. In the static setting, an overall dataset label reconstruction accuracy on MNIST of 84.73%. In the dynamic setting, we see an overall dataset label reconstruction accuracy of 91.13%.

As the dynamic model adds and trains on new classes, its ability to recover the labels over the entire training dataset improves.

One major challenge is finding new classes that are pure, where most data points correspond to a coherent concept. If discovered classes are impure, the training process will encounter incorrect and noisy labels. Training on those labels can be difficult, and lead to a degradation in training accuracy as well as a slower training process (Zhang et al., 2016). Do note, however, that the incorrect labels will be closer to correct than randombly permuted labels, because they will have been close datapoints in the embedding space.

We introduce a new technique, learnability, as a test of the quality of a cluster which could be added as a new class. The heuristic is as follows: Treat all of the clusters that come out of the OOD dataset as classes to be learned by a fresh, untrained model. Train that model on the clusters, treating the clusters that can be easily learned (for which you obtain a high accuracy) as better options for inclusion that clusters that are challenging to learn.

This will lead to selecting new classes that are separable from the rest of the data. Including existing classes in this set is also an option, which can check for the separability of discovered classes with existing classes as well as separation from potential classes.

Using learnability, MNIST reconstruction accuracy is $0.9113$. Choosing random clusters leads to an accuracy of $0.9004$. Fashion-MNIST learnability reconstruction accuracy is $0.8695$, while the random baseline gives $0.8537$. The difference on CIFAR 10 is $0.7319$ to $0.6841$.

We next show that the class discovery method also applies to the language modality. In particular, we consider the real-world task of intent detection in goal-oriented dialog modeling, where the goal is to classify a given user utterance to one of the many pre-defined task categories (i.e., intents), so that the dialog manager can direct the downstream flow of the conversation accordingly toward certain task fulfillment modules (Zhao et al., 2019). In this context, the ability of a dialog agent to automatically discover novel user intent is important for the continuous improvement and refinement of the system. For example, the discovered novel intent types (see, e.g., Table 5) can be used by the dialog system to implement new response and behavior for the discovered class.

We consider the CLINC Out-of-scope (OOS) benchmark (Larson et al., 2019), which contains 150 intent categories with 150 utterance in each category, and an extra 1200 out-of-domain utterances which do not match any of the known intents. For the representation learner, we consider the standard 1-layer, 128-hidden-unit TextCNN (Kim, 2014) with filter sizes 3, 4, 5 and initialize the word embeddings using GloVe (Pennington et al., 2014). We train the model on 120 randomly-sampled classes from the OOS dataset, and evaluate the quality of the learned representation on the rest of the 30 classes. The model was trained for 200 epochs with minibatch size 128 and step size 0.001 using Adam optimizer, and reached a final classification accuracy of 0.9466. For class discovery, we apply K-means with 30 clusters on the TextCNN’s hidden representation for the 30 hold-out classes, and compute the label rediscover accuracy of the resulting clusters against the ground truth labels.

To evaluate the benefit of semi-supervision to class discovery, we compare the result against five popular or the state-of-the-art unsupervised approaches in text domain. Specifically, we consider the classic methods of Latent Dirichlet Allocation (LDA) and Non-negative Matrix Factorization (NMF), which are based on the term frequency (TF) and term frequency–inverse document frequency (TF-IDF) representations of the utterances, respectively. Next, we consider two standard token-level embedding approaches word2vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014), where we compute the sentence-level embedding using the TF-IDF-weighted average of the token-level embeddings. Finally, we consider the state-of-the-art sentence embedding approach Universal Sentence Encoder (USE) (Larson et al., 2019). Universal Sentence Encoder is pre-trained jointly on large-scale web corpuses and on natural language inference tasks, and has illustrated state-of-the-art performance in multiple natural language tasks including semantic textual similarity, opinion polarity classification and phrase level sentiment analysis. For all the embedding methods, we perform clustering using K-means with 30 clusters.

We repeat each of the above methods for 20 times and report their mean and standard deviation of label rediscover accuracy in Table 4. As shown, the class discovery accuracy based on the semi-supervised TextCNN embedding outperforms the both classic and token-level approaches by a clear margin, and is on-par with Universal Sentence Encoder, despite not being pre-trained on large-scale corpus like USE. In particular, we notice that the GloVe-initialized TextCNN outperforms the original GloVe by around $5\%$, illustrating the benefit of semi-supervision in improving the representation quality of the learned embedding.

Finally, to illustrate the proposed approach’s practical utility in discovering novel intent domains, we perform class discovery on the out-of-scope utterances in the OOS data. Specifically, we apply K-means to the TextCNN representation of the out-of-scope utterances, with the number of clusters determined using the elbow method based on silhouette score (de Amorim & Hennig, 2015). We estimate the learnability of the discovered clusters based on the out-of-sample classification accuracy of a logistic regression model, and report the top clusters (with classification accuracy $>0.95$). We show example utterances in the top clusters in Table 5 and report the full content of the discovered clusters in the Supplementary. We observe that the learnability heuristic is able to select utterance clusters with consistent lexical and syntactical patterns, which tend to correspond to groups of semantically coherent user requests in the context of spoken dialogues. How to incorporate further a priori common sense knowledge (e.g., knowledge graph groundings) into this semi-supervised framework to further improve the semantic precision of the discovered utterance clusters an interesting avenue for future research.