Interactive Bayesian Hierarchical Clustering

Clustering is a basic tool of exploratory data analysis. There are a variety of efficient algorithms—including $k$-means, EM for Gaussian mixtures, and hierarchical agglomerative schemes—that are widely used for discovering “natural” groups in data. Unfortunately, they don’t always find a grouping that suits the user’s needs.

This is inevitable. In any moderately complex data set, there are many different plausible grouping criteria. Should a collection of rocks be grouped according to value, or shininess, or geological properties? Should animal pictures be grouped according to the Linnaean taxonomy, or cuteness? Different users have different priorities, and an unsupervised algorithm has no way to magically guess these.

As a result, a rich body of work on constrained clustering has emerged. In this setting, a user supplies guidance, typically in the form of “must-link” or “cannot-link” constraints, pairs of points that must be placed together or apart. Introduced by Wagstaff & Cardie (2000), these constraints have since been incorporated into many different flat clustering procedures (Wagstaff et al., 2001; Bansal et al., 2004; Basu et al., 2004; Kulis et al., 2005; Biswas & Jacobs, 2014).

In this paper, we introduce constraints to hierarchical clustering, the recursive partitioning of a data set into successively smaller clusters to form a tree. A hierarchy has several advantages over a flat clustering. First, there is no need to specify the number of clusters in advance. Second, the tree captures cluster structure at multiple levels of granularity, simultaneously. As such, trees are particularly well-suited for exploratory data analysis and the discovery of natural groups.

There are several well-established methods for hierarchical clustering, the most prominent among which are the bottom-up agglomerative methods such as average linkage (see, for instance, Chapter 14 of Hastie et al. (2009)). But they suffer from the same problem of under-specification that is the scourge of unsupervised learning in general. And, despite the rich literature on incorporating additional guidance into flat clustering, there has been relatively little work on the hierarchical case.

What form might the user’s guidance take? The usual must-link and cannot-link constraints make little sense when data has hierarchical structure. Among living creatures, for instance, should elephant and tiger be linked? At some level, yes, but at a finer level, no. A more straightforward assertion is that elephant and tiger should be linked in a cluster that does not include snake. We can write this as a triplet $(\{\mbox{\tt elephant},\mbox{\tt tiger}\},\mbox{\tt snake})$. We could also assert $(\{\mbox{\tt tiger},\mbox{\tt leopard}\},\mbox{\tt elephant})$. Formally, $(\{a,b\},c)$ stipulates that the hierarchy contains a subtree (that is, a cluster) containing $a$ and $b$ but not $c$.

A wealth of research addresses learning taxonomies from triplets alone, mostly in the field of phylogenetics: see Felsenstein (2004) for an overview, and Aho et al. (1981) for a central algorithmic result. Let’s say there are $n$ data items to be clustered, and that the user seeks a particular hierarchy $T^{*}$ on these items. This $T^{*}$ embodies at most ${n\choose 3}$ triplet constraints, possibly less if it is not binary. It was pointed out in Tamuz et al. (2011) that roughly $n\log n$ carefully-chosen triplets are enough to fully specify $T^{*}$ if it is balanced. This is also a lower bound: there are $n^{\Omega(n)}$ different labeled rooted trees, so each tree requires $\Omega(n\log n)$ bits, on average, to write down—and each triple provides $O(1)$ bits of information, since there are just three possible outcomes for each set of points $a,b,c$. Although $n\log n$ is a big improvement over $n^{3}$, it is impractical for a user to provide this much guidance when the number of points is large. In such cases, a hierarchical clustering cannot be obtained on the basis of constraints alone; the geometry of the data must play a role.

We consider an interactive process during which a user incrementally adds constraints.

• •

  Starting with a pool of data $X\subseteq\mathbb{R}^{d}$, the machine builds a candidate hierarchy $T$.

• •

  The set of constraints $C$ is initially empty.

• •

  Repeat:

  • –

    The machine presents the user with a small portion of $T$: specifically, its restriction to $O(1)$ leaves $S\subset X$. We denote this $T|_{S}$.

  • –

    The user either accepts $T|_{S}$, or provides a triplet constraint $(\{a,b\},c)$ that is violated by it.

  • –

    If a triplet is provided, the machine adds it to $C$ and modifies the tree $T$ accordingly.

In realizing this scheme, a suitable clustering algorithm and querying strategy must be designed. Similar issues have been confronted in flat clustering—with must-link and cannot-link constraints—but the solutions are unsuitable for hierarchies, and thus a fresh treatment is warranted.

The most basic form of hierarchical clustering is a rooted binary tree with the data points at its leaves. This is sometimes called a cladogram. Very often, however, the tree is adorned with additional information, for instance:

1. 1.

   An ordering of the internal nodes, where the root is assigned the lowest number and each node has a higher number than its parent.

   This ordering uniquely specifies the induced $k$-clustering (for any $k$): just remove the $k-1$ lowest-numbered nodes and take the clusters to be the leaf-sets of the $k$ resulting subtrees.

2. 2.

   Lengths on the edges.

   Intuitively, these lengths correspond to the amount of change (for instance, time elapsed) along the corresponding edges. They induce a tree metric on the nodes, and often, the leaves are required to be at the same distance from the root.

3. 3.

   Parameters at internal nodes.

   These parameters are sometimes from the same space as the data, representing intermediate values on the way from the root to the leaves.

Many generative processes for trees end up producing these more sophisticated structures, with the understanding that undesired additional information can simply be discarded at the very end. We now review some well-known distributions over trees and over hierarchical clusterings.

Let’s start with cladograms on $n$ leaves. The simplest distribution over these is the uniform. Another well-studied option is the Yule model, which can be described using either a top-down or bottom-up generative process. The top-down view corresponds to a continuous-time pure birth process: start with one lineage; each lineage persists for a random exponential(1) amount of time and then splits into two lineages; this goes on until there are $n$ lineages. The bottom-up view is a coalescing process: start with $n$ points; pick a random pair of them to merge; then repeat. Aldous (1995) has defined a one-parameter family of distributions over cladograms, called the beta-splitting model, that includes the uniform and the Yule model as special cases. It is a top-down generative process in which, roughly, each split is made by sampling from a Beta distribution to decide how many points go on each side. To move to arbitrary (not necessarily binary) splits, a suitable generalization is the Gibbs fragmentation tree (McCullagh et al., 2008).

In this paper, we will work with joint distributions over both tree structure and data. These are typically inspired by, or based directly upon, the simpler tree-only distributions described above. Our primary focus is the Dirichlet diffusion tree (Neal, 2003), which is specified by a birth process that we will shortly describe. However, our methodology applies quite generally. Other notable Bayesian approaches to hierarchical clustering include: Williams (2000), in which each node of the tree is annotated with a vector that is sampled from a Gaussian centered at its parent’s vector; Heller & Ghahramani (2005), that defines a distribution over flat clusterings and then specifies an agglomerative scheme for finding a good partition with respect to this distribution; Adams et al. (2008), in which data points are allowed to reside at internal nodes of the tree; Teh et al. (2008); Boyles & Welling (2012), in which the distribution over trees is specified by a bottom-up coalescing process; and Knowles & Ghahramani (2015), which generalizes the Dirichlet diffusion trees to allow non-binary splits.

Impressive as the Dirichlet diffusion tree is, there is no reason to suppose that it will magically find a tree that suits the user’s needs. But a little interaction can be helpful in improving the outcome.

Let $T^{*}$ denote the target hierarchical clustering. It is not necessarily the case that the user would be able to write this down explicitly, but this is the tree that captures the distinctions he/she is able to make, or wants to make. Figure 3 (left) shows an example, for a small data set of 5 points. In this case, the user does not wish to distinguish between points $1,2,3$, but does wish to place them in a cluster that excludes point $4$.

We could posit our goal as exactly recovering $T^{*}$. But in many cases, it is good enough to find a tree that captures all the distinctions within $T^{*}$ but also possibly has some extraneous distinctions, as in the right-hand side of Figure 3.

Formally, given data set $X$, we say $S\subseteq X$ is a cluster of tree $T$ if there is some node of $T$ whose descendant leaves are exactly $S$. We say $T$ is a refinement of $T^{*}$ if they have the same set of leaves, and moreover every cluster of $T^{*}$ is also a cluster of $T$. This, then, is our goal: to find a refinement of the target clustering $T^{*}$.

We evaluated the convergence properties of five different querying schemes. In a “simple query”, a user is presented with three random data and picks an odd one out. In a “smart query”, a user is unrealistically shown the entire candidate tree and reports a violated triplet. In a “random query”, the user is shown the induced candidate tree over a random subset of the data. In an “active query”, the user is shown a high variance subtree using tree-distance variance. Finally, in an “interleaved query”, the user is alternatively shown a random subtree and a high variance subtree. In each experiment, $T^{*}$ was known, so user queries were simulated by picking a triplet violated by the root split of the queried tree, and if no such triplet existed, recursing on a child. Each scheme was evaluated on four different datasets. The first dataset, MNIST (Lecun et al., 1998), is an 10-way image classification dataset where the data are 28 x 28 images of digits. The target tree $T^{*}$ is simply the $K$-way classification tree over the data. The second dataset is Fisher Iris, a 3-way flower classification problem, where each of 150 flowers has five features. The third dataset, Zoo  (Lichman, 2013), is a set of 93 animals and 15 binary morphological features for each of animals, the target tree being the induced binary tree from the Open Tree of Life  (Hinchliff et al., 2015). The fourth dataset is 20 Newsgroups  (Joachims, 1997), a corpus of text articles on 20 different subjects. We use the first 10 principal components as features in this classification problem. All datasets were modeled with DDT’s with acquisition function $a(t)=1/(1-t)$ and Brownian motion parameter $\sigma^{2}$ estimated from data. To better visualize the different convergence rates of the querying schemes, MNIST and 20 Newsgroups were subsampled to 150 random points.

For each dataset and querying scheme, we instantiated a SPR sampler with no constraints. Every one hundred iterations of the sampler, we performed a query. In subtree queries, we used subsets of size $|S|=10$ and in active querying, the highest-variance subset was chosen from $L=20$ different random subsets. As baselines, we measured the triplet distance of the vanilla DDT and the average linkage tree. Finally, results were averaged over four runs of each sampler. The triplet distances for Fisher Iris and MNIST can be seen in Figure 6. Results for the other datasets can be found in the supplement. Although unrealistic due to the size of the tree shown to the user, the smart query performed the best, achieving minimum error with the least amount of queries. Interleaved followed next, followed by active, random, and simple. In general, the vanilla DDT performed the worst, and the average linkage score varied on each dataset, but in all cases, the subtree querying schemes performed better than both the vanilla DDT and average linkage.

In three datasets (MNIST, Fisher Iris and Zoo), interactive methods achieve higher data likelihood than the vanilla DDT. Initially, the sampler is often restructuring the tree with new triplets and data likelihood is unlikely to rise. However, over time as less triplets are reported, the data likelihood increases rapidly. We thus conjecture that triplet constraints may help the MH algorithm find better optima.

We are interested in studying the non-realizable case, i.e. when there does not exist a tree that satisfies triplet set $C$. We would also like to better understand the effect of constraints on searching for optima using MCMC methods.