Improved Hierarchical Clustering on Massive Datasets with Broad Guarantees

There exist algorithms that can achieve up to two of these qualities at a time. Affinity Clustering, notably, exhibits good empirical performance and scalability using MPC. While Bateni et al. (2017) describe some minor theoretical guarantees for Affinity Clustering, we believe that proving an algorithm’s ability to optimize for revenue and value is a stronger and more well-founded result due to their popularity and relation to Dasgupta’s cost function. A simple random divisive algorithm proposed by Charikar & Chatziafratis (2017) was shown to achieve a $1/3$ expected approximation for revenue and can be efficiently implemented using MPC. However, it is notably nondeterministic, and we show that it does not exhibit good empirical performance. Similarly, balanced partitioning may achieve balanced clusters, but it is unclear whether it is scalable, and it has not been shown to achieve strong theoretical guarantees.

For both revenue and value, Average Linkage achieves near-state-of-the-art $1/3$ and $2/3$-approximations respectively (Moseley & Wang, 2017; Cohen-Addad et al., 2018). Charikar et al. (2019) marginally improves these factors to $1/3+\epsilon$ for revenue and $2/3+\epsilon$ for value, through semi-definite programming (SDP, a non-distributable method). However, since value and revenue both strove to characterize Average Linkage’s optimization goal, and this was only marginally beat by an SDP, we do not expect to surpass Average Linkage in the restrictive distributed context.

After the completion of our work, a new algorithm was introduced achieving an impressive 0.585 approximation for revenue Alon et al. (2020). While this result was unknown during the course of this work, it sets a new standard to strive for in future work.

In this work, we propose a new algorithm, Matching Affinity Clustering, for distributed hierarchical clustering. Inspired by Affinity Clustering’s reliance on the minimum spanning tree in order to greedily merge clusters (Bateni et al., 2017), Matching Affinity Clustering merges clusters based on iterative matchings. It notably generalizes to both the edge weight similarity and dissimilarity contexts, and achieves all four desired qualities.

In Section 4, we theoretically motivate Matching Affinity Clustering by proving it achieves a good approximation for both revenue and value (the latter depending on the existence of an MPC minimum matching algorithm), nearing the bounds achieved by Average Linkage:

Furthermore, in Theorem 3, we prove that Matching Affinity Clustering can give no guarantees with respect to revenue or value. The discussion and proof of this theorem can be found in the Appendix.

We also present an efficient and near-optimal MPC algorithm for $k$-sized maximum matching in Theorem 4 in Section 3. This is used by Matching Affinity Clustering.

To evaluate the empirical performance of our algorithm, we run Bateni et al. (2017)’s experiments used for Affinity Clustering on small-scale datasets in Section 6. We find Matching Affinity Clustering performs competitively with respect to state-of-the-art algorithms. On filtered, balanced data, we find that Matching Affinity Clustering consistently outperforms other algorithms by at least a small but clear margin. This implies Matching Affinity Clustering may be more useful on balanced datasets than Affinity Clutsering.

To confirm the balance of our algorithm, we are able to prove that Matching Affinity Clustering achieves perfectly balanced clusters on datasets of size $2^{N}$, and otherwise guarantee near balance (a cluster size ratio of at most 2). See Lemma 1. This was also confirmed in our empirical evaluation in Section 6.

Finally, we show in Section 4 that Matching Affinity Clustering is highly scalable because it was designed in the same MPC framework as Affinity Clustering. We provide similar complexity guarantees to Affinity Clustering.

Matching Affinity Clustering is ultimately a nice, simply motivated successor to Affinity Clustering that achieves all four desired qualities: empirical performance, theoretical guarantees, balance, and scalability. No other algorithm that we know of does this.

The standard hierarchical clustering problem takes in a set of data represented as a graph where weights on edges measure similarity or dissimilarity between data. In this paper, edge weights, denoted $w_{G}(u,v)$ for a graph $G$ and may be similarities or differences as specified.

One of the most simple, serial hierarchical clustering algorithms is Average Linkage, which provides a good approximation for Moseley and Wang’s dual (Moseley & Wang, 2017). It starts with each vertex being its own cluster, and builds the next cluster by merging the two clusters $A$ and $B$ that maximize the average distance between points in the clusters:

Finally, we note that much of our results occur with high probability (whp), as is true with many MPC algorithms. This means they occur with probability $1-1/(\Omega(n))$, where the denominator is generally exponential in $n$. The probabilistic aspects of the algorithm come from our use of Ghaffari et al. (2018)’s maximum matching algorithm, which finds a $(1+\epsilon)$-approximate maximum matching with high probability. As we do not introduce a probabilistic aspect ourselves, this will not be discussed in depth in our proofs, but will be stated in the theorems and lemmas.

Consider some hierarchical tree, $T$. We say $i\lor j$ for leaves $i$ and $j$ is the least common ancestor of $i$ and $j$. The subtree rooted at an interior vertex $v$ is $T[v]$, therefore the subtree representing the smallest cluster that contains both $i$ and $j$ is $T[i\lor j]$. Let $\operatorname{leaves}(T[v])$ be the set of leaves in $T[v]$, and $\operatorname{non-leaves}(T[v])$ be the set of all of the leaves of $T$ but not $T[v]$. Now we can describe Dasgupta’s function.

To minimize edge weight contribution, we want a small $|\operatorname{leaves}(T[i\lor j])|$ for heavy edges. This ensures that heavy edges will be merged earlier in the tree. To calculate this, it is easier to break it down into a series of merge costs for each node in $T$. It counts the costs that accrue due to the merge at that node so that we can keep track of the cost throughout the construction of $T$. It is defined as:

This breaks down the cost of a hierarchy tree into a series of merge costs. Consider some edge, $(i,j)$. At each merge containing exclusively $i$ or $j$, this edge contributes $w_{G}(i,j)$ times the size of the other cluster. In the hierarchical tree, this counts how many vertices accrue during merges along the paths from $i$ and $j$ to $i\lor j$. However, this does not account for the leaves $i$ or $j$ themselves, so we need to add $w_{G}(i,j)$ two extra times in addition to each merge. This means we can derive the total cost from the merge costs as: $\operatorname{cost}_{G}(T)=2\sum_{i,j\in V(G)}w_{G}(i,j)+\sum_{\text{merges }A,B}\operatorname{merge-cost}_{G}(A,B).$

Next, we consider Moseley and Wang’s dual to Dasgupta’s function Moseley & Wang (2017).

We can, in a similar fashion to Dasgupta’s cost function, break revenue down into a series of merge revenues.

Note that for some $i$ and $j$, $w_{G}(i,j)$ is contributed exactly once, when $i$ and $j$ merge at $i\lor j$, and $n-|A|-|B|$ is the number of non-leaves at that step. Therefore: $\operatorname{rev}_{G}(T)=\sum_{i,j\in V(G)}\operatorname{merge-rev}_{G}(i,j)$. In addition, note the contribution of each $i,j$ pair, which is scaled by $w_{G}(i,j)$, is the number of leaves of $i\lor j$ for revenue, and the number of non-leaves of $i\lor j$ for cost. Therefore the contribution of each edge for revenue is $n$ minus the contribution for cost, scaled by $w_{G}(i,j)$. In other words: $\operatorname{rev}_{G}(T)=n\sum_{i,j\in V(G)}w_{G}(i,j)-\operatorname{cost}_{G}(T)$.

While cost is a popular and well-founded metric, Charikar & Chatziafratis (2017) found that it is not constant factor approximable under the Small Set Expansion Hypothesis. On the other hand, Moseley & Wang (2017) proved that revenue is, and Average Linkage achieves a $1/3$-approximation. This makes it a more practical function to work with.

Our other function of interest is Cohen-Addad et al. (2018)’s value function. This was introduced as a Dasgupta-inspired optimization function where edge weights represent distances. It looks exactly like cost, except it is now a maximization function because it is in the distance context.

Like revenue, value is constant factor-approximable. In fact, the best approximation (other than an SDP) for value is Average Linkage’s $2/3$-approximation (Cohen-Addad et al., 2018). To our knowledge, there are no distributable approximations for value.

Massively Parallel Communication (MPC) is a model of distributed computation used in programmatic frameworks like MapReduce (Dean & Ghemawat, 2008b), Hadoop (White, 2009), and Spark (Zaharia et al., 2010). MPC consists of “rounds” of computation, where parts of the input are distributed across machines with limited memory, computation is done locally for each machine, and then the machines send limited messages to each other. The primary complexities of interest are machine space, which should be $\widetilde{O}(n)$, and the number of rounds. Many MPC algorithms are extremely efficient. For instance, Affinity Clustering in some cases can have constantly many rounds, and otherwise may use up to $O(\log^{2}n)$ rounds (Bateni et al., 2017).

Matching Affinity Clustering is defined in Algorithm 2. Its predecessor, Affinity Clustering, uses the MST to select edges to merge across, which sometimes causes imbalanced clusters. This is one reason why it cannot achieve a good approximation for revenue or value (Theorem 3). We fix this by, instead, using iterated maximum matchings (for similarity edge weights) and minimum perfect matchings (for dissimilarity edge weights). This ensures that on $n=2^{N}$ vertices for some $N$, clusters will always be perfectly balanced. Otherwise, we will show how to achieve at least relative balance.

The algorithm starts with one cluster per each of $n$ vertices. Let $2^{N}$ be the smallest value such that $2^{N}\geq n$. It finds a maximum (resp. minimum) matching of size $k=2n-2^{N}$ (line 8, this means it matches $2n-2^{N}$ vertices with $n-2^{N-1}$ edges) and merges these vertices (line 12, Figure 2). Note that if $n=2^{N}$, then $k=0$ and the first step is a perfect matching. After this step, we have $2^{N-1}$ clusters. We then transform the graph into a graph of clusters with edge weights equal to the Average Linkage between clusters (lines 17-21). We find a maximum (resp. minimum) perfect matching of clusters in this new graph (line 10), then iterate.

Now, we evaluate the efficiency and approximation factor of Matching Affinity Clustering with respect to revenue. In this section, edge weights represent the similarity between points. Proofs are in the Appendix in the full version of the paper. Ultimately, we will show the following.

This will be a significant motivation for Matching Affinity Clustering’s theoretical strength. As stated previously, one of the goals of Matching Affinity Clustering is to keep the cluster sizes balanced at each level. However, in the first step, note that Matching Affinity Clustering creates $n-2^{N-1}$ clusters of size 2, and the rest of the vertices form singleton clusters. Therefore, to use this benefit of Matching Affinity Clustering, we need to ensure that cluster sizes will never deviate too much.

After every matching, the algorithm creates a new graph with vertices representing clusters and edges representing the average linkage between clusters. We will call this a clustering graph.

The fact that the edge weights in the clustering graph are the average linkage between clusters denotes the similarities between Matching Affinity Clustering and Average Linkage. Essentially, we are trying to optimize for average linkage at each step, but instead of merging two clusters, we merge many pairs of clusters at once with a maximum matching.

Since Matching Affinity Clustering computes this graph, we must show how to efficiently transform a clustering graph at the $i$th level, $\mathcal{C}(G,C^{i})$ with clustering $C^{i}$, into a clustering at the $i+1$th level, $\mathcal{C}(G,C^{i+1})$ with clustering $C^{i+1}$.

This will eventually be used for our proof of efficiency of Theorem 1. For now, we return our attention to the approximation factor. Our approximation proof is going to observe the total merge cost and revenue across all merges on a single level of the hierarchy. For concision, we introduce the following notation to describe cost and revenue over a single clustering.

In order to prove an approximation for revenue, we want to compare each clustering revenue and cost. First, we must show that Matching Affinity Clustering has a large clustering revenue at any level.

Now we address clustering cost. This time, we must show an upper bound for clustering cost at the $i$th level in terms of clustering revenue at the $i$th level. Let $M_{i}$ be the matching Matching Affinity Clustering uses to merge $C^{i}$ into $C^{i+1}$. Then $M_{i}$ is a $(1-\epsilon)$-approximation of the optimum $M_{i}^{*}$.

Now we are ready to prove the approximation factor for Matching Affinity Clustering. We combine Lemma 4 with properties of revenue from Section 2.2 to obtain an expression for revenue in terms of $(n-2)$ times the sum of weights in the graph. We use this as a bound for the optimal revenue.

Finally, the round complexity is limited by the iterations and calls to the matching algorithm. The space complexity is determined by the clustering graph construction.

Lemmas 5 and 6 are sufficient to prove Theorem 1. Our algorithm achieves an approximation for revenue efficiently in the MPC model. In addition, the algorithm creates a desirably near-balanced hierarchical clustering tree.

We now prove the approximation bound tightness for Matching Affinity Clustering when $|V|=2^{N}$. Recently, Charikar et al. (2019) proved by counterexample that Average Linkage achieves at best a $(1/3+o(1))$-approximation on certain graphs. We find that Matching Affinity Clustering acts the same as Average Linkage on these graphs, and so has at best a $(1/3+o(1))$-approximation.

Now we consider Matching Affinity Clustering when edge weights represent distances instead of similarities. In this context, instead of running a $k$-sized maximum matching and then iterative general maximum matchings, we run a $k$-sized minimum matching and then iterative general minimum perfect matchings. Therefore, this algorithm is dependent on the existence of a $k$-sized minimum matching algorithm in MPC. Due to its similarity to other classical problems with $1+\epsilon$ solutions in MPC (Ghaffari et al., 2018; Behnezhad et al., 2019), we conjecture:

Given such an algorithm, we can show that Matching Affinity Clustering approximates value.

The proof for this result is quite similar to the proof for the $2/3$-approximation of Average Linkage by Cohen-Addad et al. (2018). Instead of focusing on single merges, however, we observed the entire set of merges across a clustering layer in our hierarchy. Then we can make the same argument about the value across an entire level of the hierarchy, and use the cluster balance from Lemma 1 to achieve our result.

If Conjecture 1 holds, then the approximation factors become $2/3-\epsilon$ and $1/3-\epsilon$ respectively. We see a similar pattern as the revenue result, where the algorithm nears the state-of-the-art $2/3$-approximation achieved by Average Linkage (Cohen-Addad et al., 2018) on datasets of size $n=2^{N}$, and still achieves a constant factor in general. Finally, we can additionally show the former approximation is tight. See the construction and proofs in the Appendix.

In this section, we only consider Matching Affinity Clustering in the similarity edge weight context. The round complexities of Matching Affinity Clustering and regular Affinity Clustering depend on graph qualities, and in certain cases one outperforms the other. On dense graphs with $n^{1+c}$ edges for constant $c$, Bateni et al. (2017) showed that Affinity Clustering runs in $\lceil\log(c/\epsilon)\rceil+1$ rounds. On sparse graphs, it runs in $O(\log^{2}n)$ rounds, and it runs in $O(\log n)$ rounds when given access to a distributed hash table. We saw that Matching Affinity Clustering runs in $O\left(\log(n)\log\log(n)\cdot(1/\epsilon)^{O(1/\epsilon)}\right)$ rounds on graphs of size $2^{N}$, and $O\left(\log(nW)\log\log(n)\cdot(1/\epsilon)^{O(1/\epsilon)}\right)$ in general for max edge weight $W$.

There are two situations where our algorithm outperforms Affinity Clustering. First, if the graph is sparse and the number of vertices is $2^{N}$, then our algorithm runs in $O\big{(}\log(n)\allowbreak\log\log(n)\cdot(1/\epsilon)^{O(1/\epsilon)}\big{)}$ rounds, and Affinity Clustering runs in $O(\log^{2}(n))$ rounds. Otherwise, if the graph is sparse, Matching Affinity Clustering performs better as long as the largest edge weight is $W=o\left(\frac{\exp(\log^{2}(n)/\log\log(n))}{n}\right)$. This is strictly larger than constant. If $W$ is large, Affinity Clustering is slightly more efficient. Finally, if the graph is dense, Affinity Clustering achieves an impressive constant round complexity, and is therefore more efficient. In any case, Matching Affinity Clustering is an efficient and highly scalable algorithm.

In addition, we are interested in evaluating the balance between cluster sizes in the clusterings, which indicates how good our algorithms are at evaluating balanced data. We use the cluster size ratio of a clustering, which was observed in Bateni et al. (2017). For a clustering $X=\{X_{1},\ldots,X_{r}\}$, the size ratio is $\min_{i,j\in[r]}|X_{i}|/|X_{j}|$.

In Figure 3(a), we see the Rand indices of Max Matching Affinity Clustering (ie, in the similarity context), Min Matching Affinity Clustering (ie, in the distance context), and Affinity Clustering. A Rand index is between 0 and 1, where higher scores indicate the clustering is more similar to the ground truth. Matching Affinity Clustering performs similarly to state of the art algorithms like Affinity Clustering on all data except the Soy-Bean dataset. A full evaluation on other algorithms (see the Appendix) illustrates that Matching Affinity Clustering outperforms other algorithms like Random Clustering and Average Linkage.

Figure 3(b) depicts the same information but on a slightly modified dataset. Here, we randomly remove data until (1) the dataset is of size $2^{N}$, and (2) ground truth clusters are balanced. We did this 50 times and took the average results. This is motivated by Matching Affinity Clustering’s stronger theoretical guarantees on datasets of size $2^{N}$ and ensured cluster balance. As expected, Matching Affinity Clustering performs consistently better than Affinity Clustering on filtered data, albeit by a a small margin in many cases. This shows that, experimentally, Matching Affinity Clustering performs better on balanced datasets of size $2^{N}$.

Finally, Figures 3(c) and 3(d) depict the cluster size ratios on the raw and filtered data respectively. In theory, at every level in the hierarchy of Matching Affinity Clustering, no cluster can be less than half as small as another (Lemma 1). However, in our evaluation, we are comparing a single $k$-clustering, which may not precisely correspond to a level in the hierarchy. In this case, we take some clusters from the first level with fewer than $k$ clusters and the last level with more than $k$ clusters. Therefore, since cluster sizes double at each level, the lower bound for the cluster size ratio is now $1/4$. This is reflected Figure 3(c), where Matching Affinity Clustering stays consistently above this minimum, and often exceeds it by quite a bit. On the filtered data (Figure 3(d)), Matching Affinity Clustering maintains perfect balance in every instance, whereas Affinity Clustering performs much worse. Thus, Matching Affinity Clustering has proven empirically successful on small datasets.

Here we include more complete visualizations of the performance of all tested algorithms. Like in the main body of text, we find the rand index and cluster size ratio on balanced and filtered data. These tests were run in the same way as the tests in the main body, we just present results on other common algorithms for completeness. The results are presented in Figure 4 in the Appendix.

Most of these results are as expected and simply reproduce the results from Bateni et al. (2017). However, we add one more algorithm: random clustering. Again, this is the clustering that randomly recursively partitions the data into a hierarchy. In our experiments, random clustering had surprisingly good cluster balance ratios (see Figure 4(d)). In fact, on raw data, it on average had more balanced clusters than Matching Affinity Clustering on three of the datasets.

There are three main observations about why Matching Affinity Clustering is still clearly more empirically successful than random clustering. First, notice that on filtered data in Figure 4(e), Matching Affinity Clustering has more balanced clusters than Random clustering by a very wide margin. Second, it is clear in Figures 4(b) and 4(c) that Matching Affinity Clustering consistently and significantly outperforms random clustering. Third, random clustering is nondeterministic, whereas Matching Affinity Clustering is works with high probability. Therefore, Matching Affinity Clustering’s theoretical strengths and empirical performances are much stronger assurances than that of random clustering. Therefore, while an argument can be made that random clustering seems to empirically balance clusters well, Matching Affinity Clustering still does better in a number of respects, and thus is a more useful algorithm.