Versatile linkage: a family of space-conserving strategies for agglomerative hierarchical clustering

Agglomerative hierarchical clustering constitutes one of the most widely used methods for cluster analysis. Starting with a matrix of dissimilarities between a set of elements, each element is first assigned to its own cluster, and the algorithm sequentially merges the more similar clusters until a complete hierarchy of clusters is obtained [22, 10]. This method requires the definition of the dissimilarity (or distance) between clusters, using only the original distances between their constituent elements. The way these distances are defined leads to distinct strategies of agglomerative hierarchical clustering. To name just two such clustering strategies, single linkage usually leads to an elongate growth of clusters, while complete linkage generally leads to tight clusters that join others with difficulty. Average linkage clustering strategies were developed by Sokal and Michener to avoid the extreme cases produced by single linkage and complete linkage [23]. They require the calculation of some kind of average distance between clusters; average linkage, for instance, calculates the arithmetic average of all the distances between members of the clusters.

More than fifty years ago, Lance and Williams introduced a formula for integrating several agglomerative hierarchical clustering strategies into a single system [13]. Based on this formula they proposed $\beta$-flexible clustering [14], a generalized clustering procedure that provides an infinite number of hierarchical clustering strategies just varying a parameter $\beta$. Similarly, in this work we introduce versatile linkage, a new parameterized family of agglomerative hierarchical clustering strategies that go from single linkage to complete linkage, passing through arithmetic average linkage and other clustering strategies yet unexplored such as geometric linkage and harmonic linkage.

Both $\beta$-flexible clustering and versatile linkage are presented here using variable-group methods [23, 7] that, unlike pair-group methods, admit any number of new members simultaneously into groups. In the case of pair-group methods the resulting hierarchical tree is called a dendrogram, which is built upon bifurcations, while in the case of variable-group methods the resulting hierarchical tree is called a multidendrogram [7], which consists of multifurcations, not necessarily binary ones. Here we use the variable-group algorithm introduced in [7] that solves the non-uniqueness problem, also called the ties in proximity problem, found in pair-group algorithms [22, 11, 5]. This problem arises when there are more than two clusters separated by the same minimum distance during the agglomerative process. Pair-group algorithms break ties between distances choosing a pair of clusters, usually at random. However, different output dendrograms are possible depending on the criterion used to break ties. Moreover, very frequently results depend on the order of the elements in the input data file, what is an undesired effect in hierarchical clustering except for the case of contiguity-constrained hierarchical clustering, which is used to obtain a hierarchical clustering that takes into account the ordering on the input elements. The variable-group algorithm used here always gives a uniquely determined solution grouping more than two clusters at the same time when ties occur, and when there are no ties it gives the same results as the pair-group algorithm.

Section 2 reviews the $\beta$-flexible family of hierarchical clustering strategies, while Section 3 introduces the versatile linkage family. Four case studies are used in Section 4 to perform a descriptive analysis of different hierarchical clustering strategies in terms of cophenetic correlation, mean absolute error, and the proposed new measures of space distortion and tree balance. Finally, some concluding remarks are given in Section 5.

In any procedure implementing an agglomerative hierarchical clustering strategy, given a set of individuals $\Omega=\{x_{1},x_{2},\ldots,x_{n}\}$, initially each individual forms a singleton cluster, $\{x_{i}\}$, and the distances $D(\{x_{i}\},\{x_{j}\})$ between singleton clusters are equal to the dissimilarities between individuals, $d(x_{i},x_{j})$. During the subsequent iterations of the procedure, the distances $D(X_{I},X_{J})$ are computed between any two clusters $X_{I}=\bigcup_{i\in I}X_{i}$ and $X_{J}=\bigcup_{j\in J}X_{j}$, each one of them made up of several subclusters $X_{i}$ and $X_{j}$ indexed by $I=\{i_{1},i_{2},\ldots,i_{p}\}$ and $J=\{j_{1},j_{2},\ldots,j_{q}\}$, respectively. Lance and Williams introduced a formula for integrating several agglomerative hierarchical clustering strategies into a single system [13]. The variable-group generalization of Lance and Williams’ formula, compatible with the fusion of more than two clusters simultaneously, is:

	$\displaystyle D(X_{I},X_{J})=\sum_{i\in I}\sum_{j\in J}\alpha_{ij}D(X_{i},X_{j})+\mbox{}$				(1)
			$\displaystyle+\sum_{i\in I}\sum_{\begin{subarray}{c}i^{\prime}\in I\\ i^{\prime}>i\end{subarray}}\beta_{ii^{\prime}}D(X_{i},X_{i^{\prime}})+\sum_{j\in J}\sum_{\begin{subarray}{c}j^{\prime}\in J\\ j^{\prime}>j\end{subarray}}\beta_{jj^{\prime}}D(X_{j},X_{j^{\prime}})\,,$

where the values of the parameters $\alpha_{ij}$, $\beta_{ii^{\prime}}$ and $\beta_{jj^{\prime}}$ determine the nature of the clustering strategy [7]. This formula is combinatorial [14], i.e., the distance $D(X_{I},X_{J})$ can be calculated from the distances $D(X_{i},X_{j})$, $D(X_{i},X_{i^{\prime}})$ and $D(X_{j},X_{j^{\prime}})$ obtained from the previous iteration and it is not necessary to keep the initial distance matrix $d(x_{i},x_{j})$ during the whole clustering process.

Based on Equation 1, Lance and Williams [14] proposed an infinite system of agglomerative hierarchical clustering strategies defined by the constraint

$$\underbrace{\sum_{i\in I}\sum_{j\in J}\alpha_{ij}}_{\alpha}+\underbrace{\sum_{i\in I}\sum_{\begin{subarray}{c}i^{\prime}\in I\\ i^{\prime}>i\end{subarray}}\beta_{ii^{\prime}}+\sum_{j\in J}\sum_{\begin{subarray}{c}j^{\prime}\in J\\ j^{\prime}>j\end{subarray}}\beta_{jj^{\prime}}}_{\beta}=1\,,$$

(2)

where $-1\leqslant\beta\leqslant+1$ generates a whole system of hierarchical clustering strategies for the infinite possible values of $\beta$. Given a value of $\beta$, the value for $\alpha_{ij}$ can be assigned following a weighted approach as in the original $\beta$-flexible clustering based on WPGMA (weighted pair-group method using arithmetic mean) and introduced by Lance and Williams [13], or it can be assigned following an unweighted approach as in the $\beta$-flexible clustering based on UPGMA (unweighted pair-group method using arithmetic mean) and introduced by Belbin et al. [2]. The standard WPGMA and UPGMA strategies are obtained from weighted and unweighted $\beta$-flexible clustering, respectively, when $\beta$ is set equal to $0$. The difference between weighted and unweighted methods lies in the weights assigned to individuals and clusters during the agglomerative process: weighted methods assign equal weights to clusters, while unweighted methods assign equal weights to individuals. In unweighted $\beta$-flexible clustering the value for $\alpha_{ij}$ is determined proportionally to $|X_{i}||X_{j}|$:

$$\alpha_{ij}=\frac{|X_{i}||X_{j}|}{|X_{I}||X_{J}|}(1-\beta)\,,$$

(3)

where $|X_{i}|$ and $|X_{j}|$ are the number of individuals in subclusters $X_{i}$ and $X_{j}$, respectively, and $|X_{I}|$ and $|X_{J}|$ are the number of individuals in clusters $X_{I}$ and $X_{J}$, i.e., $|X_{I}|=\sum_{i\in I}|X_{i}|$ and $|X_{J}|=\sum_{j\in J}|X_{j}|$. In a similar way, the value for $\beta_{ii^{\prime}}$ is calculated proportionally to $|X_{i}||X_{i^{\prime}}|$, and the value for $\beta_{jj^{\prime}}$ proportionally to $|X_{j}||X_{j^{\prime}}|$:

	$\displaystyle\beta_{ii^{\prime}}$	$\displaystyle=$	$\displaystyle\frac{|X_{i}||X_{i^{\prime}}|}{\sigma_{I}+\sigma_{J}}\beta\,,$		(4)
	$\displaystyle\sigma_{I}$	$\displaystyle=$	$\displaystyle\sum_{i\in I}\sum_{\begin{subarray}{c}i^{\prime}\in I\\ i^{\prime}>i\end{subarray}}|X_{i}||X_{i^{\prime}}|=\frac{1}{2}\left(|X_{I}|^{2}-\sum_{i\in I}|X_{i}|^{2}\right)\,.$		(5)

The corresponding values for weighted $\beta$-flexible clustering are:

	$\displaystyle\alpha_{ij}$	$\displaystyle=$	$\displaystyle\frac{1}{|I||J|}(1-\beta)\,,$		(6)
	$\displaystyle\beta_{ii^{\prime}}$	$\displaystyle=$	$\displaystyle\frac{1}{\sigma_{I}+\sigma_{J}}\beta\,,$		(7)
	$\displaystyle\sigma_{I}$	$\displaystyle=$	$\displaystyle\frac{|I|\left(|I|-1\right)}{2}=\frac{|I|^{2}-|I|}{2}\,,$		(8)

where $|I|$ and $|J|$ are the number of subclusters contained in clusters $X_{I}$ and $X_{J}$, respectively. These formulas derive from the unweighted ones when we take $|X_{i}|=1$, $\forall i\in I$, and $|X_{j}|=1$, $\forall j\in J$.

Arithmetic average linkage clustering iteratively forms clusters made up of previously formed subclusters, based on the arithmetic mean distances between their member individuals; for simplicity and to avoid confusion, we will denote it arithmetic linkage instead of the standard term average linkage. Substituting the arithmetic means by generalized means, also known as power means, this clustering strategy can be extended to any finite power $p\neq 0$:

	$\displaystyle D_{p}(X_{I},X_{J})=\left(\frac{1}{|X_{I}||X_{J}|}\sum_{x\in X_{I}}\sum_{y\in X_{J}}[d(x,y)]^{p}\right)^{1/p}=$				(9)
			$\displaystyle=\left(\frac{1}{|X_{I}||X_{J}|}\sum_{i\in I}\sum_{j\in J}|X_{i}||X_{j}|[D_{p}(X_{i},X_{j})]^{p}\right)^{1/p}\,.$

We call this new system of agglomerative hierarchical clustering strategies as versatile linkage. As in the case of $\beta$-flexible clustering, versatile linkage provides a way of obtaining an infinite number of clustering strategies from a single formula. The second equality in Equation 9 shows that versatile linkage can be calculated using a combinatorial formula, from the distances $D_{p}(X_{i},X_{j})$ obtained during the previous iteration, in the same way as Lance and Williams’ recurrence formula given in Equation 1.

The decision of what power $p$ to use could be taken in agreement with the type of distance employed to measure the initial dissimilarities between individuals. For instance, if the initial dissimilarities were calculated using a generalized distance of order $p$, then the natural agglomerative clustering strategy would be versatile linkage with the same power $p$. However, this procedure does not guarantee that the dendrogram obtained is the best according to other criteria, e.g., cophenetic correlation, mean absolute error, space distortion or tree balance, see Section 4. A better approach consists in scanning the whole range of parameters $p$, calculate the preferred descriptors of the corresponding dendrograms, and decide if it is better to substitute the natural parameter $p$ by another one. This is especially important when only the dissimilarities between individuals are available, without coordinates for the individuals, as is common in multidimensional scaling problems, or when the dissimilarities have not been calculated using generalized means.

We have selected four case studies, drawn from the UCI Machine Learning Repository [15], for a descriptive analysis of several agglomerative hierarchical clustering strategies. Table 2 summarizes the main characteristics of these datasets. The values of the variables in these datasets show different orders of magnitude; therefore, all the variables have been scaled first, and then the corresponding dissimilarity matrices have been built using the Euclidean distance between all pairs of individuals.

For the comparison of the hierarchical clustering strategies, we have chosen the following methods: $\beta$-flexible with $\beta=+0.9$, to avoid the completely flat hierarchical trees obtained with $\beta=+1$; versatile linkage with $p\rightarrow-\infty$, i.e., SL; centroid method; versatile linkage with $p=-1$, i.e., HL; versatile linkage with $p\rightarrow 0$, i.e., GL; versatile linkage with $p=+1$, which is the same as $\beta$-flexible with $\beta=0$, i.e., AL; versatile linkage with $p\rightarrow+\infty$, i.e., CL; Ward’s minimum variance method [25]; and $\beta$-flexible with $\beta=-1$. This selection includes five variants of versatile linkage, three of them equivalent to traditional methods (SL, AL and CL) and the other two introduced in this work (HL and GL), and three variants of $\beta$-flexible clustering, one of them equivalent to AL.

Weighted and unweighted versions of the hierarchical clustering strategies have been used. Although weighting has no effect on SL and CL, we have included both of them for visual convenience in all the figures depicted next. The software used to run these experiments is MultiDendrograms [9], which from version $5.0$ implements all the hierarchical clustering strategies analyzed here and it also computes the necessary descriptive measures.

Agglomerative hierarchical clustering methods have been continually evolving since their origins back in the 1950s, and historically they have been deployed in very diverse application domains, such as geosciences, biosciences, ecology, chemistry, text mining and information retrieval, among others [19]. Nowadays, with the advent of the big data revolution, hierarchical clustering methods have had to address the new challenges brought by more recent application domains that require the hierarchical clustering of thousands of observations [20].

In this work we have introduced versatile linkage, an infinite family of agglomerative hierarchical clustering strategies based on the definition of generalized mean. We have shown that the versatile linkage family contains as particular cases not only the traditionally important strategies of single linkage, complete linkage and arithmetic linkage, but also two new clustering strategies such as geometric linkage and harmonic linkage. In addition, we have given both weighted and unweighted versions of these hierarchical clustering strategies, and we have proved the monotonicity of versatile linkage strategies, which guarantees the absence of inversions in the hierarchy. Although we have built versatile linkage upon the multidendrograms variable-group methods to ensure the uniqueness of the clustering, it may also be used with the common pair-group approach just by breaking ties randomly.

We have shown that any descriptive analysis of hierarchical trees in terms of cophenetic correlation should be complemented with the use of other measures capable of describing the space distortion that different hierarchical clustering strategies cause. Under this point of view, we have shown that it is helpful to use other measures such as the mean absolute error or the space distortion ratio. The latter, in addition, provides a way to describe numerically the increase in space distortion observed all along a system of hierarchical clustering strategies such as versatile linkage.

Space distortion is inversely proportional to clustering intensity: space-contracting clustering strategies drive systems to cluster very weakly and produce a chaining effect, while space-dilating clustering strategies drive systems to cluster with high intensity and produce very compact clusters. These differences are described by the normalized tree balance measure introduced here, which is based on Shannon’s entropy. Tree balance and space distortion are two new descriptive measures meant to be helpful to analyze and understand any hierarchical tree.

The $\beta$-flexible clustering also integrates an infinite number of agglomerative hierarchical clustering strategies into a single system, driven by a parameter $\beta$ that works as a cluster intensity coefficient. However, to the best of our knowledge, no one has rigorously defined yet a range of values of $\beta$ for which the corresponding $\beta$-flexible clustering strategies can be regarded as space-conserving. Unlike the $\beta$-flexible clustering system, we have shown that the versatile linkage family is space-conserving.

This work has been partially supported by MINECO through grant FIS2015-71582-C2-1 (S.G.), Generalitat de Catalunya project 2017-SGR-896 (S.G.), and Universitat Rovira i Virgili projects 2017PFR-URV-B2-29 (A.F.) and 2017PFR-URV-B2-41 (S.G.).