Spectral Clustering with Smooth Tiny Clusters

Spectral clustering is a new and hot topic in recent years [1], and many researchers have focus on improving spectral cluster’s performance. Spectral clustering is a set of algorithms which use graph partitioning methods to cluster objects. These set of algorithms have been widely used in many tasks, such as text mining [2] and network analysis [3, 4, 5] and even medical image analysis [6, 7, 8]. Typically, spectral clustering algorithms use a distance-based similarity matrix to indicate the similarity among objects, and this works well in many cases. But this considers only the feature similarity, which is quite straightforward.

Researchers have found that spectral methods can perform really poor when used on multi-scale data, whose clusters various a lot in sizes and densities. Because the standard of similarity varies a lot, for sparse clusters, distances among objects are always much larger than distances among objects from dense clusters. Typically, existing solutions either scale the similarity matrix or apply the power iteration technique to derive pseudo-eigenvectors with rich cluster separation information. Among all the methods focusing on multi-scale data problem, the ROSC [9] and CAST [10] algorithms are state of the art. ROCS and CAST handle multi-scale data by extracting separation information from pseudo-eigenvectors and tuning the given traditional distance-based similarity matrix to a coefficient matrix, considering reachability similarity.

In this paper, we propose a new algorithm, namely Smooth-Clustering. Smooth-Clustering is the first algorithm that considers the smoothness of data for clustering problems. As mentioned above, spectral clustering is actually a graph partition problem, which means that the distribution of objects tend to be smooth, and this is quite consistent with real-world scenarios [11]. If there exists a sudden change among different objects, they belong to different groups with high probability. We call this data smoothness in the rest of our paper.

In fact, data smoothness is quite difficult to define because under extremely small scale, all data will seems discrete. To address this problem, first, we divide objects into a great many tiny clusters. Specifically, we use traditional cluster algorithms to cluster extremely close objects into one group and treated them as a tiny cluster. In the rest of the clustering process, we will use the center of the tiny cluster as an object. In this way, no objects will be extremely close. Second, we cluster tiny clusters, whose centers constitute smooth graphs. Specifically, we add an extra penalty term to embedding smoothness information when getting the coefficient matrix.

In this way, we can take data smoothness into account when dealing with the multi-scale data problem. For objects from different clusters, they may happen to be close or even reachable to each other, under which situation no previous algorithms can work well. However, for objects from different algorithms, their feature distribution will always vary a lot and will be in a discrete manner. When considering data smoothness, this problem can be easily fixed.

In this paper, we present a concrete spectral cluster algorithm, which considers feature similarity, reachability similarity, and data smoothness, focusing on multi-scale data. But we notice that the performance of most of the existing spectral clustering method is always affected by randomness. However, when clustering, we can hardly judge the performance of one clustering by any metrics for the lack of labels. Therefore, our idea of data smoothness can be definitely applied to any spectral cluster algorithms to achieve better performance.

Our main contributions are as follows:

• •

  This is the first work that considers smoothness of tiny clusters in spectral clustering.

• •

  We propose a new algorithm embedding smoothness information into spectral clustering algorithms.

• •

  We derived mathematical proofs to show our algorithm can work well for multi-scale data.

• •

  We conduct extensive experiments to show the effectiveness of our algorithm by comparing its performance with more than 10 previous clustering algorithms. The experimental results show that our algorithm provides high performance on different datasets. We also conduct experiments on synthetic datasets to show that our algorithm is efficient when data smoothness is related to clustering result. These experimental results will be provided in the full version of this paper.

• •

  We will show the experimental results of applying the idea of data smoothness to many more clustering algorithms.

One of the typical methods for handling multi-scale data in spectral clustering is scaling the similar matrix $S$ [44, 45]. Self-tuning spectral clustering (ZP method) [40] uses the Gaussian kernel based similarity $S_{ij}=\exp\left(-\frac{\left\|x_{i}-x_{j}\right\|^{2}}{2\sigma^{2}}\right)$ to $S_{ij}=$ $\exp\left(-\frac{\left\|x_{i}-x_{j}\right\|^{2}}{\sigma_{i}\sigma_{j}}\right),$ where $x$ is the feature vector of an object, and $\sigma$ is a scaling parameter which is always difficult to set. Instead of a hyper-parameter $\sigma$, ZP introduces a local scaling parameter $\sigma_{i}$ for each object $x_{i}$. The local scaling parameter is set as the distance from $x_{i}$ to its $l$ -th nearest neighbor (l is a hyper-parameter). Under this design, objects in a sparse cluster will have large $\sigma$ and dense cluster’s $\sigma$ will be relatively small.

Spectral clustering algorithms always try to find eigenvectors with the richest cluster seperation information [46, 47]. For most spectral clustering algorithms, the smallest eigenvectors are usually treated as the most informative ones. But in fact, researchers have already find that some of these smallest eigenvectors might contain salient noise in the data and might not contain good cluster separation information [48]. Under this circumstances, researches focusing on how to select eigenvectors with the richest cluster separation information have been raised.

The most widely used methods to generate pseudo-eigenvectors is the power iteration (PI) technique. The PI method is used to derive the dominant eigenvector of a matrix and can thus using to enhance spectral clustering.

Given a matrix $W,$ PI will first generate a random vector $v_{0}\neq 0$ and iterates as below:

$$v_{t+1}=\frac{Wv_{t}}{\left\|Wv_{t}\right\|_{1}},\quad t\geq 0$$

For simplicity, we assume that $W$ has eigenvalues $\lambda_{1}>\lambda_{2}>\ldots>\lambda_{n}$ with corresponding eigenvectors $e_{1},e_{2},\ldots,e_{n}.$ We express $v_{0}$ as

$$v_{0}=c_{1}e_{1}+c_{2}e_{2}+\ldots+c_{n}e_{n}$$

with parameters $c_{1}\ldots,c_{n}.$ Let $R=\prod_{i=0}^{t-1}\left\|Wv_{i}\right\|_{1},$ we have,

$$v_{t}=W^{t}v_{0}/R=\left(c_{1}W^{t}e_{1}+c_{2}W^{t}e_{2}+\ldots+c_{n}W^{t}e_{n}\right)/R$$

$$=\left(c_{1}\lambda_{1}^{t}e_{1}+c_{2}\lambda_{2}^{t}e_{2}+\ldots+c_{n}\lambda_{n}^{t}e_{n}\right)/R$$

$$=\frac{c_{1}\lambda_{1}^{t}}{R}\left[e_{1}+\frac{c_{2}}{c_{1}}\left(\frac{\lambda_{2}}{\lambda_{1}}\right)^{t}e_{2}+\ldots+\frac{c_{n}}{c_{1}}\left(\frac{\lambda_{n}}{\lambda_{1}}\right)^{t}e_{n}\right]$$

We can see that $v_{t}$ is thus a linear combination of the eigenvectors. It is obviously that when $c_{1}\neq 0$, $v_{t}$ converges to the scaled dominant eigenvector $e_{1}$.

By truncating the iterative PI process, we can obtain an intermediate pseudo-eigenvector, an eigenvectors’ linear combination, and thus containing rich cluster separation information. With PI technique, researchers have proposed the Power Iteration Clustering (PIC) method [48]. However, each object has only one feature value corresponding to the lone pseudo-eigenvector in PIC, and this is far from enough if we have a large clusters number as the cluster collision problem [49] may happen. Under this circumstance, the PIC- $k$ method has been further proposed to fix this issue, which directly runs PI multiple times to generate more pseudo-eigenvectors for clustering.

The most weakness of the $\mathrm{PIC}-k$ method is that the pseudo-eigenvectors could be similar becuase they are not strictly orthogonal, which will leads to redundancy. In order to reduce this redundancy, Deflation -based Power Iteration Clustering (DPIC) method [50] has been proposed to use Schur complement to derive orthogonal pseudo-eigenvectors. Besides, the PIC based methods always ignore lesser but necessary eigenvectors because dominant eigenvectors are assigned larger weights during the PI process. To figure this out, the Diverse Power Iteration Embedding (DPIE) method [51] has been proposed. In this algorithm, When we generate a new pseudo-eigenvector, previously pseudo-eigenvectors’ information will be removed from the new one.

TKNN graph is always used to regularize the coefficient matrix $Z$. The main purpose of TKNN graph is to capture the high correlations between objects. We hope that even for objects belong to the same cluster but located at distant far ends of the cluster, their correlations can still be captured.

For an object $x_{p},$ we use $z_{p}$ to denote the $p$ -th column of matrix $Z$. We know from previous works [52, 53, 54] that if $Z$ has grouping effect, then using $Z$ for spectral clustering will have an excellent performance.

The main idea of ROSC is to modify the traditional distance-based similarity matrix $S$ to a coefficient matrix $Z$ and then perform spectral clustering based on $Z$. The similarity between two objects can be treated as how much common characteristics they share. If two objects share a lot common characteristics, they are more likely to be represented by each other. Therefore, it is quite nature to determine a cofficient matrix $Z$ by

$$X=XZ$$

(1)

To construct $Z,$ ROSC first applies PI multiple times to generate $p$ pseudo-eigenvectors, which form$X\in\mathcal{R}^{p\times n}$. The $q$ -th column of the matrix $X$ is regarded as the feature vector $x_{q}$ of an object $x_{q}.$ The pseudo-eigenvectors can be treated as low dimensional embeddings of objects, which contains rich separation information. We know from previous works that the generated $p$ pseudo-eigenvectors are always noisy so that ROSC represents each object by others with

$$X=XZ+O$$

(2)

where $O\in\mathbb{R}^{p\times n}$ is the matrix used to represent the noise in the pseudo-eigenvectors.

ROSC derived the matrix $Z$ by optimizing the objective function

$$\min_{Z}\|X-XZ\|_{F}^{2}+\alpha_{1}\|Z\|_{F}^{2}+\alpha_{2}\|Z-\mathcal{W}\|_{F}^{2}$$

(3)

where the first term is to reduce the noise $O$, the second term is the Frobenius norm of $Z$ acting as a penalty term to guarantee the sparsity of $Z$ and the third term regularizes $Z$ by the TKNN graph, which means if two objects are reachable to each other, it will be treated as similar. $\alpha_{1}>0,\alpha_{2}\geq 0$ are two factors using to indicate the relative weights of these three terms. We can easily derive the optimal solution to this problem and this has been proved to have grouping effect so that ROSC can work in multi-scale data:

$$Z^{*}=\left(X^{T}X+\alpha_{1}I+\alpha_{2}I\right)^{-1}\left(X^{T}X+\alpha_{2}\mathcal{W}\right)$$

(4)

ROSC only express the high correlation among objects from the same cluster. On the other hand, suppressing the correlation among objects from different clusters is also important for correct clustering. CAST studied methods to consider both factors. The key idea of CAST is applying trace Lasso [55] to regularize the coefficient matrix. Researchers have proved that when regularized by trace lasso, the coefficient matrix exhibits "sparsity". Sparsity is the desired property that entries in the matrix corresponding to inter-cluster object pairs should be 0 or very small, thus suppressing the correlation among objects from different clusters. The optimization problem of CAST is as below:

$$\min_{z}\frac{1}{2}\|x-Xz\|_{2}^{2}+\alpha_{1}\|X\operatorname{Diag}(z)\|_{*}+\frac{\alpha_{2}}{2}\|z-w\|_{2}^{2}$$

(5)

where the second term of equation 5 is:

$$\|X\operatorname{Diag}(z)\|_{*}=\sum_{q=1}^{n}\left\|x_{q}\right\|_{2}\left|z_{q}\right|=\sum_{q=1}^{n}\left|z_{q}\right|=\|z\|_{1}$$

(6)

Although ROSC and CAST use coefficient matrix $Z$ for spectral clustering, they ignore that in most cases, data always presents in a smooth manner. In other words, some objects may be reachable to each other but with sudden direction changes, in this case they are likely to belong to different clusters.

So in our algorithm, we transform the problem into:

$$\min_{Z}\|X-XZ\|_{F}^{2}+\alpha_{1}\|Z\|_{F}^{2}+\alpha_{2}\|Z-\mathcal{W}\|_{F}^{2}+\alpha_{3}\|Z-\mathcal{W}*\mathcal{W}+\alpha_{4}*\mathcal{W}\|_{F}^{2}$$

(7)

All the notions and the first three terms above are exactly the same with those in ROSC. The forth term is a penalty term on sudden direction changes. $\mathcal{W}*\mathcal{W}$ is the second reachable matrix for objects,indicating the path numbers between objects. Intuitively, if there are many paths between two objects, then they are likely to be smooth paths between objects. As we hope this can work for multi-scale data, so a hyper-parameter $\alpha_{4}$ is used here to indicate the standard of "many paths".

We can easily derive the optimal solution, $Z^{*}$, to the optimization problem:

$$Z^{*}=\left(X^{T}X+\alpha_{1}I+\alpha_{2}I+\alpha_{3}I\right)^{-1}\left(X^{T}X+(\alpha_{2}-\alpha_{3}*\alpha_{4})\mathcal{W}+\alpha_{3}\mathcal{W}*\mathcal{W}\right)$$

(8)

LEMMA 1. Given a set of objects $\mathcal{X}$, the matrix $X\in\mathcal{R}^{p\times n}$, whose rows are pseudo-eigenvectors, the reachability matrix $\mathcal{W}$, then we have the the optimal soution $Z^{*}$ to Equation 6

$$Z_{ip}^{*}=\frac{x_{i}^{T}\left(x_{p}-Xz_{p}^{*}\right)+(\alpha_{2}-\alpha_{3}*\alpha_{4})\mathcal{W}_{ip}+\alpha_{3}\mathcal{WW}_{ip}}{\alpha_{1}+\alpha_{2}+\alpha_{3}},\quad\forall 1\leq i,p\leq n,$$

(9)

where $\mathcal{WW}=\mathcal{W}*\mathcal{W}$

PROOF

For $1\leq p\leq n,$ let

$$J\left(z_{p}\right)=\left\|x_{p}-Xz_{p}\right\|_{2}^{2}+\alpha_{1}\left\|z_{p}\right\|_{2}^{2}+\alpha_{2}\left\|z_{p}-w_{p}\right\|_{2}^{2}+\alpha_{3}\left\|z_{p}-ww_{p}+\alpha_{4}*w_{p}\right\|_{2}^{2}.$$

(10)

As $Z^{*}$ is the optimal solution to Equation $6,$ we have

$$\left.\frac{\partial J}{\partial Z_{ip}}\right|_{z_{p}=z_{p}^{*}}=0,\quad\forall 1\leq i\leq n.$$

(11)

Hence

$$-2x_{i}^{T}\left(x_{p}-Xz_{p}^{*}\right)+2\alpha_{1}Z_{ip}^{*}+2\alpha_{2}\left(Z_{ip}^{*}-\mathcal{W}_{ip}\right)+2\alpha_{3}\left(Z_{ip}^{*}-\mathcal{WW}_{ip}+\alpha_{4}\mathcal{W}_{ip}\right)=0$$

(12)

, which induces Equation 7.

LEMMA 2. $\forall 1\leq i,j,p\leq n$

$$\left|Z_{ip}^{*}-Z_{jp}^{*}\right|\leq\frac{c\sqrt{2(1-r)}+\left|\alpha_{2}-\alpha_{3}*\alpha_{4}\right|\left|W_{ip}-\mathcal{W}_{jp}\right|+\alpha_{3}\left|\mathcal{WW}_{ip}-\mathcal{WW}_{jp}\right|}{\alpha_{1}+\alpha_{2}+\alpha_{3}}$$

(13)

where $c=\sqrt{1+(\alpha_{2}+\alpha_{3}*\alpha_{4})\left\|w_{p}\right\|_{2}^{2}+\alpha_{3}\left\|ww_{p}\right\|_{2}^{2}}$ and $r=x_{i}^{T}x_{j}$

PROOF

From Equation 7, we have

$$Z_{ip}^{*}-Z_{jp}^{*}=\frac{\left(x_{i}^{T}-x_{j}^{T}\right)\left(x_{p}-Xz_{p}^{*}\right)+(\alpha_{2}-\alpha_{3}*\alpha_{4})\left(\mathcal{W}_{ip}-\mathcal{W}_{jp}\right)+\alpha_{3}\left(\mathcal{WW}_{ip}-\mathcal{WW}_{jp}\right)}{\alpha_{1}+\alpha_{2}+\alpha_{3}}$$

(14)

That implies

	$\displaystyle\left|Z_{ip}^{*}-Z_{jp}^{*}\right|$	$\displaystyle\leq\frac{\left|\left(x_{i}^{T}-x_{j}^{T}\right)\left(x_{p}-Xz_{p}^{*}\right)\right|+\left|\alpha_{2}-\alpha_{3}*\alpha_{4}\right|\left|\mathcal{W}_{ip}-\mathcal{W}_{jp}\right|+\alpha_{3}\left|\mathcal{WW}_{ip}-\mathcal{WW}_{jp}\right|}{\alpha_{1}+\alpha_{2}+\alpha_{3}}$
		$\displaystyle\leq\frac{\|x_{i}-x_{j}||_{2}||x_{p}-Xz_{p}^{*}||_{2}+\left|\alpha_{2}-\alpha_{3}*\alpha_{4}\right|\left|W_{ip}-\mathcal{W}_{jp}\right|+\alpha_{3}\left|\mathcal{WW}_{ip}-\mathcal{WW}_{jp}\right|}{\alpha_{1}+\alpha_{2}+\alpha_{3}}$

As the column vectors of $X$ are normalized vectors (i.e., $x_{q}^{T}x_{q}=1$, $\forall 1\leq q\leq n),$ we have $\left\|x_{i}-x_{j}\right\|_{2}=\sqrt{2(1-r)},$ where $r=x_{i}^{T}x_{j}$. As $Z^{*}$ is the optimal solution of Equation 5, we have

	$\displaystyle J\left(z_{p}^{*}\right)$	$\displaystyle=\left\|x_{p}-Xz_{p}^{*}\right\|_{2}^{2}+\alpha_{1}\left\|z_{p}^{*}\right\|_{2}^{2}+\alpha_{2}\left\|z_{p}^{*}-w_{p}\right\|_{2}^{2}+\alpha_{3}\left\|z_{p}-ww_{p}+\alpha_{4}*w_{p}\right\|_{2}^{2}$
	$\displaystyle\leq J(0)$	$\displaystyle=\left\|x_{p}\right\|_{2}^{2}+(\alpha_{2}+\alpha_{3}*\alpha_{4})\left\|w_{p}\right\|_{2}^{2}+\alpha_{3}\left\|ww_{p}\right\|_{2}^{2}=1+(\alpha_{2}+\alpha_{3}*\alpha_{4})\left\|w_{p}\right\|_{2}^{2}+\alpha_{3}\left\|ww_{p}\right\|_{2}^{2}$

Hence, $\left\|x_{p}-Xz_{p}^{*}\right\|_{2}\leq\sqrt{1+(\alpha_{2}+\alpha_{3}*\alpha_{4})\left\|w_{p}\right\|_{2}^{2}+\alpha_{3}\left\|ww_{p}\right\|_{2}^{2}}=c.$

Then we get

$$\left|Z_{ip}^{*}-Z_{jp}^{*}\right|\leq\frac{c\sqrt{2(1-r)}+\left|\alpha_{2}-\alpha_{3}*\alpha_{4}\right|\left|W_{ip}-\mathcal{W}_{jp}\right|+\alpha_{3}\left|\mathcal{WW}_{ip}-\mathcal{WW}_{jp}\right|}{\alpha_{1}+\alpha_{2}+\alpha_{3}}$$

LEMMA 3. Matrix $Z^{*}$ in our algorithm has grouping effect.

PROOF Given two objects $x_{i}$ and $x_{j}$ such that $x_{i}\rightarrow x_{j},$ we have (1)$x_{i}^{T}x_{j}\rightarrow 1$ and $(2)\left\|w_{i}-w_{j}\right\|_{2}\rightarrow 0.$ These imply $r=x_{i}^{T}x_{j}\rightarrow 1$ and $\left|\mathcal{W}_{ip}-\mathcal{W}_{jp}\right|\rightarrow 0$, what’s more, $\left|\mathcal{WW}_{ip}-\mathcal{WW}_{jp}\right|\leq\left\|w_{i}-w_{j}\right\|_{2}*\left\|w_{p}\right\|_{2}\rightarrow 0$ Hence, the three terms of the numerator of Equation 11 are close to 0 Therefore, $\left|Z_{ip}^{*}-Z_{jp}^{*}\right|\rightarrow 0$ and thus $Z^{*}$ has grouping effect.

We show that our algorithm also improves the performance of spectral clustering on multi-scale data. While traditional approaches focus on feature similarity and state of the art use integrated similarity of both feature and reachability, our approach further considers data smoothness without destroying the grouping effect. Two distant objects $x_{i}$ and $x_{j}$ belong to a same cluster may not share a strong feature similarity or even strong reachability similarity, but present a sudden change in some elements of the feature vector. This leads to a small $r$, and traditional approaches will likely separate them into different clusters. What’s more, ROSC and CAST may also poorly performed here because such a sudden change may still preserve reachability similarity. On the contrary, our algorithm considers the smoothness of the objects’ present manner and thus keeping them in the same cluster.