One-Step Late Fusion Multi-view Clustering with Compressed Subspace

The $k$-means algorithm, as a classical and widely used clustering algorithm, provides an intuitive and effective method for cluster analysis. Denote data matrix $\mathbf{X}\in\mathbb{R}^{n\times d}$, of which each line represents an element from the set of data samples $\left\{\mathbf{x}_{i}\right\}_{i=1}^{n}\subseteq\mathcal{X}$. Denote $\mathbf{F}\in\mathbb{R}^{n\times\mu}$ as cluster indicating matrix and $\mu$ represents total number of clusters. $\mathbf{F}_{i,j}=\frac{1}{\sqrt{|C_{j}|}}$ $\Longleftrightarrow$ $x_{i}$ belongs to $j$-th cluster, or it equals $0$. Discrete $k$-means methods can be expressed as:

To optimize the above target function concerns NP-hard problem. Thereby traditional $k$-means clustering algorithms relax the discrete constraints on $\mathbf{F}$ matrix to orthogonal constraints. The terms we use in this paper are listed in Table1.

For datasets that are not linearly separable, kernel mappings are utilized to perform kernel $k$-means algorithms[1, 2], which can be easily adopted in multi-view clustering tasks[3, 4, 5]. Denote $\phi_{i}:\mathcal{X}\rightarrow\mathcal{H}_{i}$ as the $i$-th feature mapping from $\left\{\mathbf{x}_{i}\right\}_{i=1}^{n}$ to $p$ Regenerative Kernel Hilbert Space (RKHS) $\left\{{\mathcal{H}}_{i}\right\}_{i=1}^{p}$. Each sample in multiple kernel clustering is denoted as $\phi_{\boldsymbol{\beta}}(\mathbf{x})=\left[\beta_{1}\phi_{1}(\mathbf{x})^{\top},\cdots,\beta_{p}\phi_{p}(\mathbf{x})^{\top}\right]^{\top}$, where ${\boldsymbol{\beta}}$ is the coefficient vector of $p$ base kernel functions. The kernel weights are adjusted during the clustering process to optimize the clustering performance. Kernel function can be expressed by: $\kappa_{\boldsymbol{\beta}}\left(\mathbf{x}_{i},\mathbf{x}_{j}\right)=\phi_{\boldsymbol{\beta}}\left(\mathbf{x}_{i}\right)^{\top}\phi_{\boldsymbol{\beta}}\left(\mathbf{x}_{j}\right)$, and the corresponding loss function of Multiple Kernel K-means (MKKM) is:

Optimizing the objective can be reduced to a traditional $k$-means process and solving a quadratic programming problem through alternately optimizing $\boldsymbol{\beta}$ and $\mathbf{H}$.

There are plenty of work emerged to improve MKKM, e.g., linear fusion-based methods[6] assumes that each base kernel function can correspond to a view of different dimensions, which in turn captures complementary aspects of features, and the multi-view consistent kernel matrices can be obtained by linear combination of multiple base kernel matrices. Secondly, joint training-based methods[7, 8] assumes that the clustering results can be obtained independently based on each base kernel matrix, and that the multi-view clustering results should be consistent, so the clustering information can be fused by statistical methods to further enhance the credibility of the clustering results. Thirdly, consensus structure extraction methods[9, 10] assumes that different kernel functions acting on the data matrix can preserve and extract the consistent clustering structure existing in multiple types of data, and the consistent clustering structure of multiple kernels is obtained by decomposing the kernel matrix. There are also deep learning based methods adopted to improve clustering performance and learn representations[11, 12, 13, 14, 15, 16, 17, 18]. To improve the efficiency and to better represent the sample distributions of individual views, late fusion based multi-kernel clustering methods[19, 20] use a tighter approach (base partitioning matrix $\mathbf{H}\in\mathbb{R}^{k\times n}$ extracted using $k$-means algorithm) for structural representation rather than using multiple base kernels.

We propose a One-Step Late Fusion Multi-view Clustering with Compressed Subspace method to directly obtain discrete cluster label by integrating clustering structure optimization and label learning into a unified framework. Our proposed algorithm has the following advantages:

• •

  Our algorithm is able to obtain clustering labels in one step, by negotiating label learning and cluster structure optimization through a unified framework.

• •

  The method is highly efficient with both $\mathcal{O}(n)$ time and space expenditure, which allows our algorithm to be used directly on large-scale multi-view datasets.

• •

  We propose a six-step iterative optimization algorithm with fast convergence of the target. We conduct experiments to verify the effectiveness and efficiency of the algorithm.

Most existing multi-kernel clustering methods assume that the optimal kernel exists in a linear space consisting of base kernels[6], and such assumption greatly limits the feasible domain of the optimal kernel. ONKC methods[21] propose to reconstruct the kernel matrices in a nonlinear neighborhood space, thus enlarging the search space of the optimal kernel, but the computational overheads of such methods are $\mathcal{O}(n^{3})$, which restrict the algorithms’ application to large-scale datasets. LFMVC methods[22, 20] reduce the dimensionality of the kernel matrix by constructing the corresponding base partition matrices, thus reducing the computational overhead. These methods usually use the average kernel as a reference for the alignment of partitions, and requires higher quality of these base partition matrices. These drawbacks limit their generalizability to the clustering task over wide range of datasets.

In addition, the above methods cannot directly obtain cluster labels and require an extra spectral clustering or $k$-means process for ultimate clusters. To tackle the NP-hard problem brought by discrete cluster labels, spectral rotation (SR)[23, 24, 25] and improved spectral rotation (ISR)[26] methods are proposed which learn the discrete labels and representations synchronically. However, the $\mathcal{O}(n^{3})$ time cost and $\mathcal{O}(n^{2})$ space cost greatly inhibits the scalability for large-scale datasets.

We adopt kernel subspace clustering method[27] for self-reconstruction of the consensus kernel partition, and use the trace alignment rather than Frobenius norm to avoid a re-weighing procedure[28]. We additionally adopt a compressed subspace using a uniform matrix $\mathbf{P}$ to further increase the computational efficiency. It is proved by[29] that maximizing late-fusion alignment is equivalent to minimizing the target function of MKKM in Eq.2. By integrating late fusion of kernel partition alignment maximization and self-reconstruction through a shared reconstruction matrix $\mathbf{S}$, our method is capable of optimizing cluster structure with the negotiation of multiple views. A cluster label assigning matrix $\mathbf{Y}$ is learned with the centroid $\mathbf{C}$ to refine the aligned partition $\mathbf{H}$. The overall optimization target is listed below:

where $\boldsymbol{1}_{n}\in\mathbb{R}^{n}$ is an all-$1$ column vector.

We develop a six-step iterative optimization algorithm to maximize the clustering target 3.

Likewise, the deduction for the optimization of the compress matrix $\mathbf{P}$ is similar to Eq. (4). Denote $\mathbf{A}=\mathbf{H}^{\top}\mathbf{H}\mathbf{S}=\mathbf{U}_{a}\mathbf{D}_{a}\mathbf{V}_{a}^{\top}$, the optimal solution is:

In this section, we conduct a series of experiments to evaluate the effectiveness and efficiency of our proposed method.

In this article, we propose an effective and efficient multi-view clustering method OS-LFMVC-CS, which simultaneously maximizes the alignment between different views and learns the shared subspace, and directly optimizes the cluster assignments. This mechanism greatly enhances the negotiation among views and improves the consistency in shared subspace. In this way, we obtain the clustering results in one step and reduce the time and space expenditure to linear cost. We derive a novel optimization framework using a six-step iterative optimization with verified convergence. In addition, extensive experiments are conducted to support the method well. In the future, we will explore more efficient clustering algorithms and use one-step clustering scheme for easy application on a wide range of scenarios.