Multi-view Subspace Adaptive Learning via Autoencoder and Attention

We briefly introduce the self-representation method for training data. The meaning of data self-representation is that each data point in a union set of subspace can be effectively reconstructed by combining other points in the data set. More precisely, each data point ${{x}_{i}}$ can be expressed as:

where ${{c}_{i}}=[{{c}_{i1}},{{c}_{i2}},...,{{c}_{iN}}]$ and $N$ represents the number of samples. In addition, ${{c}_{ii}}=0$ represents a simple way of eliminating the linear combination of writing points as themselves. In this way, we can represent each data point in the data point matrix $X$ as a linear combination of other data points. Since the above problem has solution vectors of infinite number, incorporating constraint, the problem (1) is transformed into the following minimizing problem:

where different choices for$q$ have different effects in the obtained solutions, such as ${{L}_{1}}$ norm, kernel norm, and F-norm, etc. $q=1$ is used in the SSC algorithm. We can also rewrite the above problem in matrix form:

where $\textbf{{C}}=[c_{1},c_{2},\cdots,c_{N}]\in R^{N\times N}$ is a matrix of self-representation coefficients.

Our work are inspired by MLRSSC, before we dive into the details of our proposed framework, let’s briefly introduce MLRSSC based on pairwise similarity. Because this one is intendedly designed for multi-view data, it has good reference value. MLRSSC mainly concerns the similarity between matrix pairs represented by self-representation matrices. MLRSSC solves the following joint optimization problems with ${{n}_{\nu}}$ views:

where $C^{(v)}\in R^{N\times N}$ is the self-representation matrix of the view $v$. ${{\beta}_{1}}$,${{\beta}_{2}}$, and ${{\lambda}^{(v)}}$ indicate the trade-off parameters between low-rank, sparse, and consistency constraints between views, respectively. In order to solve the convex optimization problem, MLRSSC used the alternating direction multiplier method (ADMM) [14].

In the training data set, the feature dimensions in different views are different. hence we need to map examples in different views into the same dimension, so that the examples for each view has the same dimension$R$:

where${{Z}^{v}}=[z_{1}^{v},z_{2}^{v},\cdots,z_{n}^{v},\cdots,z_{N}^{v}]\in{{{R}}^{R\times N}}$. $W_{1,1}^{v}\in{{{R}}^{R\times{{F}^{v}}}},\cdots,W_{1,l}^{v}\in{{{R}}^{R\times{{F}^{v}}}}$and ${{b}_{1,1}}\in{{{R}}^{R}},\cdots,{{b}_{1,l}}\in{{{R}}^{R}}$ are the weight matrices and biases of the fully connected layer, $l$ is the number of network layers, and $f(\cdot)$ is the activation function of the fully connected layer. Here we use the ReLu as activation function.

In this layer, we need to use attention mechanism to process each view in order to achieve the fusion of the content of multiple views for each view. First, we construct a query matrix $Q$, a key matrix $K$, and a value matrix $V$ by following identities:

where${{Q}^{v}}=[q_{1}^{v},\cdots,q_{n}^{v},\cdots,q_{N}^{v}],{{K}^{v}}=[k_{1}^{v},\cdots,k_{n}^{v},\cdots,k_{N}^{v}],{{V}_{v}}=[v_{1}^{v},\cdots,v_{n}^{v},\cdots,\\ v_{N}^{v}],q_{n}^{v},k_{n}^{v},v_{n}^{v}\in{{{R}}^{R\times 1}}$, and $W_{2,1},W_{2,2},W_{2,3}\in{{{R}}^{R\times R}}$ are the linear transform matrices. Secondly, we need to calculate alignment weight $a_{i}^{v}$ for the context vector$h_{i}^{v}$ of $i$-th sample in the $v$-th view. We first define the score function:

The alignment weight is defined as follows:

By getting the alignment weight $a_{i}^{v}$ we can derive the context vector $h_{i}^{v}$:

For the subspace clustering, we need to utilize self-representation coefficient matrix ${{C}^{v}}\in{{{R}}^{N\times N}}$ and ${{H}^{v}}=h_{1}^{v},\cdots,h_{N}^{v}\in{{{R}}^{R\times N}}$to obtain representation recombination matrix$H_{sr}^{v}=[h_{sr,1}^{v},h_{sr,2}^{v},\cdots,h_{sr,n}^{v},\cdots,h_{sr,N}^{v}]\in{{{R}}^{R\times N}}$, the formula is as follows:

What we need to note is that in subspace clustering, the self-representation coefficient matrix ${{C}^{v}}$ needs to satisfy the constraint $diag({{C}^{v}})=0$.

In this layer, we utilize the self-representation $h_{sr,n}^{v}$ as input of the fully connected layer to reconstruct the original data $x_{n}^{v}$of each view:

where $\hat{x}_{n}^{v}$is reconstruction of $x_{n}^{v}$. In addition, $W_{3,1}^{v}\in{{{R}}^{{{F}^{v}}\times R}},\cdots,W_{3,1}^{v}\in{{{R}}^{{{F}^{v}}\times R}}$ and ${{b}_{2,1}}\in{{{R}}^{{{F}^{v}}}},\cdots,{{b}_{2,l}}\in{{{R}}^{{{F}^{v}}}}$ are the weight matrices and bias vectors of the fully connected layer, and $f(\cdot)$ is the activation function of the fully connected layer, which uses the ReLu as activation function.We concatenate $\hat{x}_{n}^{v}$,$n\in\left\{1,\cdots,N\right\}$ to form the matrix ${{\hat{X}}^{v}}=[\hat{x}_{1}^{v},\hat{x}_{2}^{v},\cdots,\hat{x}_{n}^{v},\cdots,\hat{x}_{N}^{v}]\in{{{R}}^{{{F}^{v}}\times N}}$.

This loss function can be divided into two parts. The first part is related to the encoder and decoder:

where $\Omega(\cdot)$ is a regularization term, and its role is to constrain the parameters in the encoder and decoder. In this paper, $\Omega(\cdot)$mainly have two forms, one is ${{L}_{1}}$ norm regularization term and the other is ${{L}_{2}}$ norm regularization term, and ${{\beta}_{2}}$ is a trade-off parameter. The other part is related to the self-representation of the subspace.

Here we first impose the alignment constraint for the self-representation matrices of multiple views in the form of $\sum\limits_{1\leq v,w\leq M,v\neq w}{\frac{1}{2N}\left\|{{C}^{v}}-{{C}^{w}}\right\|_{F}^{2}}$, by imposing the alignment constraint, the information of each view and other views can be fused with each other, so that the multi-views are complementary. We introduce the F norm for the self-representation coefficient matrix ${{C}^{v}}$. In the subspace self-representation learning task, we can apply different regularization constraints on the self-representation coefficient matrix ${{C}^{v}}$, for example, ${{L}_{1}}$ norm regularization term, kernel norm Constraints, and the F-norm constraints witch we use. In addition, ${{\beta}_{1}}$ is a trade-off parameter. Due to the use of deep autoencoder architecture, an additional benefit is that we do not need to resort complex optimization approaches, such as the ADMM algorithm for iterative updates, which is used in SSC and LRSC. We can directly update the self-representation coefficient matrix ${{C}^{v}}$ by the off-the-shelf Stochastic Gradient Descent (SGD) approaches in Tensorflow.

To explore the performance of our proposed network, we have performed comparative experiments on six datasets, which are ORL, Reuters, 3-sources, Yale, UCI-digit, and Prokaryotic dataset. The characteristics of six datasets are summarized in Table 1. Where $M,C,N$ and $F^{v}$ represents the number of views, the sample category, the number of samples, and the feature dimensions of each view, respectively.

To demonstrate the efficiency of our proposed MSALAA, some state-of-the-art subspace clustering methods are chosen as the baseline methods: SSC, LRSC, LMSC [15], CSMSC, and MLRSSC and its three improved variants (MLRSSC-Centroid, KMLRSSC, KMLRSSC-Centroid).

In our experiments, we only need to set the number of layers of the network and the number of neurons in each layer, and determine whether to perform batch normalization. In this article, all our datasets are initialized with the LeCun normal distribution. We choose Adam optimization algorithm as optimizer, the learning rate is 0.001, and the decay value of learning rate is 0.99. In addition, we set the trade-off parameters ${{\beta}_{1}}$ and ${{\beta}_{2}}$ to 0.1. In the experiment, we will get the self-representation matrix of multiple views. Here we will use the self-representation matrix${{C}^{v}}$of the best experimental view to construct the affinity matrix${{A}^{v}}=\left|{{C}^{v}}\right|+{{\left|{{C}^{v}}\right|}^{T}}$for spectral clustering. In addition, we performed residual processing on the network to prevent the gradient from disappearing.

We perform spectral clustering with MSALAA and six baseline methods on six data sets. In order to verify the performance of our method, we selected six metric criteria to measure the effect of spectral clustering: Accuracy (ACC), Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), Precision, Recall and F-score. From the experimental results in Table 2, we can clearly see that our method has obvious advantages over other baseline comparison methods, validate that MSALAA can find better data self-representation. On the data set UCI-digit, MSALAA can reach more than 90% on multiple metric criteria.

In addition, we also visualize the matrix ${{H}^{v}}$generated by the attention mechanism. Here we use t-SNE. In the visualization experiment, we use t-SNE to embed feature matrix${{H}^{v}}$into a 2D latent feature matrix for clustering. We use t-SNE to derive a 2D latent feature matrix and depict it with a scatter plot. As shown in Fig. 2, this process is performed on the data set UCI-digit, and we can clearly see that each clustering can be easily distinguished.