Pseudo-supervised Deep Subspace Clustering

Existing deep clustering methods [27, 52, 53] mainly aim to combine deep feature learning with traditional clustering methods. AE [54] is a popular feature learning architecture for unsupervised tasks. It compresses data $X\in\mathcal{R}^{d\times n}$ into a low-dimensional representation $Z\in\mathcal{R}^{m\times n}$ where $m\ll d$, reconstructing the original data. It is often composed of an encoder $F$ and a decoder $G$ with a mirror construction. Their parameters are denoted as $\Theta_{e}$ and $\Theta_{d}$, respectively. Specifically, an AE can be optimized by the following objective function:

For the convenience of notation, we let $\hat{X_{i}}=G_{\Theta_{d}}(F_{\Theta_{e}}(X_{i}))$. Although this basic model has led to far-reaching success for data representation, it forces the reconstruction of its input without considering other data points in data, resulting in information loss and consequently degrades clustering performance [41]. Many methods are proposed to minimize the loss caused by traditional clustering methods to regularize the learning of the latent representation of an AE. In deep manifold clustering [55], the authors interpret the locality of a manifold as similar inputs that should have similar representations and minimize the reconstruction error of $X_{i}$ itself and its local neighborhood. However, they define reconstruction weights either in a supervised or pre-defined way, and the relations between samples are not flexible for modeling. To get effective representations, other authors manage to keep locality property on the latent space. The deep embedding network [32] combines AE and $k$-means while maintaining locality-preserving and group sparsity constraints on latent representations. Two representations are enforced to be similar if their corresponding raw samples are similar, i.e., the graph manifold assumption. The deep embedded clustering (DEC) [27] method fine-tunes the encoder by minimizing KL divergence between soft assignment and target distribution.

To take advantage of the effective self-expression property in traditional subspace clustering, DSC [46] implements the subspace clustering method in DNNs by adding a novel self-expressive layer between the encoder and decoder, achieving promising results. More recently, based on the framework of [46], a deep adversarial network with a subspace-specific generator and a subspace-specific discriminator are adopted for subspace clustering in [47]. However, the discriminator needs to use the dimension of each subspace, which is usually unknown in practice. By introducing multiple AEs, DSC can be used on multi-view data [50]. Self-paced learning mechanism is also introduced into DSC to boost its generalization ability in [56]. All these methods use the self-reconstruction loss and ignore the local structure information.

The main challenge for clustering is the lack of labels. Some recent efforts manage to boost clustering performance from two concepts: self-supervision and pseudo-supervision. Self-supervised learning [57, 58] generally needs to design a pretext task, where a target objective can be computed without supervision. It assumes that the learned representations of the pretext task contain high-level semantic information that is useful for solving downstream tasks of interest, such as image classification. Therefore, the success of self-supervision highly depends on the selected pretext. Some popular pretexts examples include predicting the colorization, predicting unpainted patches, and predicting the spatial relationship of different patches and adversarial objective [58, 59].

Pseudo-supervision trains a model using pseudo-labels rather than ground truth labels [51]. In contrast to self-supervision, it does not use any proxy. Pseudo-labeling is a more recent self-training algorithm focusing on image classification [60]. Pseudo-labeling trains a model using labeled data simultaneously with unlabeled data. The corresponding predictions on unlabeled samples are referred to as pseudo-labels [51]. Recent research shows that it is possible to conduct clustering with the help of pseudo-labels. Deep Cluster [33] uses the pseudo-label computed by $k$-means on output features as supervision to guide the training of DNNs. The deep comprehensive correlation mining (DCCM) [61] uses the local robustness assumption and uses pseudo-graphs and pseudo-labels to learn better representations. The self-supervised convolutional subspace clustering [62] introduces a spectral clustering module and a classification module into DSC, i.e., using the clustering results to supervise the training of a network. In practice, the main challenge of pseudo-supervision stems from the fact that some self-acquired labels are unreliable and mismatch the real labels. This can mislead data grouping by learning non-representative features, decreasing the discriminative ability of a model. This observation is verified in prior publications [51, 33].

In this paper, we propose a joint optimization framework by combining DSC and pseudo-supervised learning. In this way, useful pseudo-supervision information from latent representation and self-expression can be used to guide feature and similarity learning, refining pseudo-information.

We will introduce the first two modules. In the next subsection, we focus on the pseudo-supervision module.

Locality Preserving Module. To preserve the local structure, we use a weighted reconstruction for an AE instead of Eq.(1). More specifically, $X_{i}$ is reconstructed by $\hat{X}_{j}$ with weight $S_{ij}$, where $S_{ij}$ is the similarity between samples $X_{i}$ and $X_{j}$. Samples with larger distances between them should have a lower similarity. Then, we obtain the loss function of our weighted reconstruction as follows:

Since $S$ is a symmetric matrix, Eq.(2) can be further transformed as

where diagonal matrix $D=Diag(\sum_{j=1}^{n}S_{ij})$, and $L=D-S$ is the Laplacian matrix. We can see that the similarity matrix $S$ would be crucial to the performance of our network. Unlike many existing work using pre-defined values, which often fail to characterize the true relations between samples [63], we propose to automatically learn $S$ from data based on the self-expression concept.

Self-Expression Module. State-of-the-art subspace clustering methods are based on the self-expression property of data, Its basic idea is that each data point can be represented by a linear combination of other data points in the same subspace [5, 6]. The combination coefficient represents the relation between samples. To learn feature representations that are suitable for subspace clustering, the loss function of our self-expression module is

where the first term is a certain regularization function on $C$, and the second term minimizes the reconstruction error in the latent space $Z$, and $\alpha$ is a tradeoff parameter. Matrix $C$ can represent the subspace structure of data, i.e., $C_{ij}=0$, if the $i$-th sample and $j$-th sample do not lie in the same subspace. The constraint $diag(C)=0$ is optionally used to eliminate a trivial solution of $C=I$. Same as [46], we add a fully-connected layer without a bias between the encoder and decoder, whose weights represent the coefficient matrix $C$, the so-called self-expression layer, as shown in Fig.1.

Once the self-expression coefficient matrix $C$ is obtained, the similarity is usually computed based on $S=\frac{1}{2}(|C|+|C|^{T})$. Since the scale of each row and column of the similarity matrix might be different, we use the symmetric normalized Laplacian for scale normalization while maintaining the symmetry of the similarity matrix $S$ [3]. Specifically, the normalized degree matrix $D_{n}=I$ and normalized Laplacian matrix $L_{n}=D^{-\frac{1}{2}}LD^{-\frac{1}{2}}$. Then, we have $Tr[{(X-\hat{X})}^{T}D_{n}(X-\hat{X})]$ = $\|X-\hat{X}\|_{F}^{2}$. So the loss function of our weighted reconstruction is finally transformed as follows

As the second term of $L_{0}$ has a constraint for $C$, we can remove the first term of $L_{1}$.

As previously mentioned, learning clustering-friendly representations without leveraging any supervisory signal is an open and challenging problem. In fact, besides feature information, correlations are also helpful for deep image feature learning [61]. Inspired by this, we go one step further and fully exploit the above-learned similarity information. Specifically, we introduce pseudo-graph supervision and pseudo-label supervision to guide the network training by constructing pseudo-graphs and pseudo-labels.

Pseudo-graph Supervision. We introduce a classification layer parameterized by $\theta_{s}$ on top of the feature extraction module to produce pseudo-labels. More specifically, we add a fully-connected layer with a softmax function after the encoder, which transforms $Z$ to $f_{\theta_{s}}(Z)$, where $f_{\theta_{s}}(z_{i})\in\mathbb{R}^{K}$ represents the prediction feature of $z_{i}$ and $K$ is the number of clusters. It has the following properties:

In addition, the distribution of $f_{\theta_{s}}(Z)$ should be consistent with our previously learned similarity graph $S$. Since $S$ is learned in the training process, it is a pseudo-graph by default.

The loss induced by pseudo-graph supervision can be defined by

There are many choices for the loss function $\mathcal{L}_{g}$ in the literature. For example, we can use the contrastive Siamese net loss [64, 65] to regularize the distance between two samples and the binary cross-entropy loss [52] to evaluate the similarity. It is worth pointing out that our pseudo-graph is different from that in DAC [52] and DCCM [61]: Unlike previous research that uses a fixed function to calculate pseudo-graphs, we directly learn it from latent representation. This dynamic approach can reveal complex structures hidden in data. In addition, both DAC and DCCM use a threshold, which is hard to select, to find positive pairs. This could involve noisy false positive pairs. By contrast, our $S$ assigns a probability to each pair of points.

Pseudo-label Supervision. The correlation explored in previous sections is not transitive, i.e., $S_{ij}$ is not deterministic given $S_{ik}$ and $S_{jk}$ and limited to pairwise samples, which could lead to instability in training. Ideally, a pseudo-graph should have $K$-connected components or partitions, which could be regarded as pseudo-labels [19, 61]. This partition would make the optimal solution $\theta_{s}^{*}$ in the Eq. (7) lead to one-hot prediction, formulating the pseudo-label as

where $[.]_{k}$ represents the $k$-th element of the prediction vector, and its corresponding probability of the predicted pseudo-label is $p_{i}=\max[f_{\theta_{s}}(z_{i})]_{k}$ .

In practice, it is difficult to achieve optimal $\theta_{s}^{*}$ because of non-convexity. Consequently, the prediction $f_{\theta_{s}}(z_{i})$ would not follow the one-hot property. To address this problem, we set a large threshold $thres$ for probability $p_{i}$ to select highly confident pseudo-labels for supervision:

By doing so, only samples with highly confident pseudo-labels contribute to network training. Thus, the pseudo-label supervision loss can be formulated as:

where the loss function $\mathcal{L}_{l}$ is often defined by the cross-entropy loss.

To obtain a unified framework and jointly train the network, we combine loss (4), (5), (7), and (10) and reach the final objective function of PSSC:

where $\Theta$ denotes the network parameters, which include encoder parameters $\Theta_{e}$, self-expression layer parameters $C$, classification layer parameter $\Theta_{s}$ and decoder parameters $\Theta_{d}$. Note that, the output $\hat{X}$ of the decoder is a function of $\Theta_{e}$, $C$, and $\Theta_{d}$. All the unknowns in Eq.(11) are functions of the network parameters. This network can be implemented using neural network frameworks and trained by back-propagation. Once the network architecture is optimized, we obtain the lower-dimensional representation $Z$ and relation matrix $C$.

We conduct experiments on five benchmark datasets that are commonly used in subspace clustering evaluation because of their relatively high dimensionality. There are two face datasets: ORL, Umist; three object datasets: MNIST, COIL20, and COIL100. Following is the detailed information of these datasets.

• •

  ORL: This dataset comprises 40 subjects, and each subject has 10 images taken with varying poses and expressions.

• •

  MNIST: This dataset contains 10 clusters, including handwritten digits 0-9. Each cluster contains 6,000 images for training and 1,000 images for testing, with a size of 28$\times$28 pixels in each image. We use the first 100 images of each digit.

• •

  Umist: This dataset contains 480 images of 20 persons, and each image is taken under very different poses. Each image of the dataset is down-sampled to 32$\times$32.

• •

  COIL20: This dataset contains 1,440 gray-scale images of 20 different toys, and the pixels in each image are 32$\times$32.

• •

  COIL40: This dataset is composed of 40 classes of 2,880 data points.

The statistics of the datasets are summarized in Table I.

We compare our methods with both shallow and deep clustering techniques, including low-rank representation (LRR) [66], low-rank subspace clustering (LRSC) [13], sparse subspace clustering (SSC) [5], kernel SSC (KSSC) [67], SSC by orthogonal matching pursuit (SSC-OMP) [68], efficient dense subspace clustering (EDSC) [69], SSC with pre-trained convolutional AE features (AE+SSC), DSC network with $\ell 1$-norm (DSC-L1) [46], DSC network with $\ell 2$-norm (DSC-L2), deep embedding clustering (DEC) [27], deep $k$-means (DKM) [70], deep comprehensive correlation mining (DCCM) [61], deep embedded regularized clustering (DEPICT) [35], and deep spectral clustering using dual AE network (DSCDAN) [71].

To show the impact of pseudo-supervision, we make an ablation study by removing $L_{graph}$ and $L_{label}$ in Eq. (11) and term the left model as PSSC_l.

In our experiments, we use a one-layer convolutional network for the encoder and decoder in COIL20 and COIL40, a three-layer convolutional network for others. The architecture details of the networks are shown in Table II. We choose ReLU as the non-linear activation function and implement our method with TensorFlow.

First, we pre-train the encoder and decoder without the self-expression layer and softmax layer. Then, we fine-tune the entire network and grid searching for $\gamma_{1},\gamma_{2},\gamma_{3}$. The learning rate is set as 1e-3 in pre-training and 1e-4 in the fine-tuning stage. The contrastive Siamese net loss and cross-entropy loss are adopted for $L_{graph}$ and $L_{label}$ respectively. For pseudo-labels, threshold $thres$ is empirically set to 0.8. We use Adam optimizer [72] as the optimizer. The training process is summarized in Algorithm 10.

Similar to DSC [46], after we obtain the relation $C$, we use it to construct an affinity matrix for spectral clustering [3]. To enhance the block-structure and improve clustering accuracy, a typical method employed by various subspace clustering methods is summarized in Algorithm 2, where $\alpha$ is empirically selected according to the level of noise and $q$ is the maximal intrinsic dimension of subspaces [69]. Specifically, $q$ can be determined by the number of nonzero singular values of data points [5]. For these parameters, we use the same setting as our main competitor DSC [46]. To quantitatively assess clustering performance, we adopt three popular evaluation metrics: accuracy (ACC), normalized mutual information (NMI), and purity (PUR).

Table III shows the clustering results on the benchmark datasets. For each measure, the best methods are highlighted in boldface. We can observe that our method achieves the best performance among all comparison methods. In some cases, PSSC outperforms others by a significant margin. Specifically, we have the following observations.

• •

  PSSC outperforms shallow subspace clustering methods considerably. This is consistent with our expectation and mainly attributes to the powerful representation ability of neural networks.

• •

  PSSC and PSSC_l outperform the DSC and other deep clustering methods. In particular, the improvement of PSSC_l over DSC verifies the importance of locality preserving since the only difference between these two models stems from AE loss. PSSC enhances DSC by 4.48%, 4.74%, and 3.73% on average in terms of ACC, NMI, and purity, respectively.

• •

  Compared with PSSC_l, PSSC always achieves better results, indicating that pseudo-supervision guides feature and relation learning. For instance, with respect to PSSC_l, supervision boosts ACC by 5.8% and 6.46% on MNIST and Umist. This demonstrates that pseudo-graphs and pseudo-labels are promising methods for unsupervised tasks.

• •

  In some cases, DEC, DKM, and DCCM perform even worse than shallow approaches. This is because that they use Euclidean distance or cosine distance to evaluate the pairwise relation, which fails to capture the complex manifold structure. In general, the subspace learning approach works much better in this situation because it is originally designed to handle high-dimensional data. In general, the subspace learning approach works much better in this situation.

• •

  Compared to our method, DEPICT [35] and DSCDAN [71] produce inferior performance. With respect to DEPICT, the performance of DSCDAN is more stable.

Furthermore, self-supervised convolutional subspace clustering network ($S^{2}$ConvSCN-$\ell_{2}$) [62] and deep subspace clustering (DeepSC) [73] are two recent DSC methods. Since their authors have not published the source code, we directly cite their results on the datasets we used. For example, on COIL20, our proposed method’s ACC (NMI) is 0.9722 (0.9779), while DeepSC achieves 0.9801 (0.9791) and $S^{2}$ConvSCN-$\ell_{2}$ reaches 0.9767. We can see that the performance of these methods is close. On ORL, our proposed method’s ACC is 0.8675, while $S^{2}$ConvSCN-$\ell_{2}$ achieves 0.8875. Hence, our method is comparable to $S^{2}$ConvSCN-$\ell_{2}$ and DeepSC. In fact, both $S^{2}$ConvSCN-$\ell_{2}$ and DeepSC are not suitable for practical applications since the former involves spectral clustering and the latter needs to solve $\ell_{1}$ problem in each epoch, which results in high time complexity. By contrast, our pseudo-supervision is simple yet effective.

Furthermore, to intuitively show the similarity learning merit of our approach, we visualize the affinity matrix $A$ of PSSC and DSC in Fig.2, where $A_{ij}$ indicates the similarity between $Z_{i}$ and $Z_{j}$ and brighter pixel means higher similarity. As observed, most of the energy is perfectly located at the block-diagonal pixels. Compared to PSSC, more points of DSC appear in non-diagonal area, which means that they are inaccurately calculated or noisy. This partially explain the good performance of our method.

Fig.3 shows some reconstructed images on MNIST by our method and DSC. The first row is the original images, the second row is the images reconstructed by our PSSC, and third is DSC. We can find out that PSSC recovers the edges pretty well, while the images of DSC are blurred. This is also validated by PSNR value, PSSC achieves 15.54 while DSC gives 15.04. This verifies the drawback of existing self-reconstruction loss commonly used in AE.

There are three hyper-parameters $\gamma_{1},\gamma_{2},\gamma_{3}$ in PSSC. Generally, they are data-specific, thus, we perform a grid search to find the best parameter values for each dataset. To demonstrate their impacts on clustering performance, we fix $\gamma_{1}$ value and show the ACC and NMI variations induced by different $\gamma_{2},\gamma_{3}$ values in Figs.4-7. Although our method’s performance is influenced by the parameters, our method performs well for a wide range of values.

In above analysis, we follow the strategy in clustering community and treat the cluster number $k$ as a priori knowledge. In this part, we change $k$ to find out its impact on performance of ORL dataset. Since ORL has 40 classes, we set $k$ to 45 and 50 to see if no samples get assigned to the additional clusters or different variations within a class. As shown in Fig.8, all clusters are assigned with data points even $k$ goes to 50. When $k=40$, there are more bars that only have a single color than other situations, which indicates correct clustering. When $k=50$, more bars are with multiple colors, which indicates that the data samples from one class are split into different clusters.

As mentioned before, the intrinsic dimension of subspaces $q$ is determined by the data. For example, 3 and 16 are used in DSC for ORL and COIL20 respectively. In Fig.9, we plot the ACC values when $q$ varies. It can be seen that the best performance is achieved when $q$ is the number of nonzero singular values of data points.

We perform experiments on two object datasets: MNIST and USPS; a text dataset: RCV1. We use the complete MNIST dataset that has 70,000 handwritten digit images in 10 classes. USPS contains 9,298 handwritten digit images in 10 classes. RCV1 contains approximately 810,000 English news stories labeled with a category tree. Following DKM [70], we randomly sample 10,000 documents from the four largest categories: corporate/industrial, government/social, markets and economics of RCV1, each sample only belongs to one of these four categories. Note that, in contrast to experiments in DEC [27] and IDEC [28], we remain samples with multiple labels only if they do not belong to any two of the selected four categories. For text datasets, we select 2,000 words with the highest tf-idf values to represent each document. The statistics of datasets in this experiment are summarized in Table IV.

Different from the subspace clustering experiments in the last section, we use latent representations for the clustering task. On the one hand, it can directly demonstrate the quality of our latent representation. On the other hand, it can address out-of-sample and large-scale problems. In particular, we train the network with a reasonable small batch of samples (5,000 samples for each dataset in our experiments); then, we use the similarity matrix to predict the labels of selected samples following the above approach. Finally, we encode all data into latent space and use a nearest-neighbor classifier to predict the labels for the rest of the data. Following this approach, our method can address the out-of-sample problem, which is a long-standing challenge for subspace clustering. The architecture of our method for large-scale experiment is shown in Fig.10. The training process is summarized in Algorithm 3.

For a fair comparison, we use the same encoder/decoder architecture as DEC [27], IDEC [28], and DKM [70]. The encoder is a fully-connected network with dimensions of $d$-500-500-2000-$k$ for all datasets, where $d$ is the dimension of input features and $k$ is the number of clusters. Correspondingly, the decoder is a mirror of the encoder, a fully-connected network with dimensions of $k$-2000-500-500-$d$. A ReLU activation function is used for each layer except the input, output, and self-expression layers. We pre-train the AE in 50 epochs and fine-tune the whole network with an objective function (11) in 30 epochs.

We compare PSSC with $k$-means clustering (KM), an AE with KM applied to latent representation (AE+KM), and recent deep clustering approaches: DCN [29], IDEC [28], DKM [70], DCCM [61], DEPICT [35], DSCDAN [71], and $k$SCN [74]. Note that we did not use DCCM, DEPICT, $k$SCN, and DSCDAN on the text data because they are designed for image datasets. We use the SVD version of it because it gets the best performance in [74].

Table V summarizes the clustering performance of the above methods on three datasets. For each measure, the best result is highlighted in boldface. It can be observed that our method still outperforms recent deep clustering methods. In particular, compared to recent DKM, PSSC improves ACC by 5%, 17.8%, and 19.7% on the three datasets, respectively. With regard to DCCM, ACC improvement is more than 23%. Although the ACC of DEPICT on MNIST is approximately 3.9% higher than our method, our method achieves 7.9% higher on USPS. This performance benefits from embedding samples into subspaces and pseudo-supervising the learning process. $k$SCN is designed for deep subspace clustering on large image datasets. Compared to it, our method improves ACC by 2% and 6% on MNIST and USPS, respectively. Our method shows a convincing ability to address large-scale and out-of-sample problems.

Using USPS data as an example, we use the t-SNE [75] method to visualize the learned latent representations of several methods that produce good performance. As shown in Fig.11, samples of different classes merge in IDEC and DKM, which is because they force samples to move to cluster centers. In DEPICT, the black class is separated into a different place, which could degrade the performance. DSCDSN performs a little better but still has a problem that several clusters are close to each other. The same problem bothers $k$SCN, the pink and the purple classes blend with each other. Our method can precisely project samples of each class into a subspace, thus, samples can be well separated.