Dual Information Enhanced Multi-view Attributed Graph Clustering

The basic idea of multi-view clustering is to explore the consensus and complementary information embedded in multi-view data and learn a cluster partition that fits all views. Multi-view spectral clustering methods aim to learn a soft cluster indicator matrix directly from the original data. For instance, Kumar et al. [12] employ the co-regularized scheme to maximize the agreement between views. Xia et al. [13] propose a robust Markov-chain-based multi-view spectral clustering method where a low-rank transition matrix calculated by graphs is regarded as the input. Wang et al. [14] consider the clustering quality of each view and the clustering results agreement between different views. In [27], the self-weighting learning strategy is introduced to learn a block diagonal matrix for clustering from multiple graph inputs. In [28], similarity matrix learning, unified graph learning and indicator learning are incorporated into a unified framework. Liu et al. [29] consider the intra-view and inter-view correlations together and propose an adaptively weighted multi-view spectral clustering method. For graph-based multi-view subspace clustering methods, the self-representation matrices are constructed from the original graph data for clustering. In [40], the Hilbert Schmidt Independence Criterion (HSIC) term is introduced to explore the complementary information between the self-representation matrices from different views. Tang et al. [41] construct a consensus self-representation matrix with low-rank constraints for clustering. Zhang et al. [42] propose a one-step framework to explore the intra-view structure and the inter-view consistency simultaneously. In [43], the underlying correlations among different views and the specific information are explored to strengthen clustering performance. Recently, tensor learning-based multi-view clustering methods have achieved great progress. Xie et al. [44] stack the self-representation matrices from different views into a three-order tensor, and impose the t-SVD-based Tensor Nuclear Norm (TNN) on it to explore the high-order correlation among all views. Gao et al. [22] utilize the prior knowledge of the singular values and propose a weighted TNN for multi-view clustering. Wu et al. extend [13] by stacking the transition matrices into a tensor with the TNN constraint [45]. In [46], Chen et al. employ the weighted TNN for multi-view spectral clustering.

Unfortunately, the above methods can only process the attribute feature matrices or adjacent matrices separately, which can not fully explore the properties of the multi-view attributed graph data. To fill this gap, Lin et al. [8] introduce the graph filter for the attributed graph representation learning, where the high-order neighborhood correlation is also developed for clustering.

The main purpose of GNN is to learn the discriminative deep representation from the attributed graph. Graph Auto-Encoder (GAE) [39] based methods encode the adjacent matrix and attribute feature matrix into a deep representation, and then decode it with the decoder. Pan et al. [47] propose an adversarial graph embedding framework for clustering, where a discriminator is employed to force the deep representation to match the prior distribution. Wang et al. [48] incorporate cluster centroid learning and deep representation learning in a self-optimizing manner. In [38], the Graph Convolutional Network (GCN) is introduced to extract the multi-level deep representation of the attributed graph data for clustering. Sun et al. [49] employ the dual-decoder network to reconstruct the attribute feature matrix and the adjacent matrix of the input graph simultaneously. The above-mentioned methods are designed for single-view attributed graph data, and the complementary information of multi-view attributed graph data may not be fully exploited without considering the diversity of the inputs. Inspired by [50], Cheng et al. [51] introduce a consistent embedding encoder to maximize the agreements between views for a consensus representation fitting the target distribution. Fan et al. [52] define a most informative view with the modularity and decode the deep representation calculated from it to reconstruct multiple adjacent matrices. However, all GNN-based deep multi-view attributed graph clustering methods reconstruct the affinity matrix of each view from the consensus representation without explicitly considering the impact of the view-specific information, ultimately leading to sub-optimal clustering performance.

Information Maximization (or contrastive learning) has achieved great success in the field of computer vision and inspired many studies on unsupervised learning. Velickovic et al. [53] maximize the mutual information between the local patches and the global graph to obtain a compact representation. In [54], the information maximization is developed to maximize the agreement between the deep representations obtained by conducting GNN on the input graph and the augmented one. Pan et al. [55] employ a contrastive learning term to maximize the mutual information between each attribute node and its $k$-nearest neighbors. In [56], the graph information bottleneck is proposed, which aims to balance the expressiveness and robustness of the deep representation. Yu et al. [57] employ the graph information bottleneck theory to address the sub-graph recognition problem. Suresh et al. [37] propose a graph information bottleneck-based edge augmentation module to adaptively drop the redundancy edges on the input graph. To further reduce the redundancy information between the input graph and adaptive augmented one, Gong et al. develop a variant by imposing the orthogonal constraint on the deep representations [58].

Generally, most existing deep multi-view attributed graph clustering methods only focus on the consensus information between views while neglecting the specific information of each view. As illustrated in Figure 1, the adjacent matrices of the ACM dataset are distinct between different views, indicating that each view has its own specific information. For multi-view attributed graph data, except for the study of the consensus representation, how to discover the specific information of different views remains a challenging problem. In addition, these methods reconstruct the attribute graph of each view from the consensus representation, which may cause the conflict between learning the consensus representation and reconstructing the view-specific attribute graph. Specifically, the consensus objective aims to learn the common information across multiple views as much as possible while the reconstruction objective expects the common representation to preserve the view-specific information of different views. Furthermore, these methods obtain the final cluster partition directly from the low-level representations, which can not fully mine the latent high-level information, i.e., the clustering-oriented information embedded in all views.

To this end, the basic idea of the DIAGC method is to disentangle the learning procedure of the consensus and view-specific information for acquiring a view-consistent cluster partition, and maximize the mutual information between the low-level representations and the latent high-level clustering-oriented one. For clarity, the overall framework of DIAGC is illustrated in Figure 2.

The proposed DIAGC method consists of four modules, i.e., Graph Encoder (GE) module, Mutual Information Maximization (MIM) module, Specific Information Reconstruction (SIR) module and Self-supervised Clustering (SC) module.

It is worth noting that the reason for reconstructing $\hat{A}^{v}_{c}$ from $H^{v}$ instead of $S$ is to ensure that $S$ is free from the interference of specific information, which may lead to sub-optimal solutions. With the help of Eq. (5) and Eq. (12), the exploration of the consensus and specific information are disentangled elegantly, i.e., the MIM module ensures $H^{v}$ contains the view-common information while the SIR module reconstructs the features that are characteristic of the $v$-th view. After $T$ iterations, the final clustering results $\mathcal{K}$ are obtained by performing the $k$-means clustering method on $S$.

For clarity, the overall algorithm of the proposed DIAGC method is outlined in Algorithm 1.

The following three datasets are used in the experiments.

1. 1.

   ACM [60]: It is a paper network dataset containing 3025 papers as the attribute feature matrix. Besides, two kinds of relationships, i.e., co-paper and co-author relationships, are regarded as two views of the adjacent matrices, which can be divided into three categories¹¹1 https://dl.acm.org/..

2. 2.

   DBLP [61]: It is an author network dataset consisting of 4057 authors with three kinds of relationships, i.e., co-author, co-term and co-conference, as three different views of the adjacent matrices. Four categories of papers are collected: machine learning, information retrieval, database and data mining²²2 https://dblp.uni-trier.de/..

3. 3.

   IMDB [52]: It is a movie network dataset that contains 4780 movies with two kinds of relationships, i.e., co-actor and co-director. The movies are divided into three categories: drama, action and comedy³³3https://www.imdb.com/..

The statistical information of these datasets is summarized in Table II.

Thirteen clustering methods are compared with the proposed method. The details of these methods are provided as follows:

1. 1.

   LINE [62]: It is a deep single-view attributed graph clustering method, which is designed to preserve the first-order and the second-order proprieties of the attributed graph data. In our study, its best clustering results among all views are reported with LINE_best and its average clustering results of all views reported with LINE_avg⁴⁴4https://github.com/tangjianpku/LINE..

2. 2.

   MNE [63]: It is a scalable multi-view network embedding model representing the adjacent relationships from different views of multi-view attributed graph data into a unified embedding space⁵⁵5https://github.com/HKUST-KnowComp/MNE..

3. 3.

   PMNE [64]: The Project Multi-layer Network Embedding (PMNE) method consists of three sub-methods, i.e., network aggregation-based method (PMNE_n), result aggregation-based method (PMNE_r) and layer co-analysis-based method (PMNE_lc). These methods project the multi-layer network into a delegated vector space for clustering.

4. 4.

   GAE [39]: The Graph Auto-Encoder (GAE) is one of the most representative methods of single-view attributed graph clustering, which is composed of one graph encoder and one graph decoder. The best clustering results of GAE among all views are reported with GAE_best and its average clustering results of all views are reported as GAE_avg⁶⁶6https://github.com/tkipf/gae..

5. 5.

   ARGA [47]: The Adversarially Regularized Graph Autoencoder (ARGA) method is a single-view attributed graph clustering method, where a discriminator is utilized to ensure the deep representation calculated by encoder matching a prior distribution⁷⁷7https://github.com/GRAND-Lab/ARGA.

6. 6.

   DEAGC [48]: It is a goal-directed deep single-view attributed graph clustering method utilizing a self-optimizing module to learn a clustering-oriented deep representation.

7. 7.

   SDCN [38]: The Structural Deep Clustering Network (SDCN) method integrates the deep representations extracted by two deep neural networks to recover the distinct representations embedded in the single-view data⁸⁸8https://github.com/bdy9527/SDCN.

8. 8.

   RMSC [13]: It is a Markov-chain-based multi-view clustering method which takes each individual adjacent matrix as the transition probability matrix, and then obtains the shared low-rank representation from them⁹⁹9https://github.com/frash1989/ELM-MVClustering/tree/master/RMSC-ELM..

9. 9.

   PwMC [27]: It is a multi-view clustering method that adaptively learns the weights of each view for a consensus representation with low-rank constraint¹⁰¹⁰10https://github.com/kylejingli/SwMC-IJCAI17/blob/master/SwMC.

10. 10.

    SwMC [27]: It is a self-weighted multi-view clustering method introducing a self-conducted weight learning scheme to remove the explicitly defined weight factor.

11. 11.

    O2MA [52]: It is a deep multi-view attributed graph clustering method, which learns the deep representation from a most informative graph view and reconstructs all views from the deep representation¹¹¹¹11https://github.com/googlebaba/WWW2020-O2MAC..

12. 12.

    O2MAC [52]: It is a variant of O2MA containing a self-supervised clustering module with the clustering loss to generate the cluster-orient deep representation.

13. 13.

    MvAGC [8]: It is a multi-view attributed graph clustering method which adopts the graph filter learning term to capture the high-order neighborhood information¹²¹²12https://github.com/sckangz/MvAGC..

For fair comparison, similar to [8], part of the clustering results of the comparison methods are copied from [52].

Four widely used evaluation metrics, namely, ACCuracy (ACC), F1-score (F1), Normalized Mutual Information (NMI) and Adjusted Rand Index (ARI) [52] are employed to evaluate the clustering performance. Let $Y=\left\{Y_{1},Y_{2},\dots,Y_{c}\right\}\in\mathbb{R}^{n\times c}$ denote the ground-truth cluster partition, where $Y_{i}$ consists of the nodes belonging to the $i$-th cluster. Assuming that the cluster partition obtained by the clustering method is $Y^{*}=\left\{Y^{*}_{1},Y^{*}_{2},\dots,Y^{*}_{c}\right\}\in\mathbb{R}^{n\times c}$, ACC is calculated by:

where $\left|\cdot\right|$ is the number of the elements of the set.

F1 is obtained by:

where $Pr$ and $Re$ are precision and recall. They can be calculated by the following formulas:

where $TP$ denotes the number of true positives, $FP$ denotes the number of false positives and $FN$ denotes the number of false negatives.

NMI is calculated by:

The formulation of ARI can be written as:

where $E(\cdot)$ is the expected value, $RI$ denotes the rand index:

where $TN$ is the number of true negatives. Larger ACC, F1, NMI and ARI indicate better clustering performance.

In the comparison experiments, we run each method 10 times and report their average values in terms of ACC, F1, NMI and ARI in Table III, Table IV and Table V. From these tables, we have the following observations:

1. 1.

   The proposed DIAGC method achieves the best clustering performance compared with the baselines. These results verify that DIAGC could leverage the complementary information between different views to generate a more comprehensive representation for clustering.

2. 2.

   Compared with the earlier multi-view clustering methods, i.e., RMSC, PwMC and SwMC, most of deep learning-based methods achieve a large margin in terms of the evaluation metrics on the three multi-view attributed graph datasets. The results indicate that exploring the non-linearity relationships embedded in data is crucial for learning more discriminative representation.

3. 3.

   The GCN-based single-view clustering methods, i.e., GAE, AGRA, DAEGC and SDCN, perform much better than other single-view methods. The reason may be that GCN could incorporate both structural information embedded in the adjacent matrix and attribute information to mine the essential representation of attributed graph data.

4. 4.

   O2MA and O2MAC are both GCN-based multi-view attributed graph clustering methods. They utilize the consensus representation generated by GCNs as the deep representation for clustering and reconstruct adjacent matrices of all views from it. However, these two methods ignore the specific information of each view, which results in sub-optimal clustering performance. From Table III, it can be seen that the proposed DIAGC method performs much better than these two methods on all the benchmarks. Taking the ACM dataset as an example, it achieves 6.46% and 2.38% improvements in terms of NMI than O2MA and O2MAC, respectively. The reason is that the proposed DIAGC method could disentangle the consensus and the specific information learning procedures of each view. Such a process enables the network to generate a more clustering-friendly representation by eliminating the influence caused by the inherent specific information embedded in each view. With the help of specific information reconstruction module, the proposed DIAGC method could outperform all the deep baselines on the three datasets.

5. 5.

   On the IMDB dataset, all the baselines do not perform very well, which is caused by its sparsity. Even so, the proposed method can still achieve the best clustering performance on the IMDB dataset, which further validates its effectiveness.

In conclusion, the proposed DIAGC method can achieve more accurate clustering results than the existing state-of-the-art methods.

To further validate the effectiveness of MIM, SIR and SC modules of the proposed DIAGC method, in this subsection, the ablation study is conducted and the experiment results are reported in Table VI. It is noteworthy that without the MIM module, the SC module is performed on the representation generated by the linear sum of all the deep representations, i.e., the SC module is performed on $S^{{}^{\prime}}=\frac{1}{V}\sum_{v=1}^{V}H^{v}$, and the final clustering results are obtained by performing the $k$-means on $S^{{}^{\prime}}$. Without SIR module, the $L_{R}$ in Eq. (11) is reformulated as:

From Table VI, it can be seen that without the MIM module, the clustering results of DIAGC drop dramatically on all the benchmarks. The reason may be that $S^{{}^{\prime}}$ is generated by linearly summing the deep representations of all views with the same weight, and such a naive learning strategy breaks the non-linear structure constructed by GNN, where there are only the low-level representations. In addition, the clustering results for all benchmarks are degraded in the absence of the SIR module, because the low-level representations $H^{v}$ computed by GCN without the SIR module are disturbed by the inherent specific information embedded in each view. Moreover, the effectiveness of the SC module is also validated and it improves the clustering results in terms of NMI by 5.58%, 1.03% and 0.97% on the ACM, DBLP and IMDB datasets, respectively. These results demonstrate that all the modules in the proposed DIAGC method are essential to improve the clustering performance.

In this subsection, we would conduct the parameter analysis on all the benchmarks with the hyper-parameter $\alpha$, which is tuned in the range of $\{0.0001,0.001,0.01,0.1,1\}$. Figure 3 plots the clustering results in terms of ACC, F1, NMI and ARI on all the benchmarks, respectively. According to Figure 3, we can observe that the clustering performance of DIAGC is relatively stable over all the benchmarks with different values of $\alpha$, which indicates the robustness of DIAGC to obtain the promising clustering performance.

The visualization experiments of the proposed method compared with some baselines will be presented in this subsection to validate the superiority of the proposed DIAGC method. Specifically, we project the deep representations learned from the different models into a two-dimensional space. Then, the t-SNE visualization method is performed with all the representations on the ACM dataset. As shown in Figure 4, the DEAGC method does not perform well on the ACM dataset as it could not divide the data appropriately. The O2MAC method could separate the green data points from the rest of the data, but the other clusters are still blurry. One reason may be that the O2MAC method can only encode one view to calculate the deep representation, without considering the complementary information from different views. From Figure 4(c), it can be seen that the cluster structures obtained by DIAGC are generally clear, which validates the effectiveness of the proposed method.

In this subsection, the convergence property of the proposed DIAGC method would be verified. Specifically, we plot the convergence curves about Eq. (11) versus iterations in Figure 5. As shown in Figure 5, the convergence curves are generally monotonically decreasing except for the small amplitude fluctuation caused by $L_{KL}$. Since the clustering centroids of $Q$ are recalculated at each iteration, some bias would be inevitably introduced, resulting in that the $L_{KL}$ between the soft label distribution and the target distribution is not always decreasing. Even though, the overall convergence curves of the proposed DIAGC method are decreasing and remain steady on all the benchmarks, which indicates the desirable convergence property of the proposed method.