Graph Contrastive Learning with
Cross-view Reconstruction

In this work, we focus on the graph-level task, let $\mathcal{G}=\left\{G_{i}=(V_{i},E_{i})\right\}_{i=1}^{N}$ denote a graph dataset with N graphs, where $V_{i}$ and $E_{i}$ are the node set and edge set of graph $G_{i}$, respectively. We use $x_{v}\in\mathbb{R}^{d}$ and $x_{e}\in\mathbb{R}^{d}$ to denote the attribute vector of each node $v\in V_{i}$ and edge $e\in E_{i}$. Each graph is associated with a label, denoted as $y_{i}$, the goal the graph representation learning is to learn an encoder $f:G_{i}\rightarrow\mathbb{R}^{d}$ so that the learned representation $\mathbf{z}_{i}=f(G_{i})$ is sufficient to predict $y_{i}$ related to the downstream task. We clarify the sufficiency of $\mathbf{z}_{i}$ as containing no less information of the label of $G_{i}$ (Achille & Soatto, 2018), and it is formulated as:

where $I\left(;\right)$ denotes the mutual information between two variables.

Contrastive Learning (CL) is a self-supervised representation learning method which leverages instance-level identity for supervision. During the training phase, each graph $G$ firstly goes through proper data augmentation to generate two data augmentation views $t_{1}(G)$ and $t_{2}(G)$, where $t_{1}(\cdot)$ and $t_{2}(\cdot)$ are two augmentation operators. Then, the CL method encourages the encoder $f$ (a backbone network plus a projection layer) to map $t_{1}(x)$ and $t_{2}(x)$ closer in the hidden space so that the learned representations $\mathbf{z}_{1}$ and $\mathbf{z}_{2}$ maintain all the information shared by $t_{1}(G)$ and $t_{2}(G)$. The learning of the encoder is usually directed by a contrastive loss, such as NCE loss (Wu et al., 2018b), InfoNCE loss (van den Oord et al., 2018) and NT-Xent loss (Chen et al., 2020). In Graph Contrastive Learning (GCL), we usually adopt a GNN, such as GCN (Kipf & Welling, 2017) or GIN (Xu et al., 2019), as the backbone network, and the commonly-used graph data augmentation operators (You et al., 2020), such as node dropping, edge perturbation, subgraph sampling, and attribute masking.

All the GCL-based methods are built on the assumption that augmentations do not break the sufficiency requirement to make correct prediction. Here, we follow (Federici et al., 2020) to clear up the definition of mutual redundancy. $t_{1}(G)$ is redundant to $t_{2}(G)$ with respect of $y$ iff $t_{1}(G)$ and $t_{2}(G)$ share the same predictive information. Mathematically, the mutual redundancy in CL exists when:

Although GCL-based methods are usaully capable to extract useful information for label identification, it is unavoidable to include non-predictive features under the SSL setting owing lack of explicit domain knowledge. There exist the situation (e.g., OOD setting) that the latent space of learned representation is dominated by non-predictive features in SSL  (Chen et al., 2021) and it is no more informative enough to make correct prediction. Therefore, feature suppression is not just a prevalent issue in supervised learning, but also in SSL. Due to the page limitation, we provide more discussion about the relation between feature suppression and GCL in Appendix B

In GCL, we usually leverage a graph encoder, such as a GCN (Kipf & Welling, 2017) or a GIN (Xu et al., 2019), to encode the graph data into its representation. There are multiple choices of graph encoders in GCL, including GCN (Kipf & Welling, 2017) and GIN (Xu et al., 2019), etc. In this work, we adopt GIN as the backbone network $f$ for simplicity. Note that any other commonly-used graph encoders can also be adapted to our model. Given two augmentation views $t_{1}(G)$ and $t_{1}(G)$ (where $t_{1}(\cdot)$ and $t_{2}(\cdot)$ are IID sampled from the same family of augmentation $\mathcal{T}$), we firstly use the encoder $f(\cdot)$ to map them into a lower dimension hidden space for the two embeddings, $\mathbf{z}_{1}$ and $\mathbf{z}_{2}$. Instead of directly maximizing the agreement between the two representations $\mathbf{z}_{1}$ and $\mathbf{z}_{2}$, we further feed each of them into a pair decoders $(g_{p}$, $g_{c})$ (both of them are MLP-based networks or GNN) and optimize the two decoders to map each of the presentation into the two disentangled feature sub-spaces:

where a pair of embeddings for both $t_{1}(G)$ and $t_{2}(G)$ are generated. Ideally, $\mathbf{z}_{1}^{p}$ and $\mathbf{z}_{2}^{p}$ suffice the mutual redundancy assumption stated in 2.2 because $t_{1}(G)$ and $t_{1}(G)$ are augmented from the same original graph, and thus naturally share the same predictive factors.

Here, we clarify the lower bound of the mutual information between one augmented view and the two mapped representations learned from the other augmented view in Theorem 1.

The detailed proof is provided in Appendix E. Given the lower bound, we substitute the objective by the mutual information between the two representations in the predictive view ($\mathbf{z}_{1}^{p}$ and $\mathbf{z}_{2}^{p}$) to maximize the consistency between the information of the two views. Therefore, we derive the objective function ensuring view invariance as follows:

where $\mathcal{L}_{\text{CL}}(\cdot)$ is the adopted InfoNCE loss (van den Oord et al., 2018). To further pursue the feature disentanglement as illustrated in Figure 1(c), we thus propose the cross-view reconstruction mechanism. To be specific, we would like the representation pair $(\mathbf{z}^{p},\mathbf{z}^{c})$ within and cross the augmentation views be able to recover the raw data so that the two objectives can be approached simultaneously. Due to the fact that graphs are non-Euclidean structured data, we instead try to recover $\mathbf{z}=f(t(G))$ given $(\mathbf{z}^{c}$ and $\mathbf{z}^{p})$.

More specifically, we first perform the reconstruction within the augmentation view, namely mapping $(\mathbf{z}_{w}^{p},\mathbf{z}_{w}^{c})$ to $\mathbf{z}_{w}$, where $w\in\{1,2\}$ representing the augmentation view. Then, we define the $(\mathbf{z}_{w^{\prime}}^{p},\mathbf{z}_{w}^{c})$ as a cross-view representation pair and the reconstruction procedure is repeated on it to predict $\mathbf{z}_{w}$, aiming to ensure $\mathbf{z}^{p}$ and $\mathbf{z}^{c}$ is optimized to approximate mutual disentanglement, where $w=1,w^{\prime}=2$ or $w=2,w^{\prime}=1$. Intuitively, the reconstruction process is capable of separating the information of the shared features sets from the one resided in the unique feature sets between the two augmentation views. Since the two IID sampled augmentation operators ($t_{1}(\cdot)$ and $t_{2}(\cdot)$) are expected to preserve the predictive/rational features while varying the augmentation-related ones, we disentangle the rational features from $G$ according to the rationale discover studies (Chang et al., 2020) to ensure the features’ robustness for downstream tasks. Here, we formulate the reconstruction procedures as:

where $g_{r}$ is the parameterized reconstruction model and $\odot$ is the free-to-choose fusion operator, such as element-wise product or concatenation. The reconstruction procedures are optimized by minimizing the entropy $H\left(\mathbf{z}_{w}\mid\mathbf{z}_{w^{\prime}}^{p},\mathbf{z}_{w}^{c}\right)$, where $w=w^{\prime}$ or $w\neq w^{\prime}$. Ideally, we reach the optimal sufficiency and disentanglement conditions illustrated in Figure 1 (b) and (c) iff $H\left(\mathbf{z}_{w}\mid\mathbf{z}_{w^{\prime}}^{p},\mathbf{z}_{w}^{c}\right)=-\mathbb{E}_{p\left(\mathbf{z}_{w},\mathbf{z}_{w^{\prime}}^{p},\mathbf{z}_{w}^{c}\right)}\left[\log p\left(\mathbf{z}_{w}\mid\mathbf{z}_{w^{\prime}}^{p},\mathbf{z}_{w}^{c}\right)\right]=0$, where $\mathbf{z}_{w}$ is exactly recovered given its complementary representation and the predictive representation of any view. Nevertheless, the condition probability $p\left(\mathbf{z}_{w}\mid\mathbf{z}_{w^{\prime}}^{p},\mathbf{z}_{w}^{c}\right)$ is intractable, we hence use the variational distribution approximated by $g_{r}$ instead, denoted as $q\left(\mathbf{z}_{w}\mid\mathbf{z}_{w^{\prime}}^{p},\mathbf{z}_{w}^{c}\right)$. We provide the upper bound of $H\left(\mathbf{z}_{w}\mid\mathbf{z}_{w^{\prime}}^{p},\mathbf{z}_{w}^{c}\right)$ in Theorem 2.

The detailed proof is demonstrated in Appendix E. Since we adopt two augmentation views, the objective function constraining representation disentanglement can be formulated as:

With the cross-view reconstruction mechanism above, the two learned representations stated above are optimized towards the disentangled manner. However, it is still necessary to further prevent the learned predictive representation from focusing on the partial features, because we do not have access to the explicit domain knowledge and such small scope will increase the risk of shortcut solution. Therefore, we extend the Equation 4 to three contrastive views and add an extra global view without topological perturbation as the third views to guarantee the learned $\mathbf{z}^{p}$ maintain the global semantics instead of partial or even trivial features, i.e., $\mathbf{z}_{1}^{p}\sim G$ and $\mathbf{z}_{2}^{p}\sim G$. During the experiments, we find an adversarial graph sample perturbed from original graph view can help the model achieve stronger robustness. A possible explanation is that there is still redundant information that is not predictive left in the shared information of the two $\mathbf{z}^{p}$’s in the two augmentation views, especially when the implemented augmentations are moderate. An adversarial view may further alleviate redundancy. We define the adversarial objective as follows:

where the adversarial sample $G+\delta$ together with the two augmentation views, i.e., $t_{1}(G)$ and $t_{2}(G)$ are employed as the positive pair. Our crafted perturbation is spurred by recent work (Yang et al., 2021) that add perturbation $\delta$ on the output of first hidden layer $\mathbf{h}^{(1)}$, since it is empirically proved to generate more challenging views than adding perturbation on the initial node feature. Therefore, the adversarial contrastive objective is defined as:

where the optimized perturbation $\delta^{\prime}$ is solved by projected gradient descent (PGD) (Madry et al., 2018). Finally, we derive the joint objective of GraphCV by combining all of objectives above together. The joint objective is as follow:

where $\lambda_{r}$ and $\lambda_{a}$ are the coefficients to balance the magnitude of each loss term. Our proposed model is able to learn optimal representation illustrated in Figure 1(c) with the joint objective.

Datasets. For unsupervised learning setting, we evaluate our model on five graph benchmark datasets from the field of bioinformatics, including MUTAG, PTC-MR, NCI1, DD, and PROTEINS, and other four from the field of social network, which are COLLAB, IMDB-B, RDT-B, and IMDB-M, for the task of graph-level property classification. For the transfer learning setting, we follow previous work (You et al., 2020; Xu et al., 2021b) to pretrain our model on the ZINC-2M dataset, which cotains 2 million unlabeled molecule graphs sampled from MoleculeNet (Wu et al., 2018a), then evaluate its performance on eight binary classification datasets from chemistry domain, where the eight datasets are splitted according to the scaffold to simulate the out-of-distribution scenario in real-world. Additionally, We use ogbg-molhiv from Open Graph Benchmark Dataset (Hu et al., 2020a) to evaluate our model over large-scale dataset under semi-supervised setting. More details about dataset statistics are included in Appendix C.

Baselines. Under the unsupervised representation learning setting, we compare GraphCV with the eight SOTA self-supervised learning methods GraphCL (You et al., 2020), InfoGraph(Sun et al., 2019), MVGRL (Hassani & Khasahmadi, 2020a), AD-GCL(Suresh et al., 2021), GASSL(Yang et al., 2021), InfoGCL(Xu et al., 2021a), RGCL (Li et al., 2022) and DGCL(Li et al., 2021), as well as three classical unsupervised representation learning methods, including node2vec (Grover & Leskovec, 2016), graph2vec (Narayanan et al., 2017), and GVAE(Kipf & Welling, 2016). Besides, we employ AttrMasking (Hu et al., 2020b), ContextPred (Hu et al., 2020b), GraphCL (You et al., 2020), GraphLoG (Xu et al., 2021b), AD-GCL (Suresh et al., 2021) and RGCL (Li et al., 2022) as baselines to evaluate the effectiveness of our proposed GraphCV under transfer learning setting.

Evaluation Protocol. For unsupervised setting, we follow the evaluation protocols in the previous works (Sun et al., 2019; You et al., 2020; Li et al., 2021) to verify the effectiveness of our model. The mean test accuracy score evaluated by a 10-fold cross validation with standard deviation of five random seeds as the final performance. For transfer learning setting, we follow the finetuning procedures of previous work (You et al., 2020; Xu et al., 2021b) and report the mean ROC-AUC scores with standard deviation of 10 repeated runs on each downstream datasets. In addition, we follow the setting of semi-supervised representation learning from GraphCL on the ogbg-molhiv dataset, with the finetune label rates as 1%, 10%, and 20%. The final performance is reported as the mean ROC-AUC of five initialization random seeds

Unsupervised learning. The overall performance comparison is shown in Table 1 and we can have three observations: (1) The GCL-based methods generally yield higher performances than classical unsupervised learning methods, indicating the effectiveness of utilizing instance-level supervision; (2) RGCL, AD-GCL, and GASSL achieve better performances than GraphCL, which empirically proves the conclusion that InfoMax object could bring overwhelmed redundant information and thus suffer from feature suppression issue; (3) Our proposed GraphCV and DGCL consistently outperform other baselines, proving the advantage of disentangled representation. More importantly, GraphCV achieves state-of-the-art results on most of the datasets, demonstrating the model effectiveness.

Transfer learning. Table 2 demonstrates the experimental results under transfer learning setting, where No Pre-Train skips self-supervised pre-training process on the ZINC-2M dataset for model initialization before finetune. It is noteworthy that some strong baselines (AttrMasking and ContextPred) are trained under the guidance of domain knowledge. Despite in lacking of such domain knowledge, our model still outperforms all the other baselines on 3 out 8 datasets and achieve highest average performance. More importantly, JOAO, RGCL and our proposed GraphCV are all developed from GraphCL, but achieve higher average performance than GraphCL. This observation further empirically prove the poisoning effect of biased information and the necessity to to suppress them.

Semi-supervised learning. The experimental results are shown in Figure 3. It is obvious that our model gains significant improvements under the three label-rate fine-tuning settings. We also notice that as the label rate increases, the amount of improvement increases as well (1%, 1.8%, and 4.4% for label rate 1%, 10%, and 20%, respectively). A possible explanation could be that more trainable data could bring more redundant information, thereby further deteriorate the feature suppression issue. Therefore, removing redundant information causes a higher performance boost.

To further verify the effectiveness of different modules in GraphCV, we perform ablation studies on each one of the module by creating two model variants: (1) w/o CV Recon, the cross-view reconstruction process is discarded; (2) w/o Adv. Training, the third adversarial view is discarded. The comparison results are shown in Table 3. We can observe from Table 3 that our model with the combination of cross-view reconstruction and adversarial training module outperforms all of the variants. Omitting the reconstruction process could cause the failure to optimize the representation in a disentangled manner illustrated in Figure 1(c), thereby the learned representation still suffer from features suppression issue. Compared with our model, the variant w/o Adv. Training may lead to representation collapse and bring extra redundant information, therefore resulting in sub-optimal performances in downstream tasks.

In this section, we firstly conduct extra experiments on ogbg-molhiv dataset to evaluate the representation robustness under aggressive augmentation and perturbation. The results are shown in left two subplots of Figure 4, we compare our method with GASSL under different perturbation bounds and attack steps to demonstrate its robustness against adversarial attacks. Since both our model and GASSL use GIN as the backbone network, we hereby add the performance of GIN as the compared baseline. Although aggressive adversarial attacks can largely deteriorate the performance, our proposed GraphCV still achieves more robust performance than GASSL. In the right two subplots, we analysis the model sensitivity of the two important hyper-parameters in our model, $\lambda_{r}$ and $\lambda_{a}$. The consistent superiority of different values over the initial point (i.e., $\lambda_{r}$, $\lambda_{a}=0$) prove the effectiveness of our design once again. We can also observe that the appropriate range of the two hyper-parameters are 5.0 to 10.0 and 0.0 to 0.5, respectively. Depend on the datasets size and attributes, the range can have some variance, we suggest to finetune the two hyper-parameters around 10.0 and 0.25 to find the appropriate values when adopting our model to a new datasets. More experiments and discussion about hyper-parameter sensitivity in provided in the Appendix H. Besides, we also conduct extra experiments to analyze the disentanglement of $\mathbf{z}^{p}$ and $\mathbf{z}^{c}$ in Appendix F.