RARE: Robust Masked Graph Autoencoder

In this study, we mainly focus on the task of self-supervised masked graph pre-training for unlabeled graphs. Our model is designed to learn two graph encoding functions (i.e., $\mathcal{F}_{g}(\cdot)$ and $\mathcal{F}_{m}(\cdot)$), along with a hidden predicting function (i.e., $\mathcal{F}_{p}(\cdot)$) to recover masked latent features from observations. Subsequently, a decoding function (i.e., $\mathcal{F}_{d}(\cdot)$) is employed to reconstruct the raw attributes of masked samples based on the predicted representations. The learned graph embedding can be saved and utilized for various downstream tasks, such as node classification and graph classification.

As shown in Fig. 2, the pre-training procedure of RARE could be mainly grouped into three parts. The goal of the data masking part is to generate two complementary masked graphs by randomly masking some nodes with tokens under a mask ratio $r$. The masked latent feature completion part is the core of RARE, which aims to enhance the reliability of the self-supervision mechanism by leveraging more informative high-order sample correlations to drive the model training. It consists of three components. Firstly, the graph encoder maps the visible nodes into node representations. Secondly, the latent feature predictor performs a latent feature prediction from visible nodes to masked ones. Thirdly, the momentum graph encoder receives the raw data of masked nodes as inputs and takes the resultant representations as implicit self-supervision signals for matching with the predicted representations. In the data decoding part, the decoder only maps the representations of masked nodes into the raw data space. Finally, $\mathcal{L}_{M}$ and $\mathcal{L}_{R}$ are integrated to minimize the loss error in both latent feature and raw data spaces. After pre-training, only the backbone of the graph encoder is adopted for downstream evaluations. The following subsections present the technical details of the corresponding components.

The graph encoder $\mathcal{F}_{g}(\cdot)$ is responsible for transforming the masked graph $\mathcal{G}^{v}$ into a low-dimension latent space. To achieve this, we employ a graph neural network (GNN)-based architecture that consists of a sequence of graph attention layers [22] or graph isomorphism layers [23] as the encoder backbone. Inspired by BYOL [46], we incorporate a multilayer perception (MLP) layer as a projector following the backbone to form the graph encoder. This encoder generates the visible node representation matrix $\mathbf{Z}^{v}\in\mathbb{R}^{|\mathcal{V}^{v}|\times d}$, where $d$ represents the latent dimension.

Following the graph encoder $\mathcal{F}_{g}(\cdot)$, an autoencoder-style latent feature predictor $\mathcal{F}_{p}(\cdot)$ is elaborately designed, which consists of two parts, i.e., a graph attention (or graph isomorphism) layer that recovers the masked content from observations and an MLP layer that predicts the latent features of masked nodes based on recovered information. Specifically, we utilize a Concat function $C(\cdot)$ to integrate $\mathbf{Z}^{v}$ and a token-masked representation matrix $\mathbf{H}^{m}\in\mathbb{R}^{|\mathcal{V}^{m}|\times d}$, where $\mathbf{h}_{i}^{m}\in\mathbb{R}^{d}$ denotes a $d$-dimensional stochastic learnable vector. It is worth noting that the information concatenation used here to construct $\mathbf{\widetilde{Z}}\in\mathbb{R}^{|\mathcal{V}|\times d}$ is not the classic channel-wise or row-wise concatenation. Instead, we fill the visible part with $\mathbf{Z}^{v}$ and the masked part with $\mathbf{H}^{m}$ to create the recomposed representation matrix $\mathbf{\widetilde{Z}}$. Finally, we apply $\mathcal{F}_{p}(\cdot)$ to process $\mathbf{\widetilde{Z}}$ and obtain a predicted representation matrix of masked nodes $\mathbf{\widehat{Z}}^{m}\in\mathbb{R}^{|\mathcal{V}^{m}|\times d}$.

Since we have obtained the predicted representations of masked nodes, a natural question arises: how to provide effective supervision to guide the masked latent feature completion in unsupervised scenarios? Our answer is to acquire the self-supervision signals from the data itself via self-distillation learning. To this end, we introduce a momentum graph encoder $\mathcal{F}_{m}(\cdot)$ that has the same architecture as the graph encoder. This encoder is responsible for encoding the raw data of masked samples and utilizing their complete representations to provide the predicted ones with stable optimization guidance. Concretely, we take the masked graph $\mathcal{G}^{m}$ as an input and feed it into $\mathcal{F}_{m}(\cdot)$. The resultant representation matrix $\mathbf{Z}^{m}\in\mathbb{R}^{|\mathcal{V}^{m}|\times d}$ preserves high-order sample correlations and serves as implicit self-supervision signals to refine $\mathbf{\widehat{Z}}^{m}$. It is worth noting that $\mathcal{F}_{m}(\cdot)$ is detached from the gradient back-propagation and its parameters are updated by exponential moving average (EMA) [46]. The parameters of $\mathcal{F}_{g}(\cdot)$ and $\mathcal{F}_{m}(\cdot)$ are denoted as $\Theta_{\mathcal{F}_{g}}$ and $\Theta_{\mathcal{F}_{m}}$, respectively, and the parameters of $\mathcal{F}_{m}(\cdot)$ are updated by:

where $\mu$ denotes the momentum factor that has been determined empirically and fixed as 0.1. Since both graph encoders involve training on multiple subsets of a common graph, the EMA can provide $\mathcal{F}_{m}(\cdot)$ with a smooth estimate of the underlying graph data distribution from $\mathcal{F}_{g}(\cdot)$, thus promoting $\mathcal{F}_{m}(\cdot)$ to encode reliable information.

This operation acts as an implicit form of self-supervision for recovering masked content at the feature level. Rather than aiming to maintain similarity between the reconstructed nodes (or edges) and the raw information of the original graphs, our method focuses on ensuring that the predicted representations precisely match with the underlying structural statistics calculated by the momentum graph encoder. To this end, we approximate $\mathbf{\widehat{Z}}^{m}$ to $\mathbf{Z}^{m}$ by minimizing the following formulation:

where $\widehat{\mathbf{z}}^{m}_{i}\in\mathbb{R}^{d}$ and $\mathbf{z}^{m}_{i}\in\mathbb{R}^{d}$ indicate the representation vectors of $i$-th masked node within $\mathbf{\widehat{Z}}^{m}$ and $\mathbf{Z}^{m}$, respectively. According to Eq. (3), we recover masked information within the latent space by constraining the predicted representations to match with the ones that preserve more informative underlying structural information of graphs. By taking these implicit self-supervision signals as model learning guidance, we could ensure the integrity and accuracy of predicted representations, resulting in a higher quality of the learned graph embedding.

By integrating implicit and explicit self-supervision mechanisms in a united pre-training framework, the total loss of the proposed RARE can be formulated as a weighted combination of the latent feature matching loss $\mathcal{L}_{M}$ and the raw attribute reconstruction loss $\mathcal{L}_{R}$:

where $\alpha$ is a balanced coefficient. In the inference phase, the input graph $\mathcal{G}$ with $\mathbf{\widetilde{A}}$ and $\mathbf{X}$ is fed into RARE without any data-masking operations. The resultant graph embedding can be saved and used for downstream evaluations, such as graph classification and image recognition tasks. The detailed pre-training procedure of RARE is illustrated in Algorithm 1.

The time complexity of the proposed RARE could be discussed from the following two perspectives: the graph auto-encoder framework and the loss error computation. For two graph encoders, the complexities of $\mathcal{F}_{g}(\cdot)$ and $\mathcal{F}_{m}(\cdot)$ are $\mathcal{O}(LK(|\mathcal{V}|d_{i}d_{o}+|\mathcal{E}|d_{o}))$, where $|\mathcal{V}|$, $|\mathcal{E}|$, $L$, and $K$ are the numbers of nodes, edges, encoder layers, and attention heads, respectively. $d_{i}$ and $d_{o}$ are the dimension of input and output features, respectively. For the latent feature predictor, the complexity of $\mathcal{F}_{p}(\cdot)$ is $\mathcal{O}(L(K(|\mathcal{V}|d_{i}d_{o}+|\mathcal{E}|d_{o})+|\mathcal{V}|d_{i}d_{o}))$. For the decoder, the complexity of $\mathcal{F}_{d}(\cdot)$ is $\mathcal{O}(L|\mathcal{V}|d_{i}d_{o})$. For the computation of loss error, the time complexities of $\mathcal{L}_{M}$ and $\mathcal{L}_{R}$ are $\mathcal{O}(|\mathcal{V}^{M}|D)$, where $|\mathcal{V}^{M}|$ and $D$ are the numbers of token-masked samples and the dimension of node attributes. Therefore, the overall time complexity of RARE for each training iteration is $\mathcal{O}(L(K(|\mathcal{V}|d_{i}d_{o}+|\mathcal{E}|d_{o})+|\mathcal{V}|d_{i}d_{o})+|\mathcal{V}^{M}|D)$. We can observe that the complexity of RARE is linear with both the numbers of nodes $|\mathcal{V}|$ and edges $|\mathcal{E}|$ of the graph, making the proposed RARE theoretically efficient and scalable.

We start from a more intuitive masked signal modeling perspective to revisit MLFC. As aforementioned, we can obtain the latent features of visible and masked nodes from two complementary masked views (i.e., $\mathcal{G}^{v}$ and $\mathcal{G}^{m}$), respectively:

After that, $\mathcal{F}_{p}(\cdot)$ outputs the predicted representations $\mathbf{\widehat{Z}}^{m}$ for masked nodes from visible ones $\mathbf{Z}^{v}$, and then match $\mathbf{\widehat{Z}}^{m}$ with $\mathbf{Z}^{m}$ by Eq. (3), which can be rewritten as:

Based on Eq. (8), we observe that the MLFC scheme actually learns to pair two complementary views (i.e., $\mathbf{Z}^{v}$ and $\mathbf{Z}^{m}$) through a latent feature matching task. Inspired by the previous work [50, 51], we denote a bipartite graph $\mathcal{G}_{B}=\{\mathcal{Z}^{v},\mathcal{Z}^{m},\mathcal{E}_{B}\}$ to model the corresponding learning problem, where $\mathcal{Z}^{v}=\{\mathbf{Z}^{v}\}^{|\mathcal{Z}^{v}|}_{i=1}$ and $\mathcal{Z}^{m}=\{\mathbf{Z}^{m}\}^{|\mathcal{Z}^{m}|}_{j=1}$ denotes the sets of visible views and masked views, respectively. $\mathcal{E}_{B}$ is represented as an adjacency matrix $\mathbf{A}_{B}\in\mathbb{R}^{|\mathcal{Z}^{m}|\times|\mathcal{Z}^{v}|}$ whose normalized version is $\widetilde{\mathbf{A}}_{B}=\mathbf{D}^{m-\frac{1}{2}}\mathbf{A}_{B}\mathbf{D}^{v-\frac{1}{2}}$. Here, both $\mathbf{D}^{m}$ and $\mathbf{D}^{v}$ are diagonal degree matrices. Consequently, we can derive an asymmetric instance alignment loss between $\mathbf{Z}^{v}$ and $\mathbf{Z}^{m}$ to lower bounded $\mathcal{L}_{M}$.

Theorem 1. We assume that any autoencoder-style architecture $\mathcal{F}(\cdot)$ satisfies $\mathbb{E}_{\mathbf{X}}\|\mathcal{F}(\mathbf{X})-\mathbf{X}\|^{2}\leq\delta$, where $\mathbf{X}$ represents either visible content $\mathbf{Z}^{v}$ or masked content $\mathbf{Z}^{m}$, therefore, the MLFC loss on the bipartite graph $\mathcal{G}_{B}$ can be lower bounded by:

where $\mathbf{X}^{v}\in\mathbb{R}^{|\mathcal{Z}^{v}|\times(|\mathcal{V}^{m}|d)}$ denotes the output matrix of $\mathcal{F}_{p}(\cdot)$ on $\mathcal{Z}^{v}$ whose $i$-row is $\mathbf{x}^{v}_{i}$=$\sqrt{d_{i}}\mathcal{F}_{p}(\mathbf{Z}^{v},\mathbf{H}^{m},\widetilde{\mathbf{A}})_{i}\in\mathbb{R}^{|\mathcal{V}^{m}|d}$, and $\mathbf{X}^{m}\in\mathbb{R}^{|\mathcal{Z}^{m}|\times(|\mathcal{V}^{m}|d)}$ denotes the output matrix of $\mathcal{F}(\cdot)$ on $\mathcal{Z}^{m}$ whose $j$-row is $\mathbf{x}^{m}_{j}$=$\sqrt{d_{j}}\mathcal{F}(\mathbf{Z}^{m})_{j}\in\mathbb{R}^{|\mathcal{V}^{m}|d}$. Note that both $\mathcal{F}_{p}(\cdot)$ and $\mathcal{F}(\cdot)$ are normalized.

According to Eq. (9), it is evident that the MLFC scheme aims to minimize $\mathcal{L}_{M}$ by conducting an implicit similarity-based alignment between connected visible and masked samples in the latent space through an autoencoder-style predictor. Intuitively, as illustrated in Fig. 3(b), we consider a pair of second-order neighbor visible views (e.g., $\mathbf{x}^{v}_{i}$ and $\mathbf{x}^{v}_{i+1}$) that share a common complementary masked view (e.g., $\mathbf{x}^{m}_{j}$). By enforcing $\mathbf{x}^{v}_{i}$, $\mathbf{x}^{v}_{i+1}\in\mathbf{X}^{v}$ to recover $\mathbf{x}^{m}_{j}\in\mathbf{X}^{m}$ simultaneously, the latent features of visible views are implicitly correlated through the MLFC scheme. In other words, the second-order neighbors act as positive sample pairs that are pulled closer as the instance alignment (i.e., positive samples should remain close in the latent space) in contrastive learning. From this perspective, the proposed MLFC scheme serves as a hidden regularization that helps encode discriminative features, thereby promoting greater information encoding capability for better downstream performance.

To better understand the superiority of RARE compared to its competitors, we conduct a comparison of the self-supervision mechanism between the proposed RARE and state-of-the-art (SOTA) GraphMAE [9] via a toy example illustrated in Fig. 3. As previously mentioned, completing masked content within a masked graph can be regarded as a complementary view pairing problem between the visible and masked data. To solve this problem, GraphMAE [9] adopts an explicit self-supervision (i.e., Exp.-SS) mechanism by matching predicted nodes with raw ones directly, which may mislead the model into a local structural ambiguity situation. For example, as shown in Fig. 3(c), two masked samples (e.g., Node 4 and Node 5) belonging to different categories share a common visible sample (e.g., Node 1). By enforcing Node 1 to reconstruct Node 4 and Node 5, GraphMAE subconsciously reduces their distance in the latent space. Consequently, the model may struggle to differentiate Node 4 and Node 5 accurately in unsupervised scenarios. This is because the self-supervision signals provided by GraphMAE only preserve raw data information that is insufficient to ascertain whether two nodes belong to the same category or not. In contrast, RARE employs both implicit and explicit self-supervision mechanisms for masked content recovery by performing a joint mask-then-reconstruct strategy in both latent feature and raw data spaces. Particularly, in RARE, the MLFC scheme could provide more informative high-order sample correlations as implicit self-supervision signals, which are not readily available in the raw data space. As shown in Fig. 3(d), by incorporating prior knowledge that 1) Node 3 or Node 6 is closely related to Node 4 or Node 5; and 2) Node 3 and Node 6 are unconnected, it would become easier for the model to infer the relationship between Node 4 and Node 5. As a result, the two nodes are pushed apart from each other in the latent space. Empirical results in Section IV-G support our claim that the proposed RARE indeed works better than GraphMAE by making the learned samples belonging to different categories more distinguishable.

We conduct experiments to compare the proposed RARE with several baseline methods on seventeen datasets in total, including seven node classification datasets (i.e., Cora, Citeseer, Pubmed, WikiCS, Corafull, Flickr, and Yelp [52]), seven graph classification datasets (i.e., IMDB-B, IMDB-M, PROTEINS, COLLAB, MUTAG, PTC-MR, and NCI1 [53]), and three image datasets (i.e., Usps [54], Mnist [55], and Fashion-mnist [56]). Please note that all graph datasets contain both the raw attribute matrix and the adjacency matrix, while all image datasets only have the attribute matrix.

The learning procedure of RARE mainly includes two steps: 1) in the pre-training task, all nodes of datasets except for Flickr and Yelp are fed into the proposed RARE for at least 20 training iterations by minimizing Eq. (5). Since Flickr and Yelp are commonly used for inductive evaluations, we follow the public data split as GraphSAINT [57], where 50%/75%, 25%/10%, and 25%/15% nodes are randomly sampled to form the train, validation, and test sets on Flickr and Yelp, respectively; and 2) in the downstream tasks related to nodes and images, we use the publicly available train/validation/test data split for Cora, Citeseer, Pubmed, Flickr, and Yelp. For WikiCS, Corafull, Usps, Mnist, and Fashion-mnist, since these datasets have no publicly available data split, we perform a random data split where 7%, 7%, and 86% nodes are randomly sampled to form the train, validation, and test sets, respectively. We train a simple linear classifier with Adam optimizer until convergence by optimizing a cross-entropy loss by 10 times. After 10 separate runs, we report average accuracy values with standard deviations for each model. In the graph-related downstream task, as is done in GraphMAE [9], we take the support vector machine as a classifier and record the results with 10-fold cross-validation after 5 separate runs. To mitigate the adverse impact of randomness, we report average accuracy (ACC) values with standard deviations for all downstream evaluations.

To ensure a fair comparison, all experiments are conducted on the same device and under an identical configuration environment. For all compared baselines, we directly report the results listed in the existing literature if available. Otherwise, we implement their official source codes and report the reproduced performance. For our method, we perform a grid search to select hyper-parameters on the following searching space: the mask ratio $r$ is selected between {0.5, 0.75}; the balanced coefficient $\alpha$ is searched from 1 to 9; the scale factor $t$ is selected between {1, 2}; the hidden size of latent features is selected from {256, 512, 1024}; the number of feature extractor layers is selected from {1, 2, 3, 4, 5}; by default, the momentum rate is empirically fixed to 0.1 by default; the learning rate of the Adam optimizer is selected from {1.5e-4, 5e-4, 1e-3}; the maximum epoch is determined according to the results of model convergence. Particularly, in the graph classification task, we 1) choose the batch size from {32, 64}, similar to GraphMAE [9]; 2) consistently adopt a batch normalization operation to regularize the model learning; and 3) follow the sample pooing setups in GraphMAE, where a non-parameterized graph pooling function (e.g., max-pooling, mean-pooing or sum-pooling) is employed to generate graph-level representations. Please note that we employ similar hyper-parameter setups as reported in GraphMAE [9], and most hyper-parameters are not carefully tuned for ease of model learning.

In this work, we compare the performance of the proposed RARE with several baseline methods on three different tasks: node classification, graph classification, and image classification. For node classification, we compare RARE with two supervised methods (i.e., GCN [21] and GAT [22]) and ten self-supervised methods (i.e., GAE [36], DGI [28], MVGRL [31], GRACE [32], BGRL [47], InfoGCL [58], CCA-SSG [34], SeeGera [13], MaskGAE [8], and GraphMAE [9]). For graph classification, we consider twelve baseline methods from three different aspects: 1) two supervised methods (i.e., GIN [23] and DiffPool [59]); 2) two graph kernels-based methods (i.e., WL [60] and DGK [53]); and 3) eight self-supervised methods (i.e., Graph2vec [61], Infograph [33], GraphCL [29], JOAO [62], GCC [30], MVGRL [31], InfoGCL [58], and GraphMAE [9]). For image classification, we investigate two image classification methods (i.e., VGG16 [63] and ResNet18 [64]) and three graph autoencoder methods (i.e., GAE [36], MaskGAE [8], and GraphMAE [9].

As shown in Table II, we report the node classification performance of thirteen compared methods on seven datasets. From these results, several major observations can be concluded: 1) the proposed RARE consistently outperforms two supervised methods on all datasets, with margins going up to 7.7%-14.8% on Flickr and Yelp. These improvements demonstrate the great potential of masked graph autoencoders for effectively handling massive unlabeled graph data; 2) InfoGCL is one of the strongest contrastive self-supervised methods, while the proposed RARE improves it by 0.7%, 0.6%, 2.7% accuracy on Cora, Citeseer, and Pubmed, respectively. This phenomenon indicates that RARE can effectively boost the learned representations by conducting a mask-then-reconstruct mechanism instead of relying on a relatively complicated contrastive mechanism; 3) taking the results on WikiCS for example, RARE significantly outperforms SeeGera, MaskGAE, and GraphMAE by 3.2%, 3.0%, and 3.3%, respectively. These benefits are attributed to the novel idea of integrating implicit and explicit self-supervision mechanisms to drive model learning by performing a joint mask-then-reconstruct strategy in both latent feature and raw data spaces; and 4) on Cora, RARE achieves competitive results compared to the most powerful masked graph autoencoder, i.e., GraphMAE. However, it is possible that the full potential of model optimization was not demonstrated due to the insufficient size of the test dataset. Increasing the size of the test dataset, such as WikiCS and Yelp, could reveal that RARE has the potential to yield even more substantial performance gains.

Table III summarizes graph classification results of thirteen methods on seven datasets. The results reveal several key observations that are similar to those obtained from the node classification task: 1) the performance of RARE is highly competitive compared to both supervised methods and graph kernels methods, indicating that the masked graph autoencoder has the potential to be a promising alternative for self-supervised graph pre-training; 2) compared to GraphCL and JOAO, our method achieves significant performance improvements (up to 1.8%-11.7%) over them on almost all datasets. However, some contrastive learning methods, such as InfoGCL and MVGRL, demonstrate better performance than masked graph autoencoders on MUTAG and PTC-MR. This may be because in some cases, the partitioning of small-scale graph data can be easily achieved through multi-view contrastive learning; and 3) RARE achieves an approximate 1.1% average performance gain over GraphMAE on all datasets, which further indicates that RARE can effectively leverage both implicit and explicit self-supervision signals to improve the quality of the learned graph embedding.

To verify the superiority of RARE in-depth, Table IV reports the image classification performance of six methods on three datasets. From those results, we can obtain the following observations: 1) the proposed RARE shows a significant advantage against existing state-of-the-art MGAEs and other baselines on all image benchmarks. For example, on Mnist and Fashion-mnist datasets, RARE consistently outperforms the best edge-masking-based MaskGAE and node-masking-based GraphMAE by 7.1%/2.5% and 7.2%/5.9% accuracy, respectively. These improvements once again demonstrate the effectiveness of introducing implicit self-supervision signals for model learning; and 2) it is interesting to note that RARE can achieve competitive or slightly better results than typical supervised classification methods, such as VGG16 and ResNet18. This implies that improving the reliability of the self-supervision mechanism can facilitate RARE to unleash its potential for SGP. Thus, the learned representations show good robustness and generalization across a wide range of downstream tasks.

To demonstrate the effectiveness of the proposed masked latent feature completion scheme, we compare RARE with its three variants on eight datasets. Concretely, “w/o-Pred.” implies that RARE removes the latent feature predictor. “w/o-Mome.” indicates that RARE discards the momentum graph encoder. “w-$\mathcal{F}_{m}$($\mathcal{G}$)” denotes that the momentum graph encoder of RARE accepts $\mathcal{G}$ rather than $\mathcal{G}^{m}$. As shown in Table V, some major observations can be summarized: 1) when compared to “w/o-Pred.”, the latent feature predictor produces a performance gain of 1.4%-5.2% on eight datasets, indicating that this component plays a vital role in our SGP solution. By iteratively conducting the mask-then-reconstruct operation on incomplete graphs, the latent feature predictor could be regarded as a hidden regularization that assists the graph encoder in extracting more compressed features; 2) RARE consistently outperforms “w/o-Mome.” on all eight datasets. Taking the results on Corafull and IMDB-M for example, RARE achieves 3.6% and 2.3% accuracy gains respectively, demonstrating the effectiveness of providing implicit self-supervision signals to ensure the integrity and accuracy of predicted representations. Similar observations can be concluded from the results on other datasets; and 3) although “w-$\mathcal{F}_{m}$($\mathcal{G}$)” can also achieve competitive performance, it suffers from 0.3%-1.3% accuracy degradation compared to our method. The reason behind this may be that since $\mathcal{G}^{v}$ is a sub-graph of the original graph $\mathcal{G}$, a large amount of redundant information between two-source encoded representations would overwhelm the latent space, resulting in inferior representations for downstream tasks.

In this subsection, we conduct ablation studies to investigate the effect of different loss functions. Table VI reports the accuracy results of RARE and its four variants on WikiCS, Corafull, IMDB-M, and MUTAG. $\mathcal{L}_{M}$ and $\mathcal{L}_{R}$ indicate the loss functions used for implicit and explicit self-supervision mechanisms, respectively. Moreover, MAE, MSE, and ISCE are abbreviations for mean absolute error, mean square error, and improved scaled cosine error, respectively. From the results presented in Table VI, we can observe that 1) our method achieves better performance than method (A) and method (B) by 2.8%/2.5% and 2.1%/3.1% accuracy improvements on WikiCS and IMDB-M, respectively. The reason behind this is that the MSE loss is better at modeling detailed information from the data itself, while the ISCE loss focuses more on estimating the similarity between two entities. Therefore, minimizing MSE in the noisy raw data space may mislead the network to overly preserve redundant graph details, which may not always result in the required expressive encoding capability for downstream tasks; and 2) RARE and method (D) consistently outperform method (C) by 0.8%/0.6%, 1.4%/0.7%, 1.7%/1.8%, and 1.4%/0.4% on WikiCS, Flickr, IMDB-M, and MUTAG, respectively. This is because the latent features contain much category-related information that needs to be carefully preserved. As a result, completing the masked content in the latent space with MAE or MSE contributes more to the performance than that with ISCE. Moreover, when one has to choose a loss function for masked latent feature completion, both MAE and MSE are suitable for guiding the model learning implicitly.

To investigate the effect of different data-masking setups, we conduct ablation studies to make a comparison among RARE and its four variants on Pubmed, Flickr, IMBD-M, and NCI1. Specifically, “M-Node (Z)” and “M-Rep. (Z)” means that nodes and representations are masked with zero values, and “M-Node (T)” and “M-Rep. (T)” means that nodes and representations are masked with tokens (i.e., stochastic learnable vectors). “w/o M-Rep.” denotes a variant of RARE without masking representations. Some major conclusions can be summarized from the results in Table VII: 1) our method consistently achieves better accuracy performance than method (A), method (B), and method (C) on four datasets, indicating that using parameterized tokens for masked samples as initialization is more effective; and 2) the last two rows reveal the advantage of implementing the data-masking operation over latent features. It is worth noting that method (D) refines the encoded representations of masked nodes via latent feature matching directly, while RARE first recovers the masked representations from scratch via a latent feature predictor and then conducts the sample matching in the high-order latent feature space. The latter makes the self-supervised task more challenging, which further encourages the model to enhance its local modeling capability. As seen, RARE consistently outperforms method (D) by 1.8%, 1.6%, 0.9%, and 1.8% accuracy on all datasets.

To investigate how different latent feature predictor setups influence the performance of RARE, we compare RARE with its three variants and report their performance on Pubmed, Corafull, PROTEINS, and PTC-MR in Table VIII. The abbreviations used in the table are as follows: “GCL”, “GAL”, “GIL”, and “MLP” denote the graph convolution layer, graph attention layer, graph isomorphism layer, and multilayer perception, respectively. Firstly, we observe that our method produces better performance than method (C) on all datasets, indicating that GAL (or GIL) is a better option than GCL for recovering masked latent features due to its more powerful graph modeling capability. Similar observations can be obtained from the comparison between method (A) and method (B). Secondly, we also observe that our method consistently outperforms method (B) by 1.6%, 4.6%, 1.1%, and 1.8% on all datasets. Similarly, method (C) performs better than method (A) in almost all cases. These results demonstrate that MLP plays an important role in predicting the latent features of masked nodes in the MLFC scheme.

To further illustrate the superiority of RARE, we investigate its performance variation with respect to different mask ratios. Concretely, we pre-train RARE by varying the mask ratio $r$ from 0.1 to 0.9 with a step size of 0.1. From the results shown in Fig. 5(a), some key observations can be obtained: 1) taking the results on PROTEINS for example, continually increasing the value of $r$ first improves the model accuracy and then leads to relatively poor performance. This indicates that the masking-then-reconstructing mechanism is indeed effective for self-supervised graph pre-training, but a proper $r$ is required to balance the visible and masked information; 2) increasing $r$ by more than 0.6 would cause a performance drop in most cases, while the proposed RARE can still perform well within a wide range of high mask ratios. For example, the optimal masked ratio (i.e., 90%) for MUTAG is surprisingly high, indicating that in some cases, the model with a high mask ratio largely eliminates redundancy and thus yields a nontrivial and meaningful self-supervision task; and 3) the performance of RARE is relatively stable across a wide range of $r$ on COLLAB. This once again implies that RARE can also learn useful representations with limited observed information, indicating its potential to achieve a good accuracy-efficiency trade-off.

Eq. (5) introduces a hyper-parameter to balance the importance of two loss functions. To show its influence in-depth, Fig. 5(b) presents the accuracy variation of RARE on three datasets when $\alpha$ varies from 1 to 10 with a step size of 1. Our observations from this sub-figure are as follows: 1) the effect of tuning $\alpha$ on model performance varies across different datasets, but the stability of the model performance is higher in the range of [5, 10]. This indicates that searching $\alpha$ from a reasonable hyper-parameter region could positively influence the model performance; 2) the accuracy variation is relatively stable across a wide range of $\alpha$ on IMDB-B and WikiCS, while on Cora, it shows a trend of first rising and then dropping slightly. This suggests that RARE requires a suitable $\alpha$ to ensure the quality of learned representations when reconstructing the raw attributes; and 3) RARE tends to perform well by setting $\alpha=6$ according to the results on all datasets.