Inductive Representation Learning in Temporal Networks via Causal Anonymous Walks

Anonymous walks were first considered by Micali & Zhu (2016) to study the reconstruction of a Markov process from the records without sharing a common “name space”. AWs can be directly rephrased in the network context. Specifically, an AW starts from a node, performs random walks over the graph to collect a walk of nodes, e.g. $(v_{1},v_{2},...,v_{m})$. AW has an important anonymization step to replace the node identities by the orders of their appearance in each walk, which we term relative node identities in AW and define it as

Note that the set operation above removes duplicated elements. Although AW removes node identities, those nodes are still distinguishable within this walk. Therefore, an AW can be viewed as a network motif while the information on which specific nodes form this motif is removed. Examples of AWs are shown as follows. While these are two different walks, they may be mapped to the same AW when node identities get removed.
                     

We propose CAW that shares the high-level concept with AW to remove the original node identities. However, CAW has a different causal sampling strategy and a novel set-based approach for node anonymization, which are specifically designed to encode temporal network dynamics (Fig. 2).

Causality Extraction. Alg. 1 shows our causal sampling: We sample connected links by backtracking over time to extract the underlying causality of network dynamics. More recent links may be more informative and thus we introduce a non-negative hyper-parameter $\alpha$ to sample a link with a probability proportional to $\exp(\alpha(t-t_{\text{p}}))$ where $t,t_{\text{p}}$ are the timestamps of this link and the link previously sampled respectively. A large $\alpha$ can emphasize more on recent links while zero $\alpha$ leads to uniform sampling. In Sec.4.4, we will discuss an efficient sampling strategy for the step 5 in Alg.1, which avoids computing those probabilities by visiting the entire historical links.

Then, given a link $\{u_{0},v_{0}\}$ and a time $t_{0}$, we use Alg. 1 to collect $M$ many $m$-step walks starting from both $u_{0}$ and $v_{0}$, and record them in $S_{u}$ and $S_{v}$ respectively. For convenience, a walk $W$ from a starting node $w_{0}\in\{u,v\}$ can be represented as Eq.1.

Set-based Anonymization. Based on $S_{u}$ and $S_{v}$, we may anonymize each node identity $w$ that appears on at least one walk in $S_{u}\cup S_{v}$ and design relative node identity $I_{CAW}(w;\{S_{u},S_{v}\})$ for $w$. Our design has the following consideration. $I_{AW}$ (Eq.2) only depends on a single path, which results from the original assumption that any two AWs do not even share the name space (i.e., node identities) (Micali & Zhu, 2016). However, in our case, node identities are actually accessible, though an inductive model is not allowed to use them directly. Instead, correlation across different walks could be a key to reflect laws of network dynamics: Consider the case when the link $\{u,v\}$ happens only if there is another node appearing in multiple links connected to $u$ (Fig. LABEL:fig:AW-issue). Therefore, we propose to use node identities to first establish such correlation and then remove the original identities.

$g(w,S_{w_{0}})[i]\triangleq|\{W|W\in S_{w_{0}},w=W[i][0]\}|$ for $i\in\{0,1,...,m\}$

Specifically, we define $I_{CAW}(w;\{S_{u},S_{v}\})$ as follows: For $w_{0}\in\{u,v\}$, let $g(w,S_{w_{0}})\in\mathbb{Z}^{m+1}$ count the times in $S_{w_{0}}$ node $w$ appears at certain positions, i.e., $g(w,S_{w_{0}})[i]\triangleq|\{W|W\in S_{w_{0}},w=W[i][0]\}|$ for $i\in\{0,1,...,m\}$. Further, define

Essentially, $g(w,S_{u})$ and $g(w,S_{v})$ encode the correlation between walks within $S_{u}$ and $S_{v}$ respectively and the set operation in Eq.3 establishes the correlation across $S_{u}$ and $S_{v}$. For the case in Fig.3, suppose $S_{a},S_{b},S_{a^{\prime}},S_{b^{\prime}}$ include all historical one-step walks (before $t_{3}$) starting from $a,b,a^{\prime},b^{\prime}$ respectively. Then, it is easy to show that $I_{CAW}(a;\{S_{a},S_{b}\})\neq I_{CAW}(a^{\prime};\{S_{a^{\prime}},S_{b}\})$ that allows differentiating $a$ and $a^{\prime}$, while TGAT (Xu et al., 2020) as discussed in Sec.2 fails. From the network-motif point of view, $I_{CAW}$ not only encodes each network motif that corresponds to one single walk as $I_{AW}$ does but also establishes the correlation among these network motifs. $I_{AW}$ cannot establish the correlation between motifs and will also fail to distinguish $a$ and $a^{\prime}$ in Fig.3. We see it as a significant breakthrough as such correlation often gets neglected in previous works that directly count motifs or adopt AW-type anonymization $I_{AW}$.

After we collect CAWs, neural networks can be conveniently leveraged to extract their structural (in $I_{CAW}(\cdot)$) and fine-grained temporal information by encoding CAWs: We will propose the model CAW-N to first encode each walk $\hat{W}$ (Eq.4) and then aggregate all encoded walks in $S_{u}\cup S_{v}$.

Encode $\hat{W}$. Note that each walk is a sequence of node-time pairs. If we encode each node-time pair and plug those pairs in a sequence encoder, e.g., RNNs, we obtain the encoding of $\hat{W}$:

where $f_{1},f_{2}$ are two encoding function on $I_{CAW}(w_{i})$ and $t_{i-1}-t_{i}$ respectively. One may use transformer networks instead of RNNs to encode the sequences but as the sequences in our case are not long $(1\sim 5)$, RNNs have achieved good enough performance. Now, we specify the two encoding functions $f_{1}(I_{CAW}(w_{i}))$ and $f_{2}(t_{i-1}-t_{i})$ as follows. Recall the definition of $I_{CAW}(w_{i})$ (Eq.3).

Here the encoding of $I_{CAW}(w_{i})$ adopts the sum-pooling as the order of $u,v$ is not relevant. For $f_{2}(t)$, we adopt random Fourier features to encode time (Xu et al., 2019; Kazemi et al., 2019) which may approach any positive definite kernels according to the Bochner’s theorem (Bochner, 1992).

Encode $S_{u}\cup S_{v}$. After encoding each walk in $S_{u}\cup S_{v}$, we aggregate all these walks to obtain the final representation $\text{enc}(S_{u}\cup S_{v})$ for prediction. We suggest to use either mean-pooling for algorithmic efficiency or self-attention (Vaswani et al., 2017) followed by mean-pooling to further capture subtle interactions between different walks. Specifically, suppose $\{\hat{W}_{i}\}_{1\leq i\leq 2M}$ are the $2M$ CAWs in $S_{u}\cup S_{v}$ and each $\text{enc}(\hat{W}_{i})\in\mathbb{R}^{d\times 1}$. We set $\text{enc}(S_{u}\cup S_{v})$ as

• •

  Mean-AGG($S_{u}\cup S_{v}$): $\frac{1}{2M}\sum_{i=1}^{2M}\text{enc}(\hat{W}_{i})$.

• •

  Self-Att-AGG($S_{u}\cup S_{v}$): $\frac{1}{2M}\sum_{i=1}^{2M}\text{softmax}(\{\text{enc}(\hat{W}_{i})^{T}Q_{1}\text{enc}(\hat{W}_{j})\}_{1\leq j\leq n})\text{enc}(\hat{W}_{i})Q_{2}$ where $Q_{1},Q_{2}\in\mathbb{R}^{d\times d}$ are two learnable parameter matrices.

We add a 2-layer perceptron over $\text{enc}(S_{u}\cup S_{v})$ to make the final link prediction.

Attributed nodes/links and directed links. In some real networks, nodes or links may have attributes available, e.g., the message content in the case of SMS networks. In this case, the walk in Eq.1 can be associated with node/link attributes $X_{0},X_{1},...,X_{m}$ where $X_{i}$ refers to the attributes on link $(\{w_{i-1},w_{i}\},t_{i})$ or on the node $w_{i}$ or a concatenation of these two parts. Note that the direction of a link can also be viewed as a binary link attribute, where a 2-dimensional one-hot encoding can be used. To incorporate such information, we only need to change $\text{enc}(\hat{W})$ (Eq.5) as

The derived encoding is then concatenated with $\text{enc}(\hat{W})$ to obtain the enhanced final encoding of $\hat{W}$.

Efficient link sampling. A naive implementation of the link sampling in step 5 in Alg.1 is to compute and normalize the sampling probabilities of all links in $\mathcal{E}_{w_{\text{p}},t_{\text{p}}}$, which requires to memorize all historical links and costs much time and memory. To solve this problem, we propose a sampling strategy (Appendix A) with expected time and memory complexity $\min\{\frac{2\tau}{\alpha}+1,|E_{w_{\text{p}},t_{\text{p}}}|\}$ if links in $\mathcal{E}_{w_{\text{p}},t_{\text{p}}}$ come in by following a Poisson process with intensity $\tau$ (Last & Penrose, 2017). This means that our sampling strategy with a positive $\alpha$ allows the model only recording $O(\frac{\tau}{\alpha})$ recent links for each node instead of the entire history. Our experiments in Sec.5.3 show that $\alpha$ that achieves the best prediction performance makes $\frac{\tau}{\alpha}\approx$5 in different datasets. Since the time and memory complexity do not increase with respect to the number of links, our model can be used for online training and inference. Note that the $\alpha=0$ case reduces to uniform sampling, mostly adopted by previous methods (Xu et al., 2020), which requires to record the entire history and thus is not scalable.

CAW-N Variants. We test CAW-N-mean and CAW-N-attn which uses mean and attention pooling respectively to encode $S_{u}\cup S_{v}$ (Sec. 4.3). Their code is provided in the supplement.

Baselines. Our method is compared with six previous state-of-the-art baselines on representation learning of temporal networks. They can be grouped into two categories based on their input data structure: (1) Snapshot-based methods, including DynAERNN (Goyal et al., 2020), VGRNN (Hajiramezanali et al., 2019) and EvolveGCN (Pareja et al., 2020); (2) Stream-based methods, including TGAT (Xu et al., 2020), JODIE (Kumar et al., 2019) and DyRep (Trivedi et al., 2019). We give their detailed introduction in Appendix C.2.2. For the snapshot-based methods, we view the link aggregation as a way to preprocess historical links. We adopt the aggregation ways suggested in their papers. These models are trained and evaluated over the same link sets as the stream-based methods.

Dataset. We use six real-world public datasets: Wikipedia is a network between wiki pages and human editors. Reddit is a network between posts and users on subreddits. MOOC is a network of students and online course content units. Social Evolution is a network recording the physical proximity between students. Enron is a email communication network. UCI is a network between online posts made by students. We summarize their statistics in Tab.1 and give their detailed description and access in Appendix C.1.

Evaluation Tasks. Two types of tasks are for evaluation: transductive and inductive link prediction.

Transductive link prediction task allows temporal links between all nodes to be observed up to a time point during the training phase, and uses all the remaining links after that time point for testing. In our implementation, we split the total time range [0, $T$] into three intervals: [0, $T_{train}$), [$T_{train}$, $T_{val}$), [$T_{val}$, $T$]. links occurring within each interval are dedicated to training, validation, and testing set, respectively. For all datasets, we fix $T_{train}$/$T$=0.7, and $T_{val}$/$T$=0.85 .

Inductive link prediction task predicts links associated with nodes that are not observed in the training set. There are two types of such links: 1) "old vs. new" links, which are links between an observed node and an unobserved node; 2) "new vs. new" links, which are links between two unobserved nodes. Since these two types of links suggest different types of inductiveness, we distinguish them by reporting their performance metrics separately. In practice, we follow two steps to split the data: 1) we use the same setting of the transductive task to first split the links chronologically into training / validation / testing sets; 2) we randomly select 10% nodes, remove any links associated with them from the training set, and remove any links not associated with them in the validation and testing sets.

Following most baselines, we randomly sample an equal amount of negative links and consider link prediction as a binary classification problem. For fair comparison, we use the same evaluation procedures for all baselines, including the snapshot-based methods.

Training configuration. We use binary cross entropy loss and Adam optimizer to train all the models, and early stopping strategy to select the best epoch to stop training. For hyperparameters, we primarily tune those that control the CAW sampling scheme including the number $M$, the length $m$ of CAWs and the time decay $\alpha$. We will investigate their sensitivity in Sec.5.3. For all baselines, we adapt their implemented models into our evaluation pipeline and extensively tune them. Detailed description of all models’ tuning can be found in Appendix C. Finally, we adopt two metrics to evaluate the models’ performance: Area Under the ROC curve (AUC) and Average Precision (AP).

We report AUC scores in Tab.2, and report AP scores in Tab.6 of Appendix D.1. In the inductive setting and especially with "new vs new" links, our models significantly outperform all baselines on all datasets. Even in transductive setting where the baselines claim their primary contribution, our approaches still significantly outperform them on four out of six datasets. Note that our models achieve almost perfect scores (i.e. $>0.98$) on Reddit and Wikpedia when the baselines are far from perfect. Meanwhile, the strongest baseline on these two attributed datasets, TGAT (Xu et al., 2020), suffers a lot on all the other datasets where informative node / link attributes become unavailable.

Comparing the two training settings, we observe that the performance of four baselines (JODIE, VGRNN, DynAERNN,DyRep) drop significantly when transiting from the transductive setting to the inductive one, as they mostly record node identities either explicitly (JODIE, VGRNN, DynAERNN) or implicitly (DyRep). TGAT and EvolveGCN do not use node identities and thus their performance gaps between the two settings are small, while they sometimes do not perform well in the transductive setting, as they encounter the ambiguity issue in Fig. 3. In contrast, our methods perform well in both the transductive and inductive settings. We attribute this superiority to the anonymization procedure: the set-based relative node identities well capture the correlation between walks to make good prediction while removing the original node identities to keep entirely inductive. Even when the network structures greatly change and new nodes come in as long as the network evolves according to the same law as the network used for training, CAW-N will always work. Comparing CAW-N-mean and CAW-N-attn, we see that our attention-based variant outperforms the mean-pooling variant, albeit at the cost of high computation complexity. Also note that the strongest baselines on all datasets are stream-based methods, which indicates that the aggregation of links into network snapshots may remove some useful time information (see more discussion in Appendix  D.2).

We further conduct ablation studies on Wikipedia (attributed), UCI (non-attributed), and Social Evolution (non-attributed), to validate effectiveness of critical components of our model. Tab. 3 shows the results. By comparing Ab.1 with Ab.2, 3 and 4 respectively, we observe that our proposed node anonymization and encoding, $f_{1}(I_{CAW})$, contributes most to the performance, though the time encoding $f_{2}(t)$ also helps. Comparing performance across different datasets, we see that the impact of ablation is more prominent when informative node/link attributes are unavailable such as with UCI and Social Evolution. Therefore, in such scenarios our CAWs are highly crucial. In Ab.5, we replace our proposed $I_{CAW}$ with $I_{AW}$ (Eq.2), which is used in standard AWs, and we use one-hot encoding of node new identities $I_{AW}$. We see by comparing Ab.1, 2, and 5 that such anonymization process is significantly less effective than our $I_{CAW}$, though it helps to some extent. Finally, Ab.6 suggests that entirely uniform sampling of the history may hurt performance.

We systematically analyze the effect of hyperparameters used in CAW sampling schemes, including sampling number $M$, temporal decay coefficient $\alpha$ and walk length $m$. The experiments are conducted on UCI and Wikipedia datasets using CAW-N-mean. When investigating each hyperparameter, we set the rest two to an optimal value found by grid search, and report the mean AUC performance on all inductive test links (i.e. old vs. new, new vs. new) and their 95% confidence intervals.

The results are summarized in Fig.5. From (a), we observe that only a small number of sampled CAWs are needed to achieve a competitive performance. Meanwhile, the performance gain is saturated as the sampling number increases. We analyze the temporal decay $\alpha$ in (b): $\alpha$ usually has an optimal interval, whose values and length also vary with different datasets to capture the different levels of temporal dynamics; a small $\alpha$ suggests an almost uniform sampling of interaction history, which hurts the performance; an overly large $\alpha$ also damages the model, since it makes the model only sample the most recent few interactions for computation and blind to the rest. Based on our efficient sampling strategy (Sec.4.4), we may combine the optimal $\alpha$ with the average link intensity $\tau$ (Tab.1), and concludes that CAW-N only needs to online record and sample from about a constant times about 5 ($\approx\frac{\tau}{\alpha}$) most recent links for each node. Plot (c) suggests that the performance may peak at a certain CAW length, while the exact value may vary with datasets. Longer CAWs indicate that the corresponding networks evolve according to more complicated laws encoded in higher-order motifs.