Temporal Aggregation and Propagation Graph Neural Networks for Dynamic Representation

Decades ago, the emerging big graph data on the website had already attracted much concern of researchers [20, 21, 22, 1, 23]. The learned low-dimensional embeddings for nodes, edges, and subgraphs can be easily integrated with the machine learning algorithms like Logistic Regression, Random Forest, and Gradient Boosting Decision Trees to perform risk monitoring [13], link prediction [23], item recommendation [1], etc. Recently, researchers interested in graph mining have been motivated by the great success of deep learning methods in computer vision [24] and natural language processing [25]. The deep graph embedding methods can also be categorized into two kinds: graph convolutional networks [26] and skip-gram models [18].

On the one hand, graph neural networks [27] are initially defined in the spectral space, which is inspired by the convolution operations defined on the image grid. GCN [16] simplifies the learning framework of [27] and shows a successful application on semi-supervised node classification tasks. GraphSAGE [17] proposes an inductive learning paradigm on large-scale graphs, which samples subgraphs for each node. Meanwhile, GAT [28] introduces a learnable attention mechanism to impose different importances over neighbors. The attention mechanism not only achieves better performance on several benchmarks but also presents an explainable result by the neighbor importances.

On the other hand, DeepWalk [14] is the pioneering work to introduce the skip-gram models [25] into graph representation learning with the sampled node sequences by random walks. Node2Vec [15] design two hyper-parameters to control the random walk preferences, while LINE [29] explores the high-order relations in the graph topology. Their shortcomings are obvious: the generated embeddings are not task-oriented, and a new model is required to deal with the additional side information. The interested readers can refer to the survey papers [18, 26] for an overview of graph embedding methods in general.

Moreover, these methods have successful applications in industrial scenarios such as commodity recommendation and risk monitoring [11, 12, 30, 31, 8]. However, these static graph embedding methods are not designed to capture node dynamics with the continuously emerging node interactions. Hence, an additional model is needed to perform real-time predictions, which may lose the evolutionary pattern in temporal graphs.

As illustrated in Fig. 1(c), discrete-time dynamic graph (DTDG) embedding methods slice the whole temporal graph into a sequence of graph snapshots. The predefined time window choice is a crucial part of DTDG methods, which causes high costs for hyper-parameter search. Intuitively, the smaller time window models the real scenarios more precisely with the increasing computation cost. As a result, DTDG methods are designed for scenarios that only need to update embeddings periodically.

DTDG methods mainly contain two parts: the first part is to capture the evolutionary patterns; the second part is to fuse the historical information with current embeddings. For the first part, DynamicTriad [32] models the probability that open triads evolve into closed triads; SPNN [33] studies high-order dependencies of the induced subgraphs; EPNE [34] incorporates both periodic and non-periodic patterns into the learned embeddings. For the second part, Know-Evolve [35], E-LSTM-D [36], and TNODE [7] use an LSTM to fuse the historical information and current information. In addition, EvolveGCN [19] learns to generate the transformation matrix for each separate graph snapshot in a recurrent paradigm. Nevertheless, DTDG methods cannot generate dynamic node embeddings to meet the requirements of continuous-time dynamic graphs.

Continuous-time dynamic graph (CTDG), which we use with temporal graph exchangeably, keeps evolving with the appearance, disappearance, and changing of edges and nodes [5]. Recent advances for dynamic node representation on temporal graphs have been summarized in [37]. While they focus on the encoder-decoder techniques, we mainly review existing solutions to the aforementioned challenges in this paper.

Methods tackling the varying neighborhood challenge are mostly recurrent models. HTNE [6] adopts the Hawkes process and an attention mechanism to model the influence of temporal neighbors; DyRep [38] discriminates the topological evolution from node interactions and introduces a two-time scale temporal point process model; JODIE [8] uses two mutually recursive RNNs for users and items separately and proposes a projection operation to predict the node trajectory; TigeCMN [39] adopts a key-value memory network to encode the network dynamics with sequentially updated operations. Differently, CTDNE [5] incorporates temporal random walks into the static skip-gram models but can only obtain the final state for each node. The aforementioned CTDG methods mainly rely on the local node embedding update with the upcoming interactions. Consequently, the influence of historical neighbors in these models will gradually vanish, and high-order neighbors are neglected by these models, which have been proved useful in static GNNs [11, 31]. Our proposed TAP-GNN treats the historical neighbors according to their contributions to the generated embeddings and can be easily extended to obtain the high-order temporal neighbors.

In contrast, CTDG methods based on GNNs are naturally designed for the high-order neighborhood on temporal graphs but cannot capture the whole neighborhood due to the neighbor dropout technique. Hence, previous works on GNNs can only partially inject temporal information and perform online inference. TGAT [9] and APAN [10], which are the most related works to our paper, construct message-passing subgraphs for each dynamic node inspired by GraphSAGE [17]. Detailedly, TGAT [9] proposes a novel temporal encoding kernel and implements an attention-based graph convolution model. However, TGAT suffers from the exponentially expanding neighborhood and the information loss of the unseen neighbors. APAN makes substantial improvements over TGAT in consideration of online inference latency but also inherits the restricted neighbor size. Therefore, they both are unable to access the information from the whole temporal neighbors.

A temporal graph can be defined as $G=(V,E,\mathcal{T})$, where $E$ is extended as a set of temporal edges, and $\mathcal{T}:E\rightarrow\mathbb{R}^{+}$ is a function that maps each edge to a corresponding timestamp[5]. A common assumption is that edges in temporal graphs are formed according to some underlying distribution [14, 5, 40]. The neighbors of node $v$ at time $t$ can be drawn from $u\sim p(\cdot|v,t)$ in temporal graphs. In general, our target of graph embedding is to learn an embedding function revealing the underlying data distribution. Let $h(\cdot,t)$ be the dynamic node embedding function, and the dot product be the similarity function. The dynamic node representation problem can be written as

where $(u,v,t)$ is an observed interaction, $t$ is the time point, and $u^{\prime}$ iterates all nodes of $V$. In practice, the negative sampling loss [25, 40] is usually adopted instead as Eq. 1 to accelerate the training speed.

Existing Graph Neural Networks (GNNs) mainly follow the message-passing paradigm that aggregates neighborhood embeddings recursively to obtain high-order neighborhood information. Formally, the graph convolution operation is defined as

in [16, 17], where $h(\cdot)$ is an embedding function, $\mathcal{N}(\cdot)$ is the neighborhood function, $g^{\star}(\cdot)$ is the graph convolution kernel. The detailed $g^{\star}(\cdot)$ kernels can be listed as follows:

where $h(u)$ is written as $h_{u}$, $\deg(\cdot)$ denotes the degree of nodes, $\sigma(\cdot)$ is a non-linear activation function, $W$ is a transformation matrix, mean and max are reduction operations. The attention kernel is provided in Appendix .2.

Although $h(v,t)$ requires dynamic representation for node $v$ with the varying $t$, the representation should be related to the limited node interactions. The observation motivates us to define the temporal nodes restricted to the observed time points. Representation for the infinite time dimension can be solved with an additional projection layer. Let $T_{K}=\{t_{k}|(u,v,t_{k})\in E,u\in V,v\in V,t_{k}\in\mathbb{R}^{+}\}$ denotes the set of observed time points. For simplicity, we model the node dynamics with the observed $T_{K}$ as:

For temporal nodes, it’s worth noting that the set of temporal nodes $V^{T}$ is not the cartesian product of nodes $V$ and $T_{K}$ since $v_{t_{l}}\notin V^{T}$ if there are no interactions of $v$ at $t_{l}$. Nodes are only connected with their active time points, as shown in Fig. 1(b). Since temporal graphs may have parallel and dynamic edges between nodes, the spectral graph convolution way [16] is not feasible in our settings.

Generally speaking, the definition of temporal neighbors is suitable for any dynamic node $(v,t)$ even if $t$ never appears in the observed $T_{K}$. It means that we can generate dynamic node representation for any $v_{t}$ as long as node embeddings on $V^{T}$ are obtained. Moreover, the defined temporal neighbors are associated with each other via directional links, i.e. $u_{t_{l}}\in\mathcal{TN}(v_{t_{k}})$ doesn’t imply $v_{t_{k}}\in\mathcal{TN}(u_{t_{l}})$ except $t_{l}=t_{k}$. A simple corollary is that temporal neighbors of one node are also temporal neighbors of it in the future, i.e., $\mathcal{TN}(v,t_{j})\subseteq\mathcal{TN}(v,t_{i})$ when $t_{j}\leq t_{i}$.

So far, we can build the message-passing temporal graph (MPTG) as $G_{MP}=(V^{T},E^{T})$, where $V^{T}$ is the set of temporal nodes, and $E^{T}=\{(u_{t_{l}},v_{t_{k}})|u_{t_{l}}\in\mathcal{TN}(v_{t_{k}}),v_{t_{k}}\in V^{T}\}$ is the set of directional links between temporal nodes. The messages of the temporal neighbors can thus be sent to the destination nodes on the $G_{MP}$, i.e., message passing operations on temporal graphs. High-order temporal neighbors can be accessed by stacking layers of the $G_{MP}$ as described in [16, 17, 41]. For instance, if we focus on V and its one-hop neighbor {A,F} in Fig. 1(d), we will get three temporal nodes $V^{T}=\{\text{V}_{t_{1}},\text{V}_{t_{4}},\text{A}_{t_{1}},\text{F}_{t_{4}}\}$. At the time $t_{1}$, the edge set $E^{T}$ contains one temporal edge $\text{V}_{t_{1}}\leftarrow\text{A}_{t_{1}}$. At the time $t_{4}$, the edge set $E^{T}$ contains two temporal edges $\{\text{V}_{t_{4}}\leftarrow\text{A}_{t_{1}},\text{V}_{t_{4}}\leftarrow\text{F}_{t_{4}}\}$, where $\text{V}_{t_{4}}$ shares a same edge with $\text{V}_{t_{1}}$ at $t_{1}$. The overall $E^{T}$ is the union set of the edge set at each time point, namely $\{\text{V}_{t_{1}}\leftarrow\text{A}_{t_{1}},\text{V}_{t_{4}}\leftarrow\text{A}_{t_{1}},\text{V}_{t_{4}}\leftarrow\text{F}_{t_{4}}\}$.

However, the alternative MPTG based on the temporal graph may incur the computation complexity problem because of the repeated links with temporal neighbors. We provide an analysis of the cardinality of temporal nodes $V^{T}$ and directional links $E^{T}$ as follows.

Claim 3.3 implies that the size of temporal nodes increases linearly with the dynamic interactions on temporal graphs. A new temporal node is created when each new interaction is marked with a unique timestamp.

Claim 3.4 indicates that directional links expand rapidly with the increasing dynamic interactions. Table I also shows the surprising result on experimental datasets, where ratios of small graphs still exceed $50$, while the largest expansion ratio even reaches $8,537$. The resulting message-passing graph $G_{MP}=(V^{T},E^{T})$ may not fit the main memory for medium graphs. Therefore, we have to design a lightweight mechanism to replace the burdensome $G_{MP}$.

The proposed block comes from an observation that historical node embeddings already contain the information of historical neighbors. Let’s take a single node $v$ and its consecutive active time points $t_{i},t_{j}(i>j)$ into account. Here $t_{i}$ and $t_{j}$ only represent the relative sequential order. Referring to the Definition 3.2 $\mathcal{TN}(\cdot)$, $\mathcal{TN}(v_{t_{j}})$ is a subset of $\mathcal{TN}(v_{t_{i}})$. Hence, the difference set of the two sets is $\{u_{t_{i}}|(u,v,t_{i})\in E\}$, which is only related with $t_{i}$. Let $\mathcal{DN}(v_{t_{i}})$ denote the difference set, then $\mathcal{TN}(v_{t_{i}})=\mathcal{TN}(v_{t_{j}})\cup\mathcal{DN}(v_{t_{i}})$ and $\mathcal{TN}(v_{t_{j}})\cap\mathcal{DN}(v_{t_{i}})=\emptyset$. It indicates that the aforementioned temporal graph convolution can be decomposed sequentially as long as it doesn’t involve commutative computations between $\mathcal{TN}(t_{j})$ and $\mathcal{DN}(v_{t_{i}})$. Fortunately, we find that the commonly adopted graph convolution kernels on static graphs fit this property.

Thus, the Aggregation-Propagation (AP) block, as depicted in Fig. 2, is designed to save the repeated computation in $G_{MP}$. We illustrate the overall workflow of the proposed AP block before going into the technical decomposition details of several graph convolution methods. The AP block consists of two sequential layers, termed aggregation layer and propagation layer, which refer to AGG and PROP used in Algo. 1, respectively. The aggregation layer is a lightweight message-passing graph and written as $G_{\text{AGG}}=(V^{T},{O^{T}})$, where ${O^{T}=\{(u_{t_{i}},v_{t_{i}})|(u,v,t_{i})\in E\}}$ connects the temporal nodes only at the same time point. It works in a message-passing way that sends the neighbor information to the temporal nodes like $A_{t_{1}}\rightarrow V_{t_{1}};F_{t_{4}}\rightarrow V_{t_{4}}$. The propagation layer reuses the historical node embeddings to compute the temporal node embeddings and works with a linked list $L^{T}=\{\text{sorted}(\{v_{t_{i}}|v_{t_{i}}\in V^{T}\}),\forall v\in V\}$, which organizes the temporal nodes $v_{t_{i}}$ referring to the same origin node $v$ into a sorted list. It accumulates the node embeddings chronologically as $V_{t_{1}}\rightarrow V_{t_{4}}$.

Let $\textbf{H}(V^{T})$ be the embeddings of temporal nodes. Formally, we combine AGG and PROP for a node $v$ to obtain the same embeddings as graph kernels in Eq. 3 regardless of time points, written as

where AGG aggregates neighborhood embeddings and PROP summarizes historical embeddings. Let $v_{t_{i}},v_{t_{j}}(i>j)$ be the consecutive active time points of node $v$, the AGG operation is written as.

where $h_{u}$ is the node embedding of the previous layer, and $\mathcal{DN}(\cdot)$ refers to the neighbor set of the current time point. The PROP operation is then written as

where $h(v_{t_{j}})$ is $v_{t_{j}}$’s embedding of PROP, and PROP performs on $v_{t_{j}},v_{t_{i}}$ sequentially.

Suppose that $u\in\mathcal{DN}(v_{t_{i}})$, the decomposition methods are listed as follows, where the attention kernel is provided in Appendix B.

GCN. Aggregation: Let $a(v_{t_{i}})=\sum_{u}\frac{h_{u}}{\sqrt{\deg(u)}}$. Propagation: $h(v_{t_{i}})=\frac{a(v_{t_{i}})+h(v_{t_{j}})\cdot\sqrt{\deg(v_{t_{j}})}}{\sqrt{\deg(v_{t_{i}})}}$, where $\deg(\cdot)$ denotes the indegree of nodes.

MEAN. Aggregation: $a(v_{t_{i}})=\sum_{u}h_{u}$. Propagation: $h(v_{t_{i}})=\frac{a(v_{t_{i}})+h(v_{t_{j}})\cdot\deg(v_{t_{j}})}{\deg(v_{t_{i}})}$.

POOL. Aggregation: $a(v_{t_{i}})=\text{max}\{h_{u}\}$. Propagation: $h(v_{t_{i}})=\text{max}({a(v_{t_{i}}),h(v_{t_{j}}}))$.

Complexity analysis. Let $N$ be the size of nodes, and $M$ be the size of temporal edges. The AP block shares the same node set $V^{T}$ with $G_{MP}$ and preserves the links between nodes at the same time point, whose size is $M$. Suppose each node has $d$-dimension embeddings, the space complexity of the AP block is $\mathcal{O}(Md)$ since the size of $V^{T}$ is $M$. The time complexity of the aggregation layer is $\mathcal{O}(Md)$ because the size of the preserved links is $M$. The time complexity of the propagation layer is also $\mathcal{O}(Md)$ because it propagates embeddings over temporal nodes. But it requires more time than the aggregation layer in practice since the propagation cannot be fully paralleled. Overall, the time complexity of the AP block is $\mathcal{O}(Md)$, where the feature transformation operations are omitted here.

The whole process is listed in Algorithm 1. The input includes a temporal graph $G(V,E,\mathcal{T})$, features for all nodes $x_{v},\forall v\in V$, and a batch of queries $(v,t)$ asking for dynamic node representation.

The additional edge features are concatenated with node features for input, as illustrated in Fig. 3. The learning backbone consists of $K$-layer AP blocks and a final projection layer, where $K$ denotes the iterations of temporal graph aggregation-propagation. Inspired by [8, 9], a temporal activation function $\mathcal{A}$ is applied before each AP block to inject the temporal information into the node embeddings as

where $h_{v}$ is the hidden representation of node $v$, $t_{i}$ is the corresponding time point, $Wt_{i}+b$ encodes the temporal information, and $\cos(\cdot)$ may capture the repeated patterns of interactions. The temporal activation function provides distinct embeddings even though some node has few interactions. The AP block described in Fig. 2 works with the help of the proposed temporal nodes $V^{T}$, a new set of temporal edges ${O^{T}}$, and a propagation linked list $L^{T}$. $V^{T}$ splits nodes at different time points as different temporal nodes, ${O^{T}}$ links the temporal nodes at the same time point with respect to the original edges, and $L^{T}$ organizes the temporal nodes $v_{t_{i}}$ referring to the same node $v$ into sorted lists. The temporal node features $h^{0}(v_{t_{i}})$ are initialized from the given node features $x_{v}$. At each inner loop of the iterations, temporal activation function $\mathcal{A}_{k}(\cdot,\cdot)$ injects the temporal information into node features from the previous layer $h^{k-1}(v_{t_{i}})$. The function $\text{AGG}_{k}$ then aggregates neighbor features $\textbf{H}(V^{T})$ into the intermediate node embeddings $\textbf{A}(V^{T})$ following the message-passing paradigm [41]. Further, the function $\text{PROP}_{k}$ propagates the $\textbf{A}(V^{T})$ along the sorted node list $L^{T}$ for each node $v\in V$ independently. We can obtain the high-order temporal node embeddings $h^{K}(v_{t_{i}})$ after $K$ iterations. However, the required dynamic node representation is usually defined upon a future time point on tasks like future link prediction. Let $v_{t_{n}^{-}}$ be $\operatorname*{arg\,max}_{t_{i}}\{v_{t_{i}}|v_{t_{i}}\in V^{T},t_{i}<t\}$, which is the latest node embedding for a query $(v_{n},t_{n})$. A projection layer is then used to provide dynamic node representation, which works as

where MLP denotes a multilayer perceptron, and $\mathcal{A}(\cdot,\cdot)$ is the temporal activation function.

Node grouping. The training speed can be boosted much faster with GPUs since [24]. Thanks to [42], the computation speed of graphs can be largely accelerated by grouping nodes with the same degree into a computation bucket. Detailed description is provided in Appendix .3.

Model optimization. We perform future link prediction for the TAP-GNN learning framework in a self-supervised paradigm. The training objective $L$ is defined as

where the loss iterates over the temporal edges, $\sigma(\cdot)$ is the sigmoid function, $k$ denotes the number of negative samples, and $P_{n}(v)$ denotes the negative sampling distribution on nodes.

Complexity analysis. The dimensions for all input features and hidden features are assumed $d$ for simplicity. Let $N$ be the size of nodes, and $M$ be the size of temporal edges. Each AP block contains a temporal activation function and an aggregation function, so it has $\mathcal{O}(d^{2})$ parameters. Besides, the projection layer has $\mathcal{O}(d^{2}+ld^{2})$ parameters, where the first part refers to the temporal activation function, the second part refers to the multilayer perceptron, and $l$ denotes the number of layers. The total parameters of the TAP-GNN is $\mathcal{O}(Kd^{2}+ld^{2})$, where $K$ means the number of AP blocks. However, the training of the TAP-GNN requires computing the whole temporal neighborhood. As a result, the space complexity of the TAP-GNN is $\mathcal{O}(KMd)$, where $M$ denotes the number of temporal nodes $V^{T}$. The time complexity of each AP block is $\mathcal{O}(Md+Md^{2})$, where the first part refers to the aggregation-propagation process and the second part refers to the feature transformation process. Overall, the time complexity of the TAP-GNN is $\mathcal{O}(MK(d+d^{2})+Mld^{2}))$. The complexity comparison between TAP-GNN and TGAT [9] is discussed in Appendix .4.

Temporal graphs usually evolve with continuously occurring edges, called graph-stream. It is essential to adapt the node embedding model for the graph-stream scenario to capture the node dynamics as soon as possible. Given a batch of edges occurring simultaneously, our proposed TAP-GNN can be naturally extended to update the embeddings of the affected nodes efficiently. As illustrated in Section 3.4, the affected nodes only exist in the upcoming batch of edges, and they don’t influence the historical neighbors due to the temporal order. Then, the affected nodes update their embeddings by fusing the historical embeddings with current embeddings.

Let $V^{\prime}$ be the affected nodes, $E^{\prime}$ be the batch edges, $t$ be the time point, and current node embeddings can be easily computed using message-passing GNNs as illustrated in Algorithm 2. Only $V^{\prime}$ and $E^{\prime}$ are involved in the computation of current embeddings. In the graph-stream scenario, each of the propagation link lists contains two elements: the latest node embeddings and current node embeddings, which can be implemented efficiently. To compute the updated node embeddings, the historical embeddings of each layer have to be saved and fused with the propagation functions. In general, each batch of new interactions updates the embeddings of the paired nodes accordingly.

Complexity analysis. The online TAP-GNN shares the same parameters with the original TAP-GNN, which is also $\mathcal{O}(Kd^{2}+ld^{2})$. Let $n$ denote the size of $V^{\prime}$, and $m$ denote the size of $E^{\prime}$. The online TAP-GNN only needs the latest node embeddings so that its space complexity is $\mathcal{O}(KNd)$, where $K$ is the number of AP blocks, $N$ is the size of total nodes, and $d$ is the dimension of embeddings. The time complexity of computing current embeddings (line 1 to line 4 in Algorithm 2) is $\mathcal{O}(Kmd+Knd^{2})$, where the first part is the message-passing process and the second part is the feature transformation process. The time complexity of fusing historical embeddings and current embeddings is $\mathcal{O}(nd)$.

Overall, the time complexity of online inference of TAP-GNN is $\mathcal{O}(K(md+nd^{2})+nd)$.

Existing temporal GNNs [43, 9] design specific aggregation operations for neighbors, injecting temporal information into node embeddings. Firstly, their aggregation operations work on a much smaller neighborhood than that of our TAP-GNN, which implies that TDGNN [43] and TGAT [9] lose neighborhood information before their GNNs. Secondly, the scalability of TDGNN [43] and TGAT [9] is restricted since the computation complexity increases exponentially with the number of layers. In contrast, TAP-GNN could flexibly scale to large-scale temporal graphs due to its linear growth of computation complexity with the number of layers.

The proposed TAP-GNN consists of many specific modules: AP blocks, the temporal activation function, and the projection layer, each with its own hyper-parameters. Experiments in this section aim to validate the efficiency of different components of TAP-GNN by varying the hyper-parameters. Without specific settings, the batch size is 256, the number of AP blocks is 2, the embedding dimensionality is set to 128, and the dropout ratio is 0.2. Also, the names of the datasets are abbreviated without confusion for simplicity in the figures.