EPNE: Evolutionary Pattern Preserving
Network Embedding

Considering node $v_{i}$ and one of its neighbors $v_{j}$ at time $t_{1}$ and $t_{2}=t_{1}+T$, if $\mathbf{u}_{i}^{t_{1}}\cdot\mathbf{u}_{j}^{t_{1}}\approx\mathbf{u}_{i}^{t_{2}}\cdot\mathbf{u}_{j}^{t_{2}}$, then a periodicity of $T$ between $v_{i}$ and $v_{j}$ is indicated. Periodic evolutionary patterns are critical in identifying relationships between nodes. For example, in social networks, the frequency of interactions between colleagues peak during weekdays and decline during weekends, while that between families and friends is completely the opposite, shrinking in weekdays and peaking during holidays. In co-author networks, experts from different domains may exhibit different patterns of cooperation due to different frequencies at which conferences are held.

Inspired by ideas in signal analysis, we extract periodic temporal features by decomposing them into basic patterns $x(t)=\sum_{m=1}^{M}\omega_{m}x_{m}(t)$, where $x_{m}(t)$ is a basic pattern and $\omega_{m}$ is the corresponding intensity which will serve as features in our model.

Based on the general ideas above, we define a generating function $\mathbf{f}^{*}$ to generate a set of functions $\{\mathbf{f}_{a,b}\}$ as basic patterns with different scales through translating and retracting

where $a,b$ are the shifting and scaling factors respectively. We set the number of scales as $L$. These basic patterns are regarded as convolution kernels, which are used to carry out causal convolutions to extract decomposed features as

We concatenate the vectors $\mathbf{s}_{f,a,b}^{t}(v_{i})$ on different scales and shifts to get the frequency domain features of a node

For the choice of generating functions, one simple yet commonly used option is Haar wavelets, a sequence of rescaled square-shaped signals used in computer vision and time series analysis for feature extraction [16, 2]. Such common solutions prove to be sufficient by the experimental results for extracting general evolutionary patterns. For the shifting and scaling factors $a$ and $b$, we adopt the setting of

as stated in [3] to be generally adopted for Haar wavelets. In addition to common settings, our model allows for personalized designs that meet the properties of individual datasets. For instance, in social networks, we divide each time slice as a one-hour interval, and designed $b_{0}=1,b_{1}=\frac{1}{24},b_{2}=\frac{1}{7\cdot 24}$ such that basic patterns thus generated would capture daily and weekly dynamics.

In addition to periodic patterns, the local structure of a node may follow a steady, monotonous trend instead of periodicity. We model non-periodic evolutionary patterns by learning features in time domains based on decay kernels. Generally for humans, more recent behaviors exert a stronger impact on the present, and so does decay kernels operate on vector sequences. We define a decay kernel $\mathbf{f}_{\alpha}(t)=e^{-\alpha t}$, where $\alpha$ is the decay rate, which is then fed into causal convolutions on sequences $\mathbf{U}_{i}^{(t-h,t)}$ to capture non-periodic trends. Specifically, we have the time domain features similarly as

We concatenate both periodic and non-periodic features to acquire the complete temporal features

The challenge yet to overcome is that, how should the aforementioned features be incorporated to elevate the performance of the embedding algorithm.

Here we show a brief discussion about our model and DynamicTriad, HTNE. It is evident that both DynamicTriad and HTNE only capture non-periodic temporal features in that, closed triads will not be open again and the intensity of Hawkes processes is monotonously decreasing. By comparison, we propose that periodic temporal patterns do shed light on relationships between nodes and capture them, which is the clear distinction between DynamicTriad, HTNE and our model. We will show how both periodic and non-periodic temporal features contribute to the performance of our model in the experiments.

In this section, we introduce the intuition underlying our temporal objective function, followed by its formal definition.

Since the representation vectors preserve pairwise proximity between nodes, we propose that relative positions between node pairs reflect relationships between the nodes. For example, if $\|\mathbf{u}_{1}-\mathbf{u}_{2}\|=\|\mathbf{u}_{3}-\mathbf{u_{4}}\|$ but $(\mathbf{u}_{1}-\mathbf{u_{2}})\cdot(\mathbf{u}_{3}-\mathbf{u}_{4})=0$, we would believe that node pairs $1,2$ and $3,4$ are identically adjacent in network space, yet edge $\langle 1,2\rangle$ and $\langle 3,4\rangle$ represent relationships that are scarcely correlated.

Since we propose that evolutionary patterns can implicitly reveal relationships between node pairs, as illustrated by the example of colleagues and families, we would hence derive that, evolutionary patterns are also indicative of relative positions between pairwise nodes, and vise versa.

The intuition introduced above can be implemented by an auto-encoding style objective function

where $\mathbf{u}_{i}^{t}-\mathbf{u}_{j}^{t}$ implies the relative position between node pairs, $g(\cdot,\cdot)$ is some function aiming to interpret relative positions between node pairs through temporal features, and $D(\cdot,\cdot)$ measures a distance metric between two vectors. In particular, in our model, we set

where $\oplus$ denotes concatenation, $\mathbf{W}\in\mathbb{R}^{d_{1}\times 2d_{2}}$ is a trainable matrix, $\sigma(x)$ denotes the sigmoid function, and $D(\mathbf{x},\mathbf{y})=\cos(\mathbf{x},\mathbf{y})$ denotes cosine similarity between vectors.

In this paper, we preserve proximity between pairwise nodes through an objective derived from DeepWalk, but it should also be noticed that EPNE is not restricted to specific models and can be generally incorporated to capture temporal information. We maximizes the likelihood that the context $v_{j}\in N^{t}(v_{i})$ is observed conditional on the representation vectors $\mathbf{u}^{t}_{i}$ and $\mathbf{u}^{t}_{j}$. Our objective preserving proximity, accelerated by negative sampling, is defined as

We notice that the structural objective possesses the property of rotational invariance, i.e. the objective does not change regardless of how the vector space is rotated, which poses potential threats to the learning of our model. As we model temporal features in different vector spaces between time steps, it is important that those spaces should be aligned such that we do obtain evolutionary patterns instead of random rotations of vector spaces between time steps. To enforce such restriction, we propose a loss function imposing smoothness in a weighted manner between adjacent time steps

where $\|\mathbf{d}_{i}^{t}-\mathbf{d}_{i}^{t-1}\|$ measures how the structure of node $v_{i}$ changes from time $t-1$ to time $t$ as defined in [8].

We have the overall loss function of our model by summing these objectives

where $\alpha$ being the temporal weight and $\beta$ being the smoothness weight are hyperparameters to be tuned.

We apply Stochastic Gradient Descent (SGD) to optimize the objective function. We learn representation vectors for nodes $v_{i}^{t}$ at each time step $t$ incrementally based on historical representation vectors, thereby restricting the number of parameters at each time step within $O(|V|)$. We summarize our learning algorithm in Algorithm 1.

We employ the following datasets with temporal information, whose statistics are listed in Table 1.

• •

  High School [14] is a social network collected in December, 2013. We consider students as nodes and active contacts collected by wearable sensors as edges. We split time steps by one-hour intervals. The labels of edges denote Facebook friendship.

• •

  Mobile. We build a social network from a mobile phone dataset collected in October 2010. We consider users as nodes and interactions as edges . We split each time step by a two-hour interval. The labels of edges represent colleague relationships between users.

• •

  AMiner [27, 25] and DBLP [29] are both co-author networks in which nodes represent researchers and edges represent co-authorship. For temporal settings, the datasets are split with four-year and one-year intervals respectively. The node labels represent research fields according to the conferences the author published his papers in. We take edge labels in AMiner as whether two researchers share the same field of interest.

We compare our model with the following novel methods. Unless specified, these models are tested with their published codes as well as parameter settings mentioned in their original papers.

• •

  Static Skip-Gram models including DeepWalk [19], node2vec [6] and LINE [21]. For node2vec, we take $p=q=0.25$.

• •

  Static Graph Neural Networks. We take GraphSAGE [7] as an example. We take unsupervised GraphSAGE without node features. We take 2-layer networks with a hidden layer sized 100. We adopt the neighborhood sampling technique with 20 neighbors to sample at each layer.

• •

  HTNE [29]. It is a dynamic network embedding method modeling sequential neighborhood formation processes using Hawkes Process.

• •

  DynamicTriad [27]. It is a dynamic network embedding method based on triadic closure processes and social homophily. We set $\beta_{0}=1$ and tested $\beta_{1}\in\{0.01,0.1\}$ for optimal performances.

For static methods, we compress all temporal snapshots of the graphs into a “stacked” static graph as mentioned in Definition 1, just like what [29, 27] did for static methods.

For a fair comparison, the embedding dimensions are all set to $\mathbf{32}$. The number of paths for each node and the length of each path for DeepWalk, Node2Vec, and EPNE are all set to $10$. It is worth mentioning that, although Perozzi et al. [19] took 80 paths per node, it was concluded redundant in our experiments as 10 paths per node have been capable of achieving similar performance. The window size for all random-walk based models is set to 5. In addition, we set $\alpha=1.0,\beta=0.01$ for all four datasets. We set the history length $h=45,12,12,10$ and number of scales $L=3,4,4,4$ for High School, Mobile, AMiner and DBLP respectively.