Link Prediction for Temporally Consistent Networks

Earlier in the paper, we mentioned that link prediction estimates the next relationship in a dynamic network. Temporally consistent networks are a special kind of dynamic networks that smoothly evolve or devolve based on an observed or hidden relationship [8]. Temporal consistency has its roots in physics, but it was recently used to improve predictions in image-based graphics and video sequencing [19]. We now formally define the properties that hold true for temporally consistent networks.

Examples of temporally consistent networks include activity networks, road traffic networks, and social networks. The evolution of phenomena can be predicted using the spatial, temporal, or thematic relationships of a temporally consistent network.

Time-parameterized matrix (TP-matrix) is our resolution for class imbalance which hampers the effectiveness of link prediction models using an adjacency matrix [12]. The use of adjacency matrix or binary edge weights (0, 1) to indicate a link between two nodes limits the ability to model all states in a global network [16]. This limitation is particularly evident in forming networks with a large number of unobserved links. In addition, Figure 2 shows graphs of real-world events tend to be sparse. The absence of a link conveys more noise than information when binary edge weights are used [16]. We now explore the suitability of (TP-matrix) for modeling heterogeneous, time-evolving networks.

We will illustrate our approach by using concrete examples from a real-world dataset of sales activities. We transform a sequence of observed activities and their timestamps into a network in the form of a TP-matrix. Each row represents one instance of a sale. Each column represents an event or an activity during the lifetime of a corresponding sale. The values of the matrix represent timed edges $a_{(i,j)}$. In lieu of using binary edge weights, we use temporal residuals. Each value contains the transition timestamp from node $i$, the initial event of the $i^{th}$ instance of a sale to node $j$, which represents the corresponding $j^{th}$ activity. In the absence of a transition between node $i$ to node $j$, the value $a_{(i,j)}$ is the time distance between the initial event $i$ (i.e. the receipt of a sales request) and the running time $\Delta$. We use the term ’time-parameterization’ with respect to the node-level timestamps and time distances. At the network level, the term ’temporal parameterization’ is more appropriate as it emphasizes the relationship aspect of all temporal properties. We make this distinction to note that our reference to time-parameterization in this paper emphasizes the timing aspect whereas temporal parameterization emphasizes the evolutionary aspect of temporal relationship at the global network. The computation of the TP-matrix is presented below. First, we use the running time $\Delta$ in Equation 1 to eliminate class imbalance due to missing edges.

Second, we normalize the temporal values obtained in Equation 1 to $\mathbb{R}\in(0,1]$. The normalization method is shown in Equation 2.

This normalization maintains the non-zero scale to avoid re-introducing noise. It is similar to normalizing user-item ratings in collaborative filtering for recommendation systems [20], where each user’s rating is divided by the maximum rating allowed. In our case, we cannot use the global denominator because temporal distances between activities are continuously variable.

In subsection III-B, we looked at how we better capture the evolution of network structure through time. In this section, we will concentrate on the evolution of time and its predictive influence. The distinction between structure-function and time-function relationships are often overlooked, but it is a significant research problem for dynamic networks. Figure 3 shows a dynamic network timeline split into observed and unobserved parts. The current state of the art in link prediction considers the evolution as a function of continuous snapshots. A fixed time interval between timesteps must be defined during training. This means the prediction can happen only at the same fixed interval. A model trained with a 2-day fixed interval expects the same 2-day interval at prediction time. This is not ideal for practical applications because a random, on-demand prediction is not possible. An alternative approach would be to make the time interval user-configurable rather than fixed. However, that would exponentially increase the number of trained models. To make a continuous prediction possible as illustrated by the green diamonds in Figure 3, it is important to separate or decouple the time-function relationship from the structure-function relationship. We develop a time-parameterized predictive influence (TPPI) to capture the influence of time in a temporal relationship.

We start by evaluating the contribution of a given node to the evolutionary progress of its network.

The predictive influence of each node is maximized given the node’s feature vector $f(.)$ using Equation 4.

Backward and forward-looking maximization is used over the temporal space in between $T-\alpha$ and $T+\alpha$, where $\alpha$ is a tunable hyperparameter, which allows for the adjustment of the temporal degree of the neighborhood during model training. TPPI learns from feature vectors of each pair of nodes ($v_{i},v_{j})\in V_{t}$ and their temporal weights $a(i,j)$ obtained from Equation 2, yielding the following Equation 5.

Note that a higher influence threshold $\beta$ makes TPPI more conservative. A value of $0$ is equivalent to using no threshold. The threshold hyperparameter helps balance or fine-tune the temporal and structural context sensitivities of the network being modeled.

The correlation matrix using Pearson correlation coefficients is effective for showing pair-wise correlations of more than two independent variables. The matrix values are correlation confidence values in the interval [-1, 1]. A smaller or larger absolute value, respectively, indicates a weaker or stronger pair-wise correlation, whereas the negative or positive (-/+) sign shows the direction of the correlation. For predictive influence and feature selection analysis, the maximum confidence value of $1$ indicates a collinearity problem where the pair-wise relation of two variables is so high they are likely identical. A confidence value of $0$ shows no linear relationship.

Now, we will show TP-matrix can capture the spatiotemporal correlation among network nodes relatively better than the adjacency matrix. The graphical representation of the correlation matrix for sales activities are shown in Figure 4. The Pearson correlation with hierarchical clustering is shown for both the adjacency matrix and TP-matrix. We can observe several benefits of the TP-matrix over the adjacency matrix. The adjacency matrix shows a weak intra-node correlation because of the sparsity of the graph data. The TP-matrix conveys more spatiotemporal correlation with more realistic clustering. For example, a vehicle test drive, a deal negotiation, a referral to a sales manager (manager turn-over), and a conclusion of a deal (closed) usually all happen in one initial or subsequent visit to a dealership showroom. The clustering of these activities in (b) of Figure 4 is, therefore, more realistic. In addition, we can observe that it is not possible to calculate the pair-wise correlation coefficient between $closed$ and any other activity using the adjacency matrix. Since the dataset included successful deals only, an adjacency matrix would have the value $closed=1$ for all instances. As a result, a variance $\sigma^{2}$ and, therefore, a Pearson’s correlation coefficient cannot be calculated. This supports our claim (and that of others [16]) that the adjacency matrix cannot model all states in a global network. We summarize our findings below.

1. 1.

   Using the running timestep $\Delta$ eliminated a key limitation due to missing edges when using the adjacency matrix. With this approach, the absence of a link conveys latent feature variance and relevance with respect to observed links.

2. 2.

   Using timed edges improved the retention of temporal relationships with respect to the global network. It also highlighted more granular relationships among sub-groups of nodes. The rectangles in (b) of Figure 4 show clusters of correlated activities that matched the mental model of automotive experts. For example, independent activities that deal with appointment scheduling and tracking are in the same cluster. During deal negotiation, a sales manager may get involved. Such activity is tracked as ’Manager Turnover’.

3. 3.

   With this approach, the model variance is reduced because 1) the model is less prone to the cold start problem at the initial phase when the network is forming, and 2) the model regulates itself with a decay function as the network grows larger. We will expand on this in the next subsection.

Regularization is a technique used to reduce model variance. In its simplest form, a uniform decay $\frac{1}{2}$ can be applied to an objective function. A much popular approach is to use a decay function with a parameter $\frac{1}{2}\theta$. The parameter regulates how fast the decay hastens. In temporal link prediction, an exponential decay function $D(t)=e^{-\theta(T-t)}$ with time $t$ $\in T$ is commonly used to regulate the importance of the current network snapshot compared to previous snapshots [16, 8, 14].

As the network progresses, the farthest network states become less relevant than the current network state. The acceleration of the irrelevance (decay) depends on the problem and can be estimated by the decay parameter $\theta$. However, one can argue that an exponential decay function for a temporal link prediction is not necessarily a strictly monotonically decreasing function. This is especially true for temporally consistent networks. Let us consider the citation network, which is sensitive to temporal evolution. On the question of the most impactful papers, aging papers with continuous and active citations may have higher relevance than recent papers without proven staying power. This supports our argument that previous network states are not necessarily less relevant than the current state. We can take this argument further by pointing out that, on a more granular level, certain nodes may continue to contribute to a network’s evolution at a much higher rate than other nodes. With this in mind, we define a relative exponential decay function in (6) by first taking the Mean TPPI (See Equation 3) of the current network state. A lower Mean TPPI infers more nodes that are not contributing to the network evolution. We then use it as the decay parameter since it better reflects a node-level temporal decay of the network state rather than a mere time decay.

Table II shows synthetic examples of a network decay with four nodes. We assume that $T-t=0.1$ and $\theta=6$ for the commonly used decay function. With our relative exponential decay, you can see that an over-performing node (in terms of its predictive influence) can slow down the temporal decay of the overall network, even when the influence of other nodes are diminishing. In contrast, the commonly used decay function is a function of time decay and, therefore, a strictly decreasing function overtime.

We trained our model using stochastic gradient descent (SGD) with momentum [21]. The momentum $\gamma$, learning rate $\lambda$, and the temporal range $\alpha$ are tunable, user-defined hyperparameters that are initially set (as default values) to 0.9, 0.1 and 3, respectively. We chose these default values because they are widely accepted as initial values and also worked well for our model. The complete list of hyperparameters is in table III.

The model is iteratively trained until it converges. Convergence is achieved when the minimum loss variation is consistently below a tolerance threshold of $10^{-3}$ for the last 10 epochs. More specifically, the training stops when the stop condition in Equation 7 is satisfied, where $E$ is the error vector produced in a maximal number of boosting iterations $M$.

Given a TP-Matrix $\mathbf{A}$(t) and TPPI-matrix $\mathbf{R}$(t) at time $t$, we minimize the following objective function for $\mathbf{U}$ and $\mathbf{V}$.

We carried out most of the time-dependent optimization legwork in subsection III-C. The first term is regulated with the relative exponential decay function D(t), whereas the other two terms are regulated by a learning rate $\lambda$ that is continuously reduced from its initial value to $10^{-4}$.

We show the derivatives of the objective function below.

The pseudocode for our model is presented in Algorithm 1.

We used a total of four datasets of dynamic networks. Two of them, Infectious and Digg, were part of KONECT¹¹1http://konect.uni-koblenz.de/networks/ networks, which are widely used in link prediction. These two datasets were used for baseline comparative analysis. The third dataset was a collection of social reactions on Facebook posts from the British Broadcasting Corporation (FBReactions²²2https://github.com/naman/fb-posts-dataset). It contained 348 Facebook posts on a wide range of topics with at least one of 6 possible social reactions (like, love, wow, haha, sad, angry). Missing edges (e.g., missing reaction type angry) were high (66% of all edges). The published social reactions dataset did not include timestamps on social reactions. A timestamp was randomly generated for each reported social reaction. Posts without at least one social reaction were removed from consideration. Since Facebook uses the same 6 reactions across their social media platform, this experiment should apply to Facebook Live videos as well. The fourth was a real industry dataset (AutoSales) collected from a live automotive CRM [15]. This dataset contained 10,000 anonymized sales events, each representing a network with 12 nodes. Missing edges (e.g., missing activity type sales appointment) accounted for 44% of all possible edges. Table IV shows the properties of the datasets.

Like most link prediction algorithms, our work is a regression task. It uses the structure and time relationships within a network to learn its evolution. Root-Mean-Square Error (RMSE) is used to evaluate the generalization capability of regression-based algorithms. In this paper, we will report both the iterative (epoc steps) and overall RMSE performance. Mean absolute error (MAE) is also reported as an additional performance evaluation. In addition, we show the training run-time performance with six different network samples. The graph representation allows for evaluating the accuracy performance of regression-based link prediction using the area under the curve (AUC) binary measurement. AUC has been used extensively as an evaluation metric for link prediction problems [16]. However, this requires the preparation of the test data in a certain way. We now describe our approach using the automotive CRM dataset.

CRM activities were converted into a sequence of states and their corresponding timestamps $(x_{0},t_{0}),\dots,(x_{i},t_{i}),~{}where~{}\forall i\ t_{i}<t_{i+1}$. For uniformity during training and evaluation, each state or activity was assigned an identifier, as shown in Table I. The temporal property of each activity was then captured by a TP-matrix. The goal of this experiment setup was to enable the evaluation of our link prediction approach against our baseline, TMF, using AUC. For example, 1200 different networks, with ten different nodes each, were tested. For any network, we ran link predictions at most $T-1$ stages. That is for every node in the network at any given time except the destination node at T. Our training data was the activity sequence data represented as a network at timestep $T$. For our test data, we removed the node at $T$. We considered the test positive if the predicted node matched with the node at $T$. Otherwise, we considered it negative. We can now use the binary measurement and compare our results with our baseline TMF.