Exploring the Generalizability of Spatio-Temporal Traffic Prediction: Meta-Modeling and an Analytic Framework

Modeling temporal factors has been well studied in STTP, which can be traced back to a very classical research area, time-series analysis [17]. In general, most time series analysis techniques can be applied to STTP; for clarity, we just highlight some most widely used temporal knowledge and its related techniques. The most widely considered temporal knowledge includes temporal closeness, daily periodicity, and weekly periodicity:

Temporal Closeness. One of the most intuitive ways to predict future traffic data is checking back the traffic information of the recent time slots, which is called closeness. In other words, for the models considering closeness (in fact, almost all the models consider closeness), they would take the recent a few $k$ time slots’ traffic data as input and then predicts the future.

In literature, many classical statistical methods have been proposed to extract meaningful closeness patterns from the time-series data. The most famous model is perhaps ARIMA [6]. In brief, ARIMA models the traffic of a future slot as a linear combination of the flow of recent slots. As non-linear relations are hard to find with ARIMA, recently researchers started using non-linear models such as tree-based models (e.g., XGBoost [22]) and recurrent neural networks (e.g., LSTM [23, 18]). Note that, regardless of the modeling techniques, if an approach takes recent a few time slots’ traffic data as input, then its temporal consideration factor is closeness.

Daily Periodicity. As human activity has high regularity, the traffic dynamics, whatever the specific application, also often has obvious patterns. The daily periodicity is one of the most significant patterns. For example, on workdays around 8:00 a.m., people get out of home and go to workplaces, leading to the morning rush hours when many types of traffic data (metro, taxis, buses, etc.) will go from the residential area to the working area. In general, if a model considers daily periodicity, it will put the traffic data of previous days at the same time (e.g., the same hour) into the input.

To model the daily periodicity, the seasonal component can be added to ARIMA [5]. Another widely used method is selecting the historical traffic data at the same daily time slot during the last few days into the input, and then use non-linear models to extract daily patterns [1].

Weekly Periodicity. Similar to daily periodicity, the traffic data of certain scenarios may follow weekly periodicity. As an example, Saturday’s traffic will usually be much more similar to last Saturday instead of Friday.

In practice, the difference of modeling weekly periodicity compared to daily periodicity is the temporal lags. For example, suppose we want to predict the traffic data of ‘8:00-8:30 a.m. Oct. 15’. By considering daily periodicity, we take the traffic data of ‘8:00-8:30 a.m. Oct. 14’ as input; in comparison, by considering weekly periodicity, we take that of ‘8:00-8:30 a.m. Oct. 8’ as input. Apart from the different-lag inputs, the modeling techniques are almost the same for weekly and daily periodicity, such as seasonal ARIMA [24].

Compared to temporal knowledge modeling, spatial knowledge modeling recently attracts more interest from researchers as it is more complicated and heterogeneous. For different scenarios, the spatial knowledge may also be different from each other.

Geographic Proximity. ‘Everything is related to everything else. But near things are more related than distant things.’, pointed by Waldo R. Tobler in 1969, has been extensively recognized as the ‘First Law of Geography’. It clearly highlights the importance of proximity in geographic studies. To model geographic proximity in STTP, one widely used method is: for a target location, firstly select the time-series traffic data in near locations into the input, and then leverage methods such as K nearest neighbors to aggregate near locations’ knowledge. More recently, convolution neural networks (CNN) have been widely used to extract geographic proximity patterns, especially for grid-based traffic prediction problems [1]. CNN is good at this because it learns a small $m\times n$ parameter matrix (e.g., $3\times 3$ or $5\times 5$) that can well aggregate a (grid) location’s data together with its nearby (grid) locations. As CNN can be stacked to a very deep structure (e.g., by ResNet [1]), then the geographic proximity pattern can be extracted at different levels.

Location Functionality. Functionality is one fundamental character of a location. For example, some locations are residence areas, some locations are shopping areas, and others are industrial areas. Apparently, these characteristics, if obtained, can greatly improve our understanding of these locations’ traffic patterns [31].

In practice, one of the most widely used data sources to characterize the functionality of a location is points-of-interests (POIs) [8] and/or their related social media open check-ins [32, 33]. For example, the number or the distribution of different types of POIs are often used as a spatial feature vector for a location in literature [34, 7]. With such feature vectors, different modeling methods can be leveraged to extract the hidden patterns about the location functionality that are related to the traffic data [35]. Note that, if we have a large amount of historical data among many locations, it is probable that we can directly infer the location functionality from its historical traffic records, so some existing studies also develop methods to use traffic pattern or similarity to describe location functionality [33, 36, 37].

Inter-Location Relationship. In real-world applications, there are many types of spatial relationships between different locations which may indicate traffic patterns. For example, in traffic volume prediction, the locations connected by the same major city road (e.g., circle road in Beijing) will probably have related flow patterns, i.e., connectivity relationship [8]; in metro station flow prediction, the stations in the same metro line may also hold certain correlations in the flow patterns, i.e., same-line relationship.

To model such diverse inter-location relationships, a natural way is to build a graph to link locations with certain relationships. That is, we see each location as a graph node, and then link two location nodes with an edge if they have some relationship (e.g., connected by the same road). Recently, graph convolution techniques have become very powerful tools to extract hidden spatial knowledge from such constructed inter-location relation graphs [8, 9].

It is worth noting that, the geographic proximity and location functionality can also be seen as special instances of inter-location relationships. For geographic proximity, nearby locations can be linked in the graph; for location functionality, the locations with similar functionality can also be connected [8]. This indicates that the inter-location relationship is promising to be generalized to represent various spatial knowledge for STTP.

Data-Driven Modeling without Explicit Knowledge. While spatial patterns of traffic are complicated and heterogeneous in reality, some recent studies attempt to catch spatial knowledge directly by data-driven methods rather than encoding explicit knowledge factors. One representative method is first randomly constructing an inter-location graph, and then applying the graph neural network learning technique to refine the inter-location graph [26, 27]. Generally, the patterns learned by such purely data-driven methods are highly dependent on the amount and quality of the input historical data. Besides, since the inter-location relationship becomes learnable, the overall computation overhead is increased [27].

The spatio-temporal modeling part of STMeta includes two key steps. First, we decide which input format is suitable for the temporal and/or spatial knowledge. Second, we need some modeling techniques to extract useful patterns from the temporal and/or spatial knowledge inputs.

To consider the temporal knowledge including closeness, daily periodicity, and weekly periodicity, our model has constructed multiple time-series data regarding different temporal knowledge as the input. Particularly, for closeness, the time-series consists of the traffic data in recent time slots; for daily (or weekly) periodicity, the time-series includes the traffic data in the same time slot of last a few days (or the same weekday-time slot in last a few weeks). We use this input format because it is flexible and can be generalized to other cases. For example, if certain traffic data has bi-weekly/monthly/yearly periodicity patterns, it is easy to implement them into STMeta by adding a corresponding time series.

To consider spatial knowledge, we adopt the inter-location relationship graph as the input because of its generality [8, 9] (Details in ‘Inter-Location Relationship’ of last section). Then, for specific STTP scenarios, researchers can build suitable inter-location relation graphs, e.g., proximity and functionality, to encode the useful spatial knowledge.

Given the temporal and spatial inputs, the key component of our model is the spatio-temporal unit that takes both time-series data (temporal knowledge) and graph structures (spatial knowledge) into account. Recent deep learning research has developed techniques such as graph convolutional long short-term memory (GCLSTM) [9] and diffusion convolutional gated recurrent unit (DCGRU) [4].

(1) GCLSTM: Graph convolution works on a graph $G=(V,A)$ where $V$ is the vertices and $A$ is the adjacency matrix. Let $L=I-D^{-1/2}*A*D^{-1/2}$ be the normalized Laplace matrix, where $D_{ii}=\sum_{j}A_{i,j}$ is the diagonal degree matrix. The graph convolution can be computed by Chebyshev approximation [11] : $X^{\prime}=\sum_{k=0}^{K}{T_{k}(L)*X*\theta_{k}}$ where $T_{k}(L)$ is the $k$-$th$ Chebyshev polynomial and $\theta_{k}$ is a parameter matrix. To implement GCLSTM, we modify the input $X_{t}$ and the hidden state $h_{t-1}$ of the LSTM unit to a graph convoluted version:

The graph convolution part in GCLSTM can consider the spatial information while the LSTM part can take temporal information.

(2) DCGRU: From the design concept, DCGRU is very similar to GCLSTM. That is, DCGRU also re-designs the recurrent neural unit by considering node relations in a graph. The difference is that it uses GRU instead of LSTM, and also replaces the graph convolution operation with the diffusion convolution one.

As shown in Figure 2, after the spatio-temporal modeling unit, STMeta will first conduct temporal aggregation to integrate multi-temporal knowledge, and then spatial aggregation to merge multi-spatial knowledge. Similar to the spatio-temporal model unit, there are many candidate deep learning techniques that can be used for temporal and spatial aggregation.

(1) Graph Attention Layer. After firstly being introduced by Google [12], the attention mechanism has quickly been popular in the deep network structure design. One of its main usages is to aggregate multiple features into an integrated one by learning weights for each feature. Particularly, we introduce a method of using the graph attention layer (GAL) [41] to merge multiple features. The input of GAL is a set of node features $\mathbf{h}=[h_{1},h_{2},...,h_{N}]$, $h_{i}\in\mathbb{R}^{F}$, each node being the feature learned from one specific temporal or spatial knowledge (e.g., for three temporal features as closeness, daily and weekly periodicity, $\mathbf{h}$ has three elements). We first conduct linear transform parametrized by a shared weight matrix $W\in\mathbb{R}^{F,F^{\prime}}$, and then perform self-attention on each node using a shared attention parameter $a\in\mathbb{R}^{2F^{\prime}}$:

where $e_{i,j}$ represents the importance of $h_{j}$ to $h_{i}$. To make the attention coefficient comparable, we normalize them using softmax:

The output of GAL at $h_{i}$ can then be represented as

where $\sigma$ is the activation function and we use leaky RELU [41]. To make the self-attention process more robust, we add the multi-head mechanism into the model [12] ($M$ is the number of multi-head):

As each node has its own feature after GAL, we then use an average pooling layer to aggregate $h^{\prime}_{i}$ into a feature vector:

where $\bar{h}$ is the final GAL aggregated feature representation from a set of features $\mathbf{h}$.

(2) Concatenation with Dense Layer. Another widely used technique in deep learning to combine multiple features is concatenation. Then, dense layers can be applied to the concatenated features so as to extract useful representations for the target task:

For the temporal or spatial aggregation unit in Figure 2, either of the above two techniques can be selected.

With the spatio-temporal modeling unit and the temporal and spatial aggregation unit, we can then implement concrete STTP approaches following STMeta. Particularly, after spatial aggregation, we can simply stack several dense network layers on the aggregated spatio-temporal representations to predict the future traffic.

It is worth noting that, following the design principle of STMeta, we can have a simplified version by considering only temporal factors, called TMeta (Figure 3). The temporal modeling unit can be implemented by LSTM [23] or GRU [42]. Later in the benchmark experiments, we will also check how TMeta performs so as to investigate how much improvement can be brought into practice by encoding spatial knowledge.

Compared with the existing methods listed in Table I, STMeta is flexible to incorporate a variety of temporal and spatial factors. In particular, ST-MGCN [8] has also considered multiple temporal and spatial knowledge factors, but its network structure is different from STMeta in temporal modeling. More specifically, ST-MGCN directly combines the weekly, daily, and closeness historical records into one time-series sequence as the input. In comparison, the inputs of STMeta include three types of time-series data regarding closeness, daily periodicity, and weekly periodicity, respectively; thus, STMeta introduces a temporal aggregation unit to combine the three temporal patterns. We think that the design of STMeta may help to learn different temporal patterns more easily, as each temporal pattern (closeness, daily, and weekly periodicity) has its own time-series sequence for dedicated learning.

The configurations of the experiments are as follows:

Hardware. Our experiment platform is a computation server with AMD Ryzer 9 3900X CPU (12 cores @ 3.80 GHz), 64 GB RAM, and Nvidia RTX 2080Ti GPU (11GB).

Train/Validation/Test Split. We choose the last 10% duration in each dataset as test data, the 10% data before the test for validation. The prediction granularity is set to three settings including 15, 30, and 60 minutes for all datasets.

Training Stopping Criteria. In training, as different datasets and methods need a varying number of epochs for convergence, we thus leverage t-test for the training stopping criteria instead of setting a fixed number of epochs. Particularly, we divide the validation loss of recent epochs (the number of recent epochs is called early stop patience) into two halves. For example, if the early stop patience is 100, then the two halves are last 1-50 epochs and last 51-100 epochs. Then, we perform the independent sample t-test on the validation losses of the two halves. When the p-value is smaller than a threshold (we set it to 0.1), the validation losses of the two halves are statistically different, and thus we continue training. The setting of early stop patience is critical, as a small value may stop training too early when the model is still unstable, while a large value may let early stop become useless and lead to overfitting. With trial-and-error, the early stop patience values of the bikesharing, ridesharing, metro, and EV datasets are set to 200, 1000, 400, and 400, respectively, in our experiments.

We consider the following spatio-temporal factors.

Temporal: We consider temporal closeness, daily, and weekly periodicity in our experiments. For each temporal factor in STMeta, we build one time-series data.

Spatial: For all the datasets, we consider the spatial information including proximity and functionality. For proximity, we build a graph by linking locations whose distance is smaller than a threshold. For functionality, we use the Pearson correlation of the historical traffic data between two locations to indicate their functionality similarity, and link locations with a large correlation. For ridesharing and bikesharing cases, we also build an interaction graph by linking the two locations with frequent interactions (many users go from one location to another) [9]; for metro, we construct a same-line graph by linking the stations in the same line together. The thresholds for building the graphs are shown in Table II. We select these thresholds so that the average number of connections for each node is around 20–30% of the total number of nodes, which performs well in our experiments.

As shown in Table III, we implement three variants of STMeta by changing the techniques used in spatio-temporal modeling and aggregation units (see Section 4.2).

Specifically, GCLSTM and DCGRU both contain one layer and 64 hidden units; GAL is set to include two heads and 64 hidden units. After temporal and spatial aggregations, we further stack two dense layers with 64 hidden units to generate the prediction output. We choose the optimization algorithm as ADAM [43]; the learning rate is 1e-5. We also implement TMeta (only temporal factors) with LSTM [23].

We implement a number of benchmark approaches in the literature. These approaches fall into two types, the first only considers the temporal factors and the second considers both temporal and spatial factors.

Approaches with only temporal factors:

• •

  HM (Historical Mean) predicts the traffic according to the mean value of the historical records. We implement two variants of the HM algorithms. The first considers only the recent time slots (closeness), denoted as HM (TC). The second averages recent time slots (closeness), the historical records in the same time of last day (daily periodicity) and last week (weekly periodicity), thus considering multiple temporal factors, denoted as HM (TM).

• •

  ARIMA [24] is a widely used time series prediction model, which mainly considers temporal closeness.

• •

  GBRT (Gradient Boosted Regression Trees) [44] can be applied to predict the traffic for each location. Our GBRT implementation uses historical traffic data as features, not only from recent time slots, but also from last day and last week. Hence, GBRT considers multiple temporal factors.

• •

  XGBoost [22] is similar to GBRT, another widely used tree-based machine learning model.

• •

  LSTM [18] neural networks take the recent time-slot traffic data as inputs (closeness) and predict the future.

Approaches with both temporal and spatial factors:

• •

  ST-ResNet [1] leverages residual convolution networks to consider spatial proximity and also considers temporal closeness, daily and weekly periodicity simultaneously. Note that ST-ResNet can only work for grid-based traffic prediction (ridesharing).

• •

  DCRNN [4] combines diffusion convolution and recurrent networks for capturing spatio-temporal features. The original DCRNN model only considers the temporal closeness and the spatial proximity graph.

• •

  STGCN [15] combines graph convolution and gated convolution units to catch spatial and temporal features, respectively. It considers the temporal closeness and spatial proximity graph.

• •

  GMAN [28] leverages attention mechanisms to model both temporal closeness and spatial proximity patterns for traffic prediction.

• •

  Graph-WaveNet [26] designs a data-driven graph convolution method for adaptively learning spatial knowledge in addition to proximity. For temporal knowledge, it considers only recent traffic data.

• •

  ST-MGCN [8] captures multiple spatial relations with graph convolutions. It also considers temporal closeness, daily and weekly periodicity. Not like STMeta, ST-MGCN concatenates the inputs regarding different temporal factors into one time-series data.

• •

  ARGCN-CDW purely relies on data-driven graph convolution methods for extracting spatial knowledge [27]. Meanwhile, it considers temporal closeness, daily and weekly periodicity same as ST-MGCN.⁴⁴4The original ARGCN [27] considers only temporal closeness, but it is hard to converge in our experiments. By modifying its input as the time-series data combined by closeness, daily and weekly periodicity (same as ST-MGCN), the performance of ARGCN is much improved. We denote this model as ARGCN-CDW.

The hyperparameters of the above benchmark approaches follow the default settings as their original papers.⁵⁵5 We have also tested some other approaches in Table I, such as ASTGCN [25] and STSGCN [29]; however, they are hard to converge under certain experiment scenarios.

We use RMSE (root mean square error) to report the prediction error, as this is the most widely used metric in almost all the STTP research [1, 7, 8, 9, 14].⁶⁶6We have also tested methods in MAE (Mean Absolute Error), and the results are consistent with RMSE. For clarity, we just report RMSE. We also compute two aggregation scores to indicate the overall performance (i.e., generalizability) of an approach $x\in\mathcal{X}$ ($\mathcal{X}$ is the set of all approaches) among all the datasets $\mathcal{D}$:

where $RMSE_{x,d}$ is the RMSE of approach $x$ in dataset $d$. The meaning of the score is the average of the normalized RMSE (normalized by the smallest RMSE for every dataset $d$). Ideally, if an approach can perform the best among all datasets, then AvgNRMSE is $1$.

If a method is generalizable, its WstNRMSE should also be close to $1$, i.e., in the worst dataset, its performance is still near the best one.

Regarding the temporal knowledge factors, we first investigate the prediction performance of the ‘temporal’ approaches in Table IV/S1/S2. According to the results, considering more temporal knowledge would improve prediction significantly. For example, in 60-minute prediction (Table IV), HM (TM)’s AvgNRMSE = 1.190; in comparison, HM (TC)’s AvgNRMSE = 2.597, which is much worse than HM (TM). This reveals involving multiple temporal factors indeed enhances the prediction accuracy.

To further elaborate on the impacts of different temporal factors, we conduct an experiment with STMeta by removing certain temporal factors. The results are shown in Table V. We observe that adding periodicity can always enhance the prediction accuracy as human mobility has intrinsic regularity [45]; meanwhile, the training and inference time also increases a bit. Generally, adding both daily and weekly periodicity can bring much improvement, while adding only daily periodicity might be the best (metro, Shanghai). Hence, the temporal periodicity knowledge is often generalizable, while which periodicity performs the best depends on applications.

Besides, with the decrease of the time slot length, the performance gap between ‘TC’ and ‘TM’ methods becomes smaller, indicating that the temporal closeness plays a more important role. For example, in the 60-min experiment, AvgNRMSE of LSTM (TC) and TMeta-LSTM-GAL (TM) are 1.882 and 1.047, respectively, leading to a gap of 0.835; in the 15-min experiment, AvgNRMSE of LSTM (TC) and TMeta-LSTM-GAL (TM) are 1.332 and 1.086, respectively, indicating a gap of only 0.246. This observation signifies that the temporal closeness factor becomes more important when the time slot length is reduced. This also fits our expectation — the correlation between the next 15-minute traffic data and the recent 15-minute should be higher than the correlation between the next 60-minute traffic data and the recent 60-minute, as the time difference of the two continuous slots in the former case is much smaller than the latter case.

Similarly, to analyze how spatial knowledge impacts prediction, we remove some spatial factors from STMeta-DCG-GAL and then compare it to the original STMeta-DCG-GAL (with full spatial knowledge). In particular, for the 60-minute prediction, we implement some variants of STMeta-DCG-GAL with partial spatial information: one of the proximity, functionality, and interaction/same-line graphs. Due to the page limitation, we only show one dataset for each scenario.

Results are shown in Table VI. First, among different spatial graphs, the functionality graph performs always the best. In this regard, we deem that it is more generalizable across different datasets. Besides, the metro, EV, and speed scenarios benefit from the integration of multiple spatial knowledge more significantly than the bikesharing and ridesharing scenarios. For example, in ridesharing, STMeta with full spatial knowledge perform worse than the approach with only one type of spatial knowledge in Chengdu; moreover, training STMeta with full spatial knowledge (i.e., three graphs) costs more than 5 times of the computation hours compared with STMeta with only one spatial graph. Hence, if we blindly add a variety of spatial knowledge graphs into a model, it may degrade both the model effectiveness and efficiency. This inspires that, while generally considering spatial knowledge can improve prediction, selecting and integrating which specific spatial knowledge should be carefully determined for a specific application. Otherwise, it is possible that the overall prediction performance is degraded by adding more spatial knowledge.

With the popularity of machine learning, especially deep learning, more and more studies focus on applying novel modeling techniques to the STTP model design, while which temporal or spatial knowledge is considered often lacks detailed analysis. However, our benchmark results indicate that, for STTP model design, it is probable that which part of knowledge in consideration is more important than which modeling technique in use. For example, when we consider complete temporal closeness, daily and weekly periodicity together in the naive historical mean method, HM (TM) can significantly outperform the advanced deep modeling technique with inadequate temporal knowledge, LSTM (TC) that considers only the temporal closeness in all the benchmark scenarios. In a word, regarding the knowledge and modeling techniques, we believe that researchers and practitioners should make more efforts on the knowledge part, i.e., considering more carefully about the temporal and spatial knowledge that may be suitable for the target STTP scenario.

With the benchmark experiments, we can also compare the relative importance between temporal and spatial knowledge in their generalizability. Overall, we find that purely using temporal knowledge can already achieve competitive generalizability performance across different datasets, compared to the approaches with both temporal and spatial knowledge. For example, TMeta-LSTM-GAL’s AvgNRMSE ranges from 1.047 to 1.086 for 15/30/60-minute prediction experiments. This indicates that the best model with extra spatial knowledge may help improve only $\sim$5% beyond temporal knowledge. Specifically, for Bikesharing-CHI (30-minute), TMeta-LSTM-GAL performs the best (tie with STMeta-GCL-CON). This reveals that temporal knowledge can dominate the spatio-temporal factors in some cases. In comparison, as shown in Sec. 5.5.2, adding spatial knowledge does not always improve the prediction accuracy. For different datasets, which spatial knowledge contributes more varies significantly. Hence, our benchmark results reveal that temporal knowledge is more generalizable than spatial knowledge across different STTP scenarios.

For the researchers needing to design STTP approaches for their applications, we summarize the following guidelines.

Guideline 0 (most important): Always think deeply about which knowledge should be encoded in the model design before focusing on sophisticated modeling tricks. For a targeted STTP scenario, finding a good set of knowledge suitable for the scenario is a prerequisite and fundamental step before designing model details.

Guideline 1 (Temporal Knowledge): For temporal knowledge, consider temporal closeness, daily and weekly periodicity together. They are perhaps generalized and robust knowledge for various STTP scenarios, as these are fundamental human activity patterns.⁷⁷7Other scenario-specific temporal knowledge may also be carefully designed. For example, if we predict traffic in special seasons such as the Spring Festival Travel Season in China, yearly periodicity is critical.

Guideline 2 (Spatial Knowledge): For spatial knowledge, it needs more careful consideration. Overall, spatial knowledge is not as generalizable as temporal knowledge, whether it is pre-specified or purely learned from data. Hence, instead of simply aggregating all the varieties of spatial knowledge, conduct trial-and-error tests to select the best spatial knowledge combination.

Guideline 3 (STMeta): Our STMeta can be used as a meta-model to integrate multiple temporal and spatial knowledge. With STMeta, researchers only need to consider how to find effective spatial knowledge and encode the knowledge into an inter-location relationship graph.

Guideline 4 (TMeta): To deal with a new STTP problem, we recommend firstly building a TMeta model with only temporal knowledge as an easy-to-implement baseline. This is because temporal knowledge is more generalized than spatial knowledge, and the computation efficiency of TMeta is better than STMeta.

Guideline 5 (Traditional Models): For STTP with short-length time slot settings, the traditional machine learning models with only temporal knowledge, such as GBRT (TM), can be a nice option to trade off the prediction accuracy and computation efficiency.