Revitalizing Multivariate Time Series Forecasting: Learnable Decomposition with Inter-Series Dependencies and Intra-Series Variations Modeling

Given a multivariate time series input $X\in\mathbb{R}^{N\times T}$, time series forecasting tasks are designed to predict its future $F$ time steps $\hat{Y}\in\mathbb{R}^{N\times F}$, where $N$ is the number of variates or channels, and $T$ represents the look-back window length. We aim to make $\hat{Y}$ closely approximate $Y\in\mathbb{R}^{N\times F}$, which represents the ground truth.

We employ a superior learnable 1D convolutional decomposition kernel instead of a moving average kernel to comprehensively encapsulate the nuanced temporal variations in the time series.

Projection and position embedding. Following iTransformer (Liu et al., 2024), we first map time series data $X\in\mathbb{R}^{N\times T}$ from the original space to a new space, subsequently incorporating positional encoding $Pos\in\mathbb{R}^{N\times D}$ as $X_{embed}\in\mathbb{R}^{N\times D}$ following $X_{embed}=(XW+b)+Pos,$ with weights $W\in\mathbb{R}^{T\times D},b\in\mathbb{R}^{1\times D}$, where $D$ is the dimension of the layer.

Learnable 1D convolutional decomposition kernel. To realize the learnable convolutional decomposition, we need to define the convolutional decomposition kernel first. Specifically, we pre-define a stride of $S=1$ and a kernel size of $K=25$ experimentally. Regarding its weight, initialization is performed utilizing a Gaussian distribution. We assume its weight is $\omega\in\mathbb{R}^{1\times 1\times K}$, and a hyperparameter $\sigma\in\mathbb{R}$. Here we set $\sigma=1.0$. Then, we have

Therefore, this initialization results in the central position of the convolutional kernel having the maximum weight, while the edge positions of the kernel have relatively smaller weights. This is typically beneficial for convolutional layers to be more sensitive to the central position when recognizing specific features. Given a multivariate time series input $X_{embed}\in\mathbb{R}^{N\times D}$, where $N$ represents the dimensionality of channels, corresponding to the number of variables in the time series, and $D$ is the dimensionality of the embedding after positional and temporal encoding, to maintain the equivalence of sequence lengths before and after convolution, padding is employed using terminal values. So we get $X_{padded}\in\mathbb{R}^{N\times(D+K-1)}$, we split it into $N$ individual time series $x_{i}\in\mathbb{R}^{1\times(D+K-1)},\quad i=1,2,\ldots,N$. Subsequently, for each $x_{i}$, we apply a learnable 1D convolutional kernel with shared weights to extract its trend component denoted as $\hat{x_{i}}\in\mathbb{R}^{1\times D}$. Subsequently, the convolutional outputs of all $\hat{x_{i}}$ are concatenated to form the resultant matrix $X_{Trend}\in\mathbb{R}^{N\times D}$. We get the seasonal part $X_{Seasonal}\in\mathbb{R}^{N\times D}$ by $X_{Seasonal}=X_{embed}-X_{Trend}$. The whole process can be summarized as

Trend part. Given the smoother and more predictable nature of the trend part, we employ a simple MLP for projection to derive the trend part’s output akin to (Zeng et al., 2023; Wang et al., 2023), which reads:

Seasonal part. Considering the suitability of the seasonal component for modeling inter-series dependencies and intra-series variations, we transform $X_{Seasonal}\in\mathbb{R}^{N\times D}$ into two distinct embeddings: ‘Whole Series Embedding’ and ‘Auto-regressive Embedding’. This facilitates the modeling and learning processes for the two patterns.

We propose a ‘Dual Attention Module’ to model the inter-series dependencies and intra-series variations simultaneously. Concretely, we devise ‘Channel-wise self-attention’ for modeling the former and ‘Auto-regressive self-attention’ for modeling the latter.

Inter-series dependencies modeling. To model inter-series dependencies, follow iTranformer (Liu et al., 2024), we consider $X_{Seasonal}[i,:]\in\mathbb{R}^{1\times D},\quad i=1,2,\ldots,N$ as a token as shown in Figure 3. Subsequently, all tokens are sent into a vanilla transformer encoder for learning purposes to get $X_{Inter}\in\mathbb{R}^{N\times D}$:

We denote this process as ‘Channel-wise self-attention’ and the generated embeddings as ‘Whole Series Embedding’, which, while, maintaining most information of the sequence in contrast to patches or segments, is better suitable for modeling inter-series dependencies, as all semantic information of the variates are saved.

Intra-series variations modeling. To capture intra-series variations, we propose an advanced methodology. Herein, the initial sequence undergoes auto-regressive processing, generating tokens that, while meticulously retaining the entirety of the original information, partially emulate the dynamic variations present in raw time series.

For intra-series variations, as shown in Figure 3, we first split $X_{Seasonal}$ into $N$ individual time series $x_{s}^{i}\in\mathbb{R}^{1\times D},\quad i=1,2,\ldots,N$. For each $x_{s}^{i}$, given a length $L$, we generate $S^{i}\in\mathbb{R}^{\frac{D}{L}\times D}$ tokens by cutting the given length of the sequence from the beginning and concatenate it to the end of the time series:

Tokens generated in this process are nominated as ‘Auto-regressive Embedding’. This auto-regressive token generation approach enables the dynamic simulation of temporal variations within the time series. In contrast to methods involving the utilization of sampling strategies to derive sub-sequences or the partitioning of the primary time series into patches or segments, which inevitably lead to information loss, this approach maximally preserves the information of the original time series. Then we still use another vanilla transformer encoder whose weight is shared across all channels to model intra-series variations, but considering our primary interest in the temporal information of the raw sequence, we designate the raw sequence $x_{s}^{i}$ solely as Q, while utilizing the entire sequence $S^{i}$ as both K and V:

And the output $X_{Intra}^{i}$ of all channels is concatenated as $X_{Intra}\in\mathbb{R}^{N\times D}$. The ultimate output results of the seasonal part have been obtained by:

Prediction generating. We obtain the ultimate predictive outcomes through $\hat{Y}=X_{S_{out}}+X_{T_{out}}$.

In this section, we first introduce the whole experiment settings under a fair comparison. Secondly, we illustrate the experiment results by comparing Leddam with the eight current state-of-the-art (SOTA) methods. Further, we conducted an ablation study to comprehensively investigate the effectiveness of the learnable convolutional decomposition module and the effectiveness of the Dual attention module.

Datasets. We conduct extensive experiments on selected eight widely-used real-world multivariate time series forecasting datasets, including Electricity Transformer Temperature (ETTh1, ETTh2, ETTm1, and ETTm2), Electricity, Traffic, Weather used by Autoformer (Wu et al., 2021), and Solar-Energy datasets proposed in LSTNet (Lai et al., 2018). For a fair comparison, we follow the same standard protocol (Liu et al., 2024) and split all forecasting datasets into training, validation, and test sets by the ratio of 6:2:2 for the ETT dataset and 7:1:2 for the other datasets. The characteristics of these datasets are shown in Table 1 (More can be found in the Appendix A.1).

Evaluation protocol. Following TimesNet (Wu et al., 2023), we use Mean Squared Error (MSE) and Mean Absolute Error (MAE) as the core metrics for the evaluation. To fairly compare the forecasting performance, we follow the same evaluation protocol, where the length of the historical horizon is set as $T=96$ for all models and the prediction lengths $F\in\{96,192,336,720\}$. Detailed hyperparameters of Leddam can be found in the Appendix A.2.

Baseline setting. We carefully choose very recently EIGHT well-acknowledged forecasting models as our baselines, including 1) Transformer-based methods: iTransformer (Liu et al., 2024), Crossformer (Zhang & Yan, 2023), PatchTST (Nie et al., 2023); 2) Linear-based methods: DLinear (Zeng et al., 2023), TiDE (Das et al., 2023); and 3) TCN-based methods: SCINet (LIU et al., 2022), MICN (Wang et al., 2023), TimesNet (Wu et al., 2023).

Quantitative comparison. Comprehensive forecasting results are listed in Table 2 with the best bold in red and the second underlined in blue. We leave full forecasting results in APPENDIX G to save place. The lower MSE/MAE indicates the better prediction result. It is unequivocally evident that Leddam has demonstrated superior predictive performance across all datasets except Traffic, in which iTransformer gets the best forecasting performance. Like PatchTST and iTransformer, Leddam employs a vanilla transformer encoder as its backbone, devoid of any structural modifications. It is noteworthy, however, that these three models consistently exhibit superior performance across all datasets. This at least partially indicates that, compared to intricately designed model architectures, superior representation of raw time series features, such as ‘Whole Series Embedding’ used in iTransformer and Leddam along with PatchTST’s patches, may indeed constitute the pivotal factors for achieving more efficient time-series predictions. It is noteworthy that among these three models, PacthTST employs a channel-independent design, exclusively addressing intra-series variations without considering inter-series dependencies. iTransformer utilizes channel-wise self-attention to model inter-series dependencies but falls short of adequately capturing intra-series variations. Compared to both, the proposed Leddam incorporates ‘Whole Series Embedding’ and ‘Auto-regressive Embedding’ to model inter-series dependencies and intra-series variations. Consequently, Leddam demonstrates superior performance across various datasets. However, these three models still maintain a leading position over other models across most datasets. This aligns with our hypothesis that appropriately modeling inter-series dependencies and intra-series variations in multivariate time series is key for achieving more precise forecasting.

Ablation study of dual attention module We conducted ablation experiments across five datasets, including ETTh1, Traffic, Electricity, Solar-Energy, and Weather, to validate the performance enhancement introduced by our ‘Auto-regressive self-attention’ and ‘Channel-wise self-attention’ components. ‘Channel’ means we only used ‘Channel-wise self-attention’. ‘Auto’ means we only used ‘Auto-regressive self-attention’. ‘w/o All’ means we simply replace the ‘Auto-regressive self-attention’ and ‘Channel-wise self-attention’ components with a linear layer.

We observe substantial performance improvements in Table 3 introduced by the ‘Auto-regressive self-attention’ and ‘Channel-wise self-attention’ components. ‘Auto-regressive self-attention’ brings an average of 19.02% of MSE to decrease across five datasets compared to using linear layer, and ‘Channel-wise self-attention’ achieves 21.09% improvement. Moreover, their synergistic integration yields further enhancements in model performance, getting an average 25.03% of MSE decrease, reaching an optimal level. This attests to the efficacy of both design elements and once again validates our hypothesis, namely, appropriately modeling inter-series dependencies and intra-series variations in multivariate time series can yield better predictive performance.

Superiority of Learnable Decomposition Module over Moving Average Kernel. To better emphasize the advantages of our Learnable Decomposition Module with weights initialized using a Gaussian distribution, in comparison to conventional Moving Average Kernel, we conducted an extensive experiment. Given that DLinear arguably represents the most prominent instantiation utilizing a Moving Average Kernel for trend information extraction, and achieves performance comparable to other state-of-the-art methods with the mere use of two simple linear layers, we select it as our baseline. For comparison, we substitute its Moving Average Kernel with our Learnable Decomposition Module, denoting the modified model as LD_TL if the kernel is set to trainable, else LD_UTL if the kernel is set to untrainable.

In comparison to a simple Moving Average Kernel, the Learnable Decomposition Module, as depicted in Table 4, consistently exhibits superior predictive performance across all four datasets regardless of its trainability. The untrainable version of the 1D convolutional kernel brings an average 7.28% of MSE decrease across five datasets compared to using a Moving Average Kernel, while the trainable version gives 11.98%. The obtained results conclusively demonstrate the superior efficacy of our Learnable Decomposition Module over a simple Moving Average Kernel, and the trainability of the kernel plays a significant role in its adaptability. We leave further analysis to APPENDIX B.1-B.4.

Decomposition result analysis. To further investigate why our Learnable Decomposition Module (LD) is a better time series decomposition solution than Moving Average Kernel (MOV), we conducted the following analysis on the seasonal part and trend part obtained.

Since the seasonal part represents repetitive patterns in the raw sequence, a good seasonal part should capture all major frequencies in the raw sequence. We separately calculated the amplitude similarity of the dominant frequencies (top 25%) between the seasonal part obtained by each method and the raw sequence, denoted as FFT. A better decomposition strategy should yield a seasonal part with higher similarity to the dominant frequencies of the raw sequence.

A good trend part should effectively capture the trend changes in the original sequence. Thus, we employed Dynamic Time Warping (DTW) (Mü, 2007) to compute the similarity between the raw sequence and the trend parts obtained from the two decompositions, denoted as DTW. A superior decomposition strategy should result in a trend part with higher DTW similarity to the raw sequence.

We selected the final variate of each dataset. The variates used are as follows.

• •

  ETT Dataset: Oil Temperature (OT) every hour or 15 minutes.

• •

  Electricity: hourly electricity consumption of the 321st (last) user.

• •

  Solar-Energy: solar power production every 10 minutes of the 137th (last) PV plant.

• •

  Traffic: hourly road occupancy rates measured by the 862nd (last) sensor.

• •

  Weather: CO2 (ppm) collected every 10 minutes.

LD is pre-trained on task: input-96-forecast-720. To avoid cherry-picking, both metrics are computed by averaging the results calculated over the entire test dataset.

The DTW and FFT of LD are consistently superior to that of the MOV across all eight datasets in Table 5. This demonstrates that LD is a better time series decomposition method compared to MOV.

Analysis of different attention mechanisms in inter-series dependencies modeling. We devised a comprehensive experiment to investigate three prevalent approaches for modeling inter-series dependencies: ‘Channel-wise self-attention’, ‘Point-wise self-attention’, and ‘Patch-wise self-attention’. The first considers the entire sequence of a variate as a token (Liu et al., 2024), the second regards distinct variables at the same timestamp as tokens (Zhou et al., 2022a, c), and the third treats patches of the raw sequence as tokens (Zhang & Yan, 2023; Nie et al., 2023). Specifically, we eliminate the ‘Auto-regressive self-attention’ branch from the original Leddam structure to concentrate on inter-series dependencies modeling. We sequentially employ ‘Channel-wise self-attention’, ‘Point-wise self-attention’, and ‘Patch-wise self-attention’. In Figure 4, it is readily apparent that compared to ‘Point-wise self-attention’, and ‘Patch-wise self-attention’, ‘Channel-wise self-attention’ exhibits superior predictive performance, implies a much better inter-series dependencies modeling ability.

Analysis of different attention mechanisms in intra-series variations modeling. Similarly, we replace the ‘Channel-wise self-attention’ branch from the original Leddam structure and use ‘Point-wise self-attention’, ‘Patch-wise self-attention’, and ‘Auto-regressive self-attention’ for intra-series variations modeling. The superiority of the ‘Auto-regressive self-attention’ architecture is proved by the experimental results in Figure 4.

In this subsection, we investigate the generalizability of the learnable decomposition module of Leddam by plugging it into other different kinds of models.

Model selection and experimental setting. To achieve this objective, we conduct experiments across a spectrum of representative time series forecasting model structures, including (1) Transformer-based methods: Informer (Zhou et al., 2022a), Transformer (Vaswani et al., 2017); (2) Linear-based methods: LightTS (Zhang et al., 2022); and (3) TCN-based methods: SCINet (LIU et al., 2022); (4) RNN-based methods: LSTM (Hochreiter & Schmidhuber, 1997). We standardize the input length $T$ to 96, and similarly, the prediction length $F$ is uniformly set to 96. Subsequently, comparative experiments were conducted on five datasets: ETTh2, ETTm2, Weather, Electricity, and Traffic. Specifically, we sequentially replaced the ‘Auto-regressive self-attention’ and ‘Channel-wise self-attention’ components in Leddam with each model. The comparative analysis was performed to assess the predictive performance before and after the incorporation of Leddam, comparing the original models with the augmented counterparts. And more results can be found in APPENDIX D.

Quantitative results. In Table 6, it is apparent that the incorporation of the Leddam structure leads to a notably substantial enhancement in the predictive performance of various models, even with the introduction of only a single linear layer. Specifically, LightTS demonstrates an average MSE reduction of 11.87% across five datasets, other models are LSTM: 48.56%, SCINet: 23.15%, Informer: 31.72%, and Transformer: 26.27%. Particularly noteworthy is the performance enhancement observed in the classical LSTM model, where the MSE experiences a remarkable decrease of 76.97% and 80.28% on ETTh2 and ETTm2, respectively, a profoundly surprising result. This unequivocally substantiates the generality of the Leddam structure.