SparseTSF: Modeling Long-term Time Series Forecasting with 1k Parameters

The task of LTSF involves predicting future values over an extended horizon using previously observed multivariate time series (MTS) data. It is formalized as $\bar{x}_{t+1:t+H}=f(x_{t-L+1:t})$, where $x_{t-L+1:t}\in\mathbb{R}^{L\times C}$ and $\bar{x}_{t+1:t+H}\in\mathbb{R}^{H\times C}$. In this formulation, $L$ represents the length of the historical observation window, $C$ is the number of distinct features or channels, and $H$ is the length of the forecast horizon. The main goal of LTSF is to extend the forecast horizon $H$ as it provides richer and more advanced guidance in practical applications. However, an extended forecast horizon $H$ also increases the complexity of the model, leading to a significant increase in parameters in mainstream models. To address this challenge, our research focuses on developing models that are not only extremely lightweight but also robust and effective.

Recent advancements in the field of LTSF have seen a shift towards a Channel Independent (CI) approach, especially when dealing with multivariate time series data (Han et al., 2024). This strategy simplifies the forecasting process by focusing on individual univariate time series within the dataset. Instead of the traditional approach, which utilizes the entire multivariate historical data to predict future outcomes, the CI method finds a shared function $f:x^{(i)}_{t-L+1:t}\in\mathbb{R}^{L}\rightarrow\bar{x}^{(i)}_{t+1:t+H}\in\mathbb{R}^{H}$ for each univariate series. This approach provides a more targeted and simplified prediction model for each channel, reducing the complexity of accounting for inter-channel relationships.

As a result, the main goal of mainstream state-of-the-art models in recent years has shifted towards effectively predict by modeling long-term dependencies, including periodicity and trends, in univariate sequences. For instance, models like DLinear achieve this by extracting dominant periodicity from univariate sequences using a single linear layer (Zeng et al., 2023). More advanced models, such as PatchTST (Nie et al., 2023) and TiDE (Das et al., 2023), employ more complex structures on single channels to extract temporal dependencies, aiming for superior predictive performance. In this paper, we adopt this CI strategy as well and focus on how to create an even more lightweight yet effective approach for capturing long-term dependencies in single-channel time series.

Assuming that the time series $x^{(i)}_{t-L+1:t}$ has a known periodicity $w$, the first step is to downsample the original series into $w$ subsequences of length $n=\left\lfloor\frac{L}{w}\right\rfloor$. A model with shared parameters is then applied to these subsequences for prediction. After prediction, the $w$ subsequences, each of length $m=\left\lfloor\frac{H}{w}\right\rfloor$, are upsampled back to a complete forecast sequence of length $H$.

Intuitively, this forecasting process appears as a sliding forecast with a sparse interval of $w$, performed by a fully connected layer with parameter sharing within a constant period $w$. This can be viewed as a model performing sparse sliding prediction across periods.

Technically, the downsampling process is equivalent to reshaping $x^{(i)}_{t-L+1:t}$ into a $n\times w$ matrix, which is then transposed to a $w\times n$ matrix. The sparse sliding prediction is equivalent to applying a linear layer of size $n\times m$ on the last dimension of the matrix, resulting in a $w\times m$ matrix. The upsampling step is equivalent to transposing the $w\times m$ matrix and reshaping it back into a complete forecast sequence of length $H$.

However, this approach currently still faces two issues: (i) loss of information, as only one data point per period is utilized for prediction, while the rest are ignored; and (ii) amplification of the impact of outliers, as the presence of extreme values in the downsampled subsequences can directly affect the prediction.

To address these issues, we additionally perform a sliding aggregation on the original sequence before executing sparse prediction, as depicted in Figure 2. Each aggregated data point incorporates information from other points within its surrounding period, addressing issue (i). Moreover, as the aggregated value is essentially a weighted average of surrounding points, it mitigates the impact of outliers, thus resolving issue (ii). Technically, this sliding aggregation can be implemented using a 1D convolution with zero-padding and a kernel size of $2\times\left\lfloor\frac{w}{2}\right\rfloor\ +1$. The process can be formulated as follows:

Time series data often exhibit distributional shifts between training and testing datasets. Recent studies have shown that employing simple sample normalization strategies between the input and output of models can help mitigate this issue (Kim et al., 2021; Zeng et al., 2023). In our work, we also utilize a straightforward normalization strategy. Specifically, we subtract the mean of the sequence from itself before it enters the model and add it back after the model’s output. This process is formulated as follows:

In alignment with current mainstream practices in the field, we adopt the classic Mean Squared Error (MSE) as the loss function for SparseTSF. This function measures the discrepancy between the predicted values $\bar{x}^{(i)}_{t+1:t+H}$ and the actual ground truth $y^{(i)}_{t+1:t+H}$. It is formulated as:

We conducted experiments on four mainstream LTSF datasets that exhibit daily periodicity. These datasets include ETTh1&ETTh2¹¹1https://github.com/zhouhaoyi/ETDataset, Electricity²²2https://archive.ics.uci.edu/ml/datasets, and Traffic³³3https://pems.dot.ca.gov/. The details of these datasets are presented in Table 1.

We compared our approach with state-of-the-art and representative methods in the field. These include Informer (Zhou et al., 2021), Autoformer (Wu et al., 2021), Pyraformer (Liu et al., 2022b), FEDformer (Zhou et al., 2022b), Film (Zhou et al., 2022a), TimesNet (Wu et al., 2023), and PatchTST (Nie et al., 2023). Additionally, we specifically compared SparseTSF with lightweight models, namely DLinear (Zeng et al., 2023) and FITS (Xu et al., 2024). Following FITS, SparseTSF defaults to a look-back length of 720.

All experiments in this study were implemented using PyTorch (Paszke et al., 2019) and conducted on a single NVIDIA RTX 4090 GPU with 24GB of memory. More experimental details are provided in Appendix A.2.

The Sparse technique, combined with a simple single-layer linear model, forms the core of our proposed model, SparseTSF. Additionally, the Sparse technique can be integrated with other foundational models, including the Transformer (Vaswani et al., 2017) and GRU (Cho et al., 2014) models. As demonstrated in the results of Table 5, the incorporation of the Sparse technique significantly enhances the performance of all models, including Linear, Transformer, and GRU. Specifically, the Linear model showed an average improvement of 4.7%, the Transformer by 21.4%, and the GRU by 12.4%. These results emphatically illustrate the efficacy of the Sparse technique. Therefore, the Sparse technique can substantially improve the performance of base models in LTSF tasks.

In Section 3.3, we theoretically analyzed the reasons why the Sparse technique can enhance the performance of forecasting tasks. Here, we further reveal the role of the Sparse technique from a representation learning perspective. Figure 3 shows the distribution of normalized weights for both the trained Linear model and the SparseTSF model. The weight of the Linear model is an $L\times H$ matrix, which can be directly obtained. However, as the SparseTSF model is a sparse model, we need to acquire its equivalent weights. To do this, we first input $H$ one-hot encoded vectors of length $L$ into the SparseTSF model (when $L$ equals $H$, this can be simplified to a diagonal matrix, i.e., diagonal elements are 1, and other elements are 0). We then obtain and transpose the corresponding output to get the equivalent $L\times H$ weight matrix of SparseTSF. When $L$ equals $H$, this process is formulated as:

From the visualization in Figure 4, two observations can be made: (i) The Linear model can learn evenly spaced weight distribution stripes (i.e., periodic features) from the data, indicating that single linear layer can already extract the primary periodic characteristics from a univariate series with the CI strategy. These findings are consistent with previous research conclusions (Zeng et al., 2023). (ii) Compared to the Linear model, SparseTSF learns more distinct evenly spaced weight distribution stripes, indicating that SparseTSF has a stronger capability in extracting periodic features. This phenomenon aligns with the conclusions of Section 3.3.

Therefore, the Sparse technique can enhance the model’s performance in LTSF tasks by strengthening its ability to extract periodic features from data.

The Sparse technique relies on the manual setting of the hyperparameter $w$, which represents the a priori main period. Here, we delve into the influence of different values of $w$ on the forecast outcomes. As indicated in the results from Table 6, SparseTSF exhibits optimal performance when $w=24$, aligning with the intrinsic main period of the data. Conversely, when $w$ diverges from 24, a slight decline in performance is observed. This suggests that the hyperparameter $w$ should ideally be set consistent with the data’s a priori main period.

In practical scenarios, datasets requiring long-term forecasting often exhibit inherent periodicity, such as daily or weekly cycles, common in domains like electricity, transportation, energy, and consumer goods consumption. Therefore, empirically identifying the predominant period and setting the appropriate $w$ for such data is both feasible and straightforward. However, for data lacking clear periodicity and patterns, such as financial data, current LTSF models may not be effective (Zeng et al., 2023). Thus, the SparseTSF model may not be the preferred choice for these types of data. Nonetheless, we will further discuss the existing limitations and potential improvements of the SparseTSF model in the Section 5.1.

The Sparse technique enhances the model’s ability to extract periodic features from data. Therefore, the generalization capability of a trained SparseTSF model on different datasets with the same principal periodicity is promising. To investigate this, we further studied the cross-domain generalization performance of the SparseTSF model (i.e., training on a dataset from one domain and testing on a dataset from another). Specifically, we examined the performance from ETTh2 to ETTh1, which are datasets of the same type but collected from different machines, each with 7 variables. Additionally, we explored the performance from Electricity to ETTh1, where these datasets originate from different domains and have a differing number of variables (i.e., Electricity has 321 variables). On datasets with different numbers of variables, models trained with traditional non-CI strategies (like Informer) cannot transfer, whereas those trained with CI strategies (like PatchTST) can, due to the decoupling of CI strategies from channel relationships. These datasets all have a daily periodicity, i.e., a prior predominant period of $w=24$.

Experimental results, as shown in Table 7, reveal that SparseTSF outperforms other models in both similar domain generalization (ETTh2 to ETTh1) and less similar domain generalization (Electricity to ETTh1). It is expected that performance on ETTh2 to ETTh1 would be superior to Electricity to ETTh1. Furthermore, in both scenarios, the generalization performance of SparseTSF is nearly on par with the performance of direct modeling in the SparseTSF source domain as shown in Table 2 and surpasses other baselines that model directly in the source domain. This robustly demonstrates the generalization capability of SparseTSF, indirectly proving the Sparse technique’s ability to extract stable periodic features.

Therefore, the SparseTSF model exhibits outstanding generalization capabilities. This characteristic is highly beneficial for the application of the SparseTSF model in scenarios involving small samples and low-quality data.

N-HiTS incorporates novel hierarchical interpolation and multi-rate data sampling techniques to achieve better results (Challu et al., 2023). The downsampling and upsampling techniques proposed in SparseTSF are indeed quite different from those used in N-HiTS, including:

• •

  The downsampling and upsampling in SparseTSF occur before and after the model’s prediction process, respectively, whereas N-HiTS conducts these operations within internally stacked modules.

• •

  SparseTSF’s downsampling involves resampling by a factor of $w$ to $w$ subsequences of length $L/w$, which is technically equivalent to matrix reshaping and transposition, whereas N-HiTS employs downsampling through max-pooling.

• •

  SparseTSF’s upsampling involves transposing and reshaping the predicted subsequences back to the original sequence, whereas N-HiTS achieves upsampling through interpolation.

Seasonal-trend decomposition (STD) is a classical and powerful tool for time series forecasting, and OneShotSTL makes a great contribution to advancing the lightweight long-term forecasting process, featuring fast, lightweight, and powerful capabilities (He et al., 2023). However, SparseTSF differs significantly from OneShotSTL in several aspects:

• •

  SparseTSF is a neural network model while OneShotSTL is a non-neural network method focused on online forecasting.

• •

  OneShotSTL minimizes residuals and calculates trend and seasonal subseries separately from the original sequence with lengths of $L$, whereas our SparseTSF resamples the original sequence into $w$ subseries of length $L/w$ with a constant period $w$.

• •

  OneShotSTL accelerates inference by optimizing the original computation for online processing, while SparseTSF achieves lightweighting by using parameter-sharing linear layers for prediction across all subseries.