Self-Supervised Time Series Representation Learning via Cross Reconstruction Transformer

As an emerging research field, self-supervised learning has drawn much attention, especially for image data and text data [26, 27, 28, 29, 30]. Self-supervised learning is to manually design “pretext tasks” on unlabelled data, which is a way to provide “supervised” information for deep learning models. The model is expected to capture inherent characteristics among large-scale unlabelled data, and show a better performance on downstream tasks compared to training from scratch [31].

For text data, BERT [32] is a typical example of self-supervised learning, which is designed to pre-train deep bidirectional representations from the unlabeled text by joint conditioning on both left and right context in all layers. Due to BERT’s great performance, multiple variations were proposed. For example, SpanBERT proposed to mask random contiguous spans, rather than random individual tokens [33]. ERNIE proposed entity-level masking and phrase-level masking, which integrate phrase and entity-level knowledge into the language representation [34]. For image data, contrastive learning is more popular to learn representations in an unsupervised way. Contrastive learning methods typically rely on discriminating manually constructed pairs through an InfoNCE loss [15]. InfoNCE maximizes the similarity between positive instances and minimizes the similarity between negatives. The largest challenge and difficulty is how to construct reasonable and practical positive and negative sets to facilitate the model to capture similarities and differences among instances. Some works propose to construct pairs via data augmentations methods [29, 35], clustering [36, 37], and so on. Contrastive learning is also applied to other fields, such as graph representation learning [38], multi-modal learning [39], and so on. However, contrastive learning heavily depends on constructing positive and negative pairs, which makes the performance less robust and unstable. Recent works have started to apply reconstruction-based self-supervised learning on image and other data [40, 41], which are straightforward but show great performance.

Research on self-supervised representation learning on sequence data has been well-studied, but representation learning for time series still needs to be promoted. Inspired by the well-studied self-supervised learning methods in computer vision and natural language processing areas, recent works mainly leverage the contrastive learning framework for time series representation learning [14]. The researchers design different time-slicing strategies to construct positive and negative pairs with the assumption that temporally similar fragments could be viewed as positive samples and remote fragments are treated as negative samples [42]. Unsupervised Scalable Representation Learning [12] introduces a novel unsupervised loss with time-based negative sampling to train a scalable encoder, shaped as a deep convolutional neural network with dilated convolutions [43]. Temporal Neighborhood Coding [13] introduces the concept of a temporal neighborhood with stationary properties as the distribution of similar windows in time. Nevertheless, these time-based methods are sometimes unreasonable for long time series and fail to capture the long-term dependencies [14]. Besides, the performance will substantially deteriorate when applied to downstream tasks containing periodic time series.

A recent work [23] proposes a Transformer-based reconstruction self-supervised task for time series data for the first time. This method masks part of the original time series data by setting the values as zeros, and uses a linear layer to reconstruct the masked data. The results show that reconstructing the masked data can help extract dense vector representations of multivariate time series and facilitate models to understand contextual information. However, masking original data introduces a gap between the pre-training stage and fine-tuning stage, because it changes the original distribution of time series significantly. In addition, all methods including contrastive learning methods neglect an important characteristic of time series data - the frequency domain.

Masking parts of data and reconstructing them is a common paradigm in self-supervised learning tasks [32, 40]. Inspired by this, some works mask the original time series by setting the value of certain temporal segments to 0 and reconstructing these segments [23]. However, this may significantly change the original pattern of time series and bring noises to the representation learning process. Besides, it may cause the gap between the pre-training stage and fine-tuning stage since it leads to distribution shifts. As shown in Figure 3(b), masking significantly changes the shape and distribution of the original time series. Different from images or text, the shape is an important and unique pattern for time series [44]. Thus masking would bring noises into the pre-training stage and deteriorate the downstream performance.

In this work, we choose to drop some segments of the time series rather than masking them. Dropping discards certain parts of the input time series, and feeds the rest of the data into the model. The difference between masking and dropping is shown in Figure 3. In detail, we slice the input into patches with the same length, and randomly discard patches in a probability of $r$ following:

where $p$ is a random value following uniform distribution $U(0,1)$. In this way, the model receives segments of real data without corrupted zeros portions. Adding corresponding position information of undropped parts to the Transformer model can maximally preserve the global context and capture the intrinsic long-term dependencies.

When learning representations for time series, most of the existing methods neglect to involve the spectral information of time series. The frequency domain provides another perspective to discover the patterns for time series data. The spectral perspective can provide several characteristics of the time series that are not easily acquirable in the time domain. Consequently, we attempt to explicitly input both time and frequency domains into the model to enable it to learn both temporal and spectral information and model cross-domain correlations.

The most common method to transform time series into the frequency domain is the Fast Fourier Transform (FFT) [45], which is a rapid implementation of the Discrete Fourier Transform (DFT):

where $\mathbf{t}$ is original time series data, $N$ is the length of $\mathbf{t}$, $k$ is the index of frequency data, $i$ is called imaginary unit satisfying $i^{2}=-1$, and $\mathcal{F}$ is the frequency domain data. After FFT, the original time series is transformed into the frequency domain, and is converted to a sequence of complex numbers. However, these complex numbers cannot be used to train the neural networks directly. Most of the methods considering the frequency domain attempt to store the spectral information using the magnitude of the complex numbers. For any complex number $z=a+bi$, where $a$ and $b$ are arbitrary real numbers. The magnitude of $z$ is $||z||=\sqrt{a^{2}+b^{2}}$. However, using only magnitude to restore the complex numbers would cause information loss. Because we cannot restore $z$ only using $||z||$.

To tackle this problem, we propose a novel way to better utilize the spectral information rather than simply using the magnitude. We introduce phase $\phi(z)\in(-\pi,\pi]$ as follows:

where

Phase is another characteristic of the frequency domain. Intuitively, a complex number $z$ can be regarded as a point $P$ of a circle with radius of $||z||$ and the angle between $\vec{OP}$ and the positive direction of x-axis. With phase and magnitude, we can restore the full spectral information which is represented by the complex number $z$ through

In addition to complementing magnitude, the phase contains more information relative to the shape of the times series. We compare the information contained by phase and magnitude by restoring the original time series only using phase or magnitude. In detail, we calculate the phase and magnitude of a time series, and replace the phase or magnitude with a constant vector. In both cases, we restore the complex vector following Equation 4, and conduct inverse FFT to transform complex vectors to the time domain. As shown in Figure 4, we can see that the original time series is more similar to the restored time series with the correct phase. This intuitively explains the importance of phase information for the shape of time series data.

In this section, we introduce our Dropped Temporal-Spectral Modeling. For a given time series data $\mathbf{t}$, we firstly conduct FFT and convert $\mathbf{t}$ from the time domain to the frequency domain. The obtained vector is a complex vector, of which we store both the magnitude and the phase as mentioned in Section III-B. Thus, the complex vector is transformed into two real vectors $\mathbf{m}$ (magnitude) and $\mathbf{p}$ (phase) with the same length. We then concatenate $\mathbf{t}$, $\mathbf{m}$, and $\mathbf{p}$ to get the input $\mathbf{x}=[\mathbf{t},\mathbf{m},\mathbf{p}]$ of length $L$.

For the input $\mathbf{x}$, we choose to drop parts of it as mentioned in Section III-A. Each of the sliced patches contains only one type of data (time, magnitude, or phase). After that, we drop $r\times N$ ($r\in(0,1)$ is the given dropping ratio and $N$ is the number of patches) patches and use remaining $(1-r)\times N$ patches to reconstruct the dropped patches. For patches containing time information, we randomly drop them. Also, for patches containing frequency information (phase or magnitude), we drop the corresponding pair of patches (phase and magnitude of the same segments of complex vectors). As explained in Section III-A, dropping the patches can reduce the gap between the pre-training stage and fine-tuning stage because it does not change the shape of the original data.

Before the Transformer encoder, we add three convolutional neural networks (CNNs) to extract local features for three types of patches respectively. We hypothesize CNN can help extract local features, and Transformer can help model cross-domain information using the local features. The patches $\mathbf{x_{i}}$ from time domain and magnitude and phase are projected to three types of embeddings $\mathbf{E_{i}}$ respectively. We then add a different learnable [CLS] token before each type of embeddings (the [CLS] before the time, magnitude, and phase embeddings are referred to as $\mathbf{E_{T}}$, $\mathbf{E_{M}}$ and $\mathbf{E_{P}}$ respectively). The three types of embeddings are summed with their corresponding learnable domain-type embeddings and concatenated into a vector $\mathbf{E}\in\mathbb{R}^{(N+3)\times D}$, where $D$ is the dimension of the embeddings. Before fed into the Transformer, $\mathbf{E}$ is summed with the corresponding learnable position embeddings.

We input the whole $\mathbf{E}$ into the Transformer encoder and use the produced features to reconstruct the original data (time, magnitude, and phase). The learned representations are used to reconstruct the original data of both domains, so that the Transformer encoder can automatically capture complementary information from both domains.

Formally, in a batch, there are $B$ inputs $\mathbf{x_{i}},i=1,\dots,B$. $\mathbf{x_{i}}$ is projected to $\mathbf{E_{i}}\in\mathbb{R}^{(N+3)\times D}$ by CNNs, and $\mathbf{E_{i}}$ is projected by Transformer to $\mathbf{E_{i}^{\prime}}\in\mathbb{R}^{(N+3)\times D}$. We use $\bar{\mathbf{E_{i}‘}}=\frac{\mathbf{E_{T}^{\prime}}+\mathbf{E_{M}^{\prime}}+\mathbf{E_{P}^{\prime}}}{3}$ for down-stream tasks. We concatenate each $\mathbf{E_{i}^{\prime}}$ with learnable decoding tokens, and feed them into the decoder to reconstruct all patches. The reconstruction loss is:

where $\mathbf{x^{\prime}}$ is the reconstructed input (including both dropped and undropped patches), and d is the dimension of $\mathbf{x}$.

When reconstructing the original data of both domains, we hope the latent representations contain more sample-specific information and less similar information to other samples. Hence, we propose a simple Instance Discrimination Constraint (IDC) module to remove redundancy and sharpen decision boundary. IDC maximizes the mutual information between the representations and original input, and simultaneously constrains the redundant information that other samples have in common. Also, we maximize the distance between $\mathbf{\bar{E_{i}^{\prime}}}$ and $\mathbf{E_{j}}$ where $i\neq j$ in latent space, that is minimizing:

where $B$ is the batch size, $sim(\cdot,\cdot)$ is cosine similarity $sim(x,y)=\frac{x\cdot y}{||x||||y||}$, and $Proj_{1},Proj_{2}$ are two projection heads to project $\mathbf{\bar{E_{i}^{\prime}}}$ and $\mathbf{E_{j}}$ into the same latent space. The projection heads can be either a linear layer or a multi-layer perceptron, and we use a two-layer perceptron in our experiments.

Combining the reconstruction loss and IDC loss, our final loss is:

where $\beta$ is a hyper-parameter. By updating this loss, we hope the representations learned by the model contain as much sample-specific information as possible. Thus, the representations is both informative and discriminable. As for the complexity, the FFT process is conducted before pre-training the model and frequency data is directly used during self-supervised training. Consequently, the increase in computational complexity of optimizing $\mathcal{L}$ is negligible.

However, it is difficult to reconstruct the dropped patches when $r$ is initially set as a large value. As a result, we introduce Curriculum Learning (CL) [46], which is a strategy that trains the model from “easy” to “hard”. In detail, we set a relatively small dropping ratio $r$ at the beginning of the pre-training stage, and increase $r$ as the self-supervised learning goes. Formally, we refer to the initial $r$ as $r_{min}$, final $r$ as $r_{max}$, and the number of epochs for increasing $r$ as $N_{epoch}$. $r_{min}$ and $r_{max}$ are hyperparameters, and should be tuned on the validation set. Then, the $r$ for each epoch $i$ is

When the current epoch $i$ is smaller than $r_{min}\times N_{epoch}$, $r_{i}$ is equal to $r_{min}$. When $r_{min}\times N_{epoch}\leq i\leq r_{max}\times N_{epoch}$, $r_{i}$ is equal to $\frac{i}{N_{epoch}}$. Then, $r_{i}$ remains equal to $r_{max}$ until the pre-training stage ends.

We compare our CRT with baseline methods and show the results in Table IV. Our CRT outperforms all baselines in terms of ROC-AUC, F1-score and accuracy. We also observe that the results of CRT have a relatively small standard deviation, which implies a more stable performance on downstream tasks. In addition, we notice that TST (another reconstruction-based self-supervised learning method) also obtains a relatively small standard deviation. The reason might be that reconstruction tasks do not require constructing negative and positive pairs, which is difficult and may cause instability. It indicates that reconstruction-based methods may provide a more stable way for self-supervised learning. One baseline named Scalable Representation Learning [12] is not included in our results, as it requires a much longer running time and we failed to produce its results in several days.

More ablation studies and comparison experiments are conducted to further verify the effectiveness of our method.