AnomalyBERT: Self-Supervised Transformer for Time Series Anomaly Detection using Data Degradation Scheme

An overview of our framework is illustrated in Figure 2. Our model is composed of three parts; a linear embedding layer, a Transformer body, and a prediction block. A window of multivariate time series $X=x_{t_{0}:t_{1}}\in\mathbb{R}^{N\times D}$ is fed into the model as an input. The linear embedding layer first projects each data patch $x_{t:t+p}$ (a patch consists of several neighboring points) in a window $X$ to an embedded feature $f_{i}$. Then the Transformer body takes all embedded features $\{f_{i}\}_{1\leq i\leq M}$ from $X$ and yields latent features $\{h_{i}\}_{1\leq i\leq M}$. These latent features share information among themselves and reflect the temporal context in the window. The prediction block finally outputs anomaly scores of data points $a_{t_{0}:t_{1}}\in[0,1]^{N}$ of the window. A data point $x_{t}$ is regarded as more anomalous as the score $a_{t}$ is closer to 1.

We adopt the Transformer encoder, in which each layer contains a multi-head self-attention (MSA) module and an MLP block, as the main body. A LayerNorm (LN) layer is placed before each module, and GELU activation is used for activation layers. Unlike the original Transformer or ViT, we do not use sinusoidal positional encodings (Vaswani et al., 2017) or absolute position embeddings (Dosovitskiy et al., 2020) to inject positional information. We instead add 1D relative position bias (Raffel et al., 2019; Liu et al., 2021) to each attention matrix to consider the relative positions between features in a window. A self-attention in each head with the relative bias is computed as:

$$\textrm{Attention}(Q,K,V)=\textrm{SoftMax}\left(\frac{QK^{T}}{\sqrt{d}}+B\right)V,$$

(1)

where $Q$, $K$, and $V$ are query, key, and value of input features, respectively, and $d$ is the dimension of features in an attention head. $B=[b_{i,j}]\in\mathbb{R}^{M\times M}$ is a relative position bias and an element $b_{i,j}=\hat{b}_{j-i}$ is brought from a learnable bias table $\hat{B}=\{\hat{b}_{n}\}_{-M+1\leq n\leq M-1}$. We apply a different position bias to each MSA module as in Liu et al. (2021).

As we use unlabeled training data, we create degraded inputs by replacing a portion of a window with an outlier in the training phase (Figure 3). Similar to the span masking in SpanBERT (Joshi et al., 2020), we randomly select an interval $[t^{{}^{\prime}}_{0},t^{{}^{\prime}}_{1}]\subset[t_{0},t_{1}]$ in a window $X=x_{t_{0}:t_{1}}$. The selected sequence $X^{{}^{\prime}}=x_{t^{{}^{\prime}}_{0}:t^{{}^{\prime}}_{1}}$ is replaced with one of the synthetic outliers below.

• •

  A weighted sequence with the outside of the window (Soft replacement).

• •

  A constant sequence (Uniform replacement).

• •

  A lengthened or shortened sequence (Length adjustment).

• •

  A single peak value (Peak noise).

The soft replacement denotes the replaced sequence fetched from the outside of the window. Technically, it represents the replacement with a weighted sum of the original interval and an external interval. The uniform replacement is the replacement with a constant sequence, and the length adjustment denotes a lengthened or shortened sequence. Lastly, the peak noise is the addition of a single peak value. Unlike the existing method (Lai et al., 2021), our data degradation scheme can be processed without prior knowledge of a given time series.

We apply the binary cross entropy loss to our objective. For an input window $X=x_{t_{0}:t_{1}}$ with a degraded interval $[t^{{}^{\prime}}_{0},t^{{}^{\prime}}_{1}]$ and predicted anomaly scores $a_{t_{0}:t_{1}}$, the objective function is defined as:

$$L=-\frac{1}{N}\sum_{t=t_{0}}^{t_{1}}\bm{1}_{\mathrm{[t^{{}^{\prime}}_{0},t^{{}^{\prime}}_{1}]}}(t)\cdot\log a_{t}+\left(1-\bm{1}_{\mathrm{[t^{{}^{\prime}}_{0},t^{{}^{\prime}}_{1}]}}(t)\right)\cdot\log(1-a_{t}),$$

(2)

where $N=t_{1}-t_{0}+1$ is the window size. The function $\bm{1}_{\mathrm{[t^{{}^{\prime}}_{0},t^{{}^{\prime}}_{1}]}}$ plays a role of artificial labels in this equation. Compared to the MLM in the field of NLP (Devlin et al., 2018; Yang et al., 2019), our model is directed to classify the entire data points in a window into normal/abnormal points at once. At every training step, a synthetic outlier of random type, length, and values is added to an original window under the data degradation scheme. A mini-batch of degraded windows is fed into the model, and the model is trained to detect the degraded parts. The implementation details of the training procedure are described in Appendix A.2.

We create a simple sine wave dataset that consists of a normal sequence and five abnormal sequences categorized by Lai et al. (2021), and conduct a preliminary experiment on this dataset. We then produce the experimental results on five widely-used benchmark datasets, SWaT, WADI, SMAP, MSL, and SMD (Goh et al., 2017; Ahmed et al., 2017; Hundman et al., 2018; Su et al., 2019). These datasets are collected from multiple sensors in server machines, spacecrafts, or water treatment/distribution systems. Each dataset consists of an unlabeled training set and a labeled test set. The information of datasets is summarized in Table 4 and described in Appendix A.1 in detail.

In the evaluation stage, the trained model takes windows in the test set and predicts anomaly scores of the data points. We use the sliding window strategy (Shen et al., 2020) and average the scores on the overlapped intervals. After the score prediction, we categorize data points whose anomaly scores exceed a threshold as anomalies. We mainly use F1-score ($\mathrm{F1}$) over the ground-truth labels and anomaly predictions to evaluate the performance. We count the number of true positives (TP), false positives (FP), and false negatives (FN), and compute $\mathrm{F1}$ as $2\text{TP}\,/\,(2\text{TP}+\text{FP}+\text{FN})$. In practice, many approaches process prediction results using the point adjustment (Xu et al., 2018), in which the entire points in an abnormal segment are regarded as anomalous if at least one point is detected as an anomaly. This process, however, has been shown to overestimate the detection performance because it may increase TP but decrease FN dramatically (Kim et al., 2022). Following the protocol in Kim et al. (2022), thresholds that yield the best $\mathrm{F1}$ and F1-score after the point adjustment ($\mathrm{F1}_{\mathrm{PA}}$) are obtained, and the best evaluation values are used for comparisons.

Previous work (Lai et al., 2021) categorizes sequential outliers into five specific behavior-driven taxonomy; global, contextual, shapelet, seasonal, and trend outliers. Synthesizing such patterns in time series data is a difficult task because it strictly requires knowledge of patterns that appear in existing data in advance. For example, it is necessary to select a shape to be added to synthesize a shapelet outlier, and the data trend is required to synthesize a trend outlier. However, our synthesis technique covers these five outliers in an easier way because it does not require analysis of the original data. We now refer to these five outlier types as the typical outlier types.

In Table 1, we examine the relationships between the typical outlier types and our synthetic outliers using a simplified version of our model and the sine wave dataset. We measure $\mathrm{F1}$ and area under ROC curve (AUROC) for comparisons of the model performances, and consider that a synthetic outlier covers a typical outlier type ( mark) if a model trained with the synthetic one achieves both $\mathrm{F1}$ and AUROC over 0.9 on the sine wave including the corresponding typical outlier. As shown in Table 1, the soft replacement covers all typical types of outliers, and the uniform replacement and peak noise partially cover them. Meanwhile, the length adjustment only covers the seasonal outlier. From the simple experiment, we draw inspiration and get ideas for imitating anomalous behavior using synthetic outliers.

We report the results of AnomalyBERT on the five real-world datasets introduced in Section 4.1, and compare them to those of the previous works. The reproduced evaluation scores are brought from Kim et al. (2022). In Table 2, we show that AnomalyBERT outperforms the previous methods on all datasets with $\mathrm{F1}$. Our method particularly surpasses the others on MSL and SMAP, which may contain unlabeled outliers in the training set. Despite the difficulty in training networks, our method detects anomalies well in this kind of data. AnomalyBERT also performs well with $\mathrm{F1}_{\mathrm{PA}}$, although $\mathrm{F1}_{\mathrm{PA}}$ tends to distort the model performance. Our qualitative results are visualized in Figure 4.

We conduct ablation studies on various synthetic outliers in the training phase and show their impacts on the model performance. In Table 3(a), we first report the model performance affected by the three types of outliers, soft replacement, uniform replacement, and peak noise on WADI dataset. We set a baseline by mixing all three outlier types and set experimental conditions by excluding outliers from the baseline one by one. The sum of all probability of synthesizing outliers is fixed at 80%. As shown in Table 3(a), using all three outliers yields the best $\mathrm{F1}$, and using the soft replacement and peak noise yields the next. The absence of soft replacement obviously reduces the capability of the model. Also noteworthy, mixing other outlier types with soft replacement enhances the performance compared to using it only. This indicates that the uniform replacement and peak noise complement the soft replacement, though each of them does not perfectly cover the typical outliers in Table 1.

On the other hand, the length adjustment has different influences depending on the datasets. In Table 3(c) and 3(c), we report the results of ablation studies on the length adjustment on SWaT and WADI datasets. Using the length adjustment in the training stage enhances the model performance on SWaT dataset, but it degrades that on WADI dataset. Because the length adjustment specializes in detecting abnormal frequencies as shown in Table 1, it may confuse the model if data contains various frequencies (SMAP, MSL) or low-frequencies (WADI).