DDMT: Denoising Diffusion Mask Transformer Models for Multivariate Time Series Anomaly Detection

We categorize machine learning-based anomaly detection methods into four categories: linear-based models, distance-based models, density-based models, and classifier-based models.

One popular method in linear models is Principal Component Analysis (PCA) [2]. PCA is a multivariate data analysis technique that can extract significant variability information from the data while reducing its dimensionality. It aims to preserve the diversity of the data as much as possible. However, this method is noise-sensitive and assumes that the data follow a multivariate Gaussian distribution [31].

K-means [5] is a distance-based clustering algorithm that aims to partition data points into different clusters. The algorithm calculates the distances between all samples and the cluster centers, and then determines whether certain points are outliers based on the compactness of the clusters. Another commonly used distance-based method is K-Nearest Neighbors (KNN) [3]. This algorithm calculates the distances between a data point and its K nearest neighbors and derives an anomaly score based on these distances. These distance-based methods often require prior knowledge about the duration of anomalies and the number of anomalies [13]. Moreover, they may not capture temporal correlations [12].

Local Outlier Factor (LOF) [4, 32] and Density Based Spatial Clustering of Applications with Noise (DBSCAN) [33] are density-based anomaly detection algorithms. These algorithms compute the density difference between each data point and its surrounding neighbors and calculate an outlier factor for each data point based on this difference to identify anomalies [4]. These algorithms are suitable for large-scale datasets and can be effectively scaled and computed in parallel. However, they are only effective when the outliers in the data do not form significant clusters, making them more demanding in terms of data requirements.

Bayesian networks [34] and Support Vector Machines (SVM) [35, 36] are two common anomaly detection algorithms based on classifier models. Anomaly detection algorithms based on Bayesian networks have been applied in areas such as network intrusion and image anomaly detection. However, this algorithm relies on prior knowledge to determine the dependencies between variables. If the prior knowledge is insufficient or inaccurate, it may result in an inaccurate network structure and affect the model’s effectiveness. SVM constructs a hyperplane enclosing most of the normal samples and then uses the distance from a test sample to the boundary of the hyperplane to determine if it is an anomaly. However, its training process involves solving large-scale quadratic programming problems, which can be computationally expensive, especially for high-dimensional data, requiring significant computational time and resources.

Labeling anomalies in time series requires costly expert involvement and cannot guarantee coverage of all anomalies. Therefore, deep learning-based unsupervised anomaly detection methods have been widely adopted due to their excellent performance [13]. Deep learning methods have the ability to learn complex dynamics within the data without making assumptions about underlying patterns, making them highly attractive choices in contemporary time series analysis [37]. For instance, autoencoders (AE) identify anomalies by calculating reconstruction errors through sequence reconstruction [38]. Other methods, such as VAE [10] and LSTM-based variational autoencoders [39], have also demonstrated strong performance. Chen et al. [40] proposed a semi-supervised anomaly detection method based on variational autoencoders. They introduced a parallel multi-head attention mechanism to capture the long-term dependencies in multivariate time series, while using LSTM to capture short-term dependencies. Furthermore, anomaly detection models based on Generative Adversarial Networks (GANs) have also gained attention. Li et al. [13] replaced the generator and discriminator in GANs with the structure of LSTM-RNN. They randomly generated samples in the latent space and trained the generator model to generate realistic samples. Niu et al. [41] investigated a model that combines LSTM, VAE, and GAN for TSAD. They proposed an LSTM-based VAE-GAN method by jointly training the encoder, generator, and discriminator. However, using the LSTM structure directly within the GAN models leads to significantly high computational complexity due to the iterative process involved [42], and it is prone to problems such as training instability and mode collapse.

Since Vaswani introduced the Transformer for translation [28], many Transformer-based models have been gradually applied in time series research. Chen et al. [43] combined multi-task learning with the Transformer architecture and proposed the MTL-Trans model for time series modeling and multi-dimensional time series prediction. Yan et al. [44] introduced a time series classification approach using the Informer network, which leverages both local and global dependency features of land cover time series. Wang et al. [45] proposed an unsupervised TSAD method based on the Transformer. They employed a multi-scale fusion algorithm for time series data, combining features from multiple time scales to obtain expressive representations of the time series. These studies demonstrate that Transformer-based models can better capture the long-range dependencies in time series. Therefore, we incorporate this model into anomaly detection for time series data. In contrast to the vanilla Transformer, our approach incorporates a simple yet effective masking mechanism called ADNM before performing self-attention computations with the Transformer. ADNM masks out anomalous nodes within the sequence along with their neighboring information. This approach prevents information leakage from input features to output features, thus alleviating the problem of WIM during sequence reconstruction.

Recently, the Denoising Diffusion Model has demonstrated tremendous potential in image generation tasks and has shown remarkable performance in various domains such as computer vision, natural language processing, and audio processing. For example, Brempong et al. [46] proposed a denoising-based decoder pretraining method and connected a denoising autoencoder with a diffusion probabilistic model for semantic image segmentation. Li et al. [47] developed a novel continuous diffusion-based non-autoregressive language model that enables fine-grained control in text generation. Kong et al. [48] introduced a universal diffusion probabilistic model called DiffWave for raw audio synthesis that is capable of generating high-quality audio signals for both conditional and unconditional waveform generation tasks. Currently, several methods based on the Denoising Diffusion Model are being used in time series research. For instance, Alcaraz et al. [49] combined the diffusion model with structured state space models and proposed a novel method for time series imputation. Rasul et al. [50] introduced an autoregressive model called TimeGrad for multivariate probabilistic time series forecasting. It samples from the data distribution at each time step by estimating gradients and then performs predictions for multivariate time series data.

Despite the gradual application of the Denoising Diffusion Model in time series research, multivariate time series can involve multiple dimensions such as time, space, and attributes. Moreover, time series data often exhibit long-range dependencies. To address these challenges, we propose combining the Denoising Diffusion Model with the Transformer architecture to construct a generative model suitable for time series data. This integration aims to facilitate efficient TSAD, considering the complex nature of multivariate time series and their long-range dependencies.

We consider a multivariate time series $\mathcal{T}$={$x_{1}$,…,$x_{T}$}, where each element $x_{i}$ has $t$ timestamps, $t\in[1,T]$, and C dimensions, $c\in[1,C]$. Here, a univariate time series is a special case where $c=1$. The multivariate time series $\mathcal{T}$ can be represented as a matrix:

$\mathcal{T}=\begin{pmatrix}\mathbf{x}_{(1,1|i)}&\mathbf{x}_{(2,1|i)}&...&\mathbf{x}_{(T,1|i)}\\ \mathbf{x}_{(1,2|i)}&\mathbf{x}_{(2,2|i)}&...&\mathbf{x}_{(T,2|i)}\\ \vdots&\vdots&\ddots&\vdots\\ \mathbf{x}_{(1,C|i)}&\mathbf{x}_{(2,C|i)}&...&\mathbf{x}_{(T,C|i)}\end{pmatrix}$

The task of anomaly detection involves providing a multivariate time series $\mathcal{T}$ and training it to discover pattern features within the time series. Subsequently, in a test set where the patterns are the same as $\mathcal{T}$, a label $\hat{y}_{t}$ (0 for normal and 1 for anomaly) is assigned to each time point $x_{t}$. This process enables the detection of abnormal instances that do not conform to the established patterns.

The overall architecture of DDMT, as shown in Fig. 2, consists of three components: ADNM, DDT, and Anomaly Detection. The ADNM calculates the error between the reconstructed time series and the original time series and dynamically adjusts masks based on this error. DDT comprises the denoising diffusion model and the Transformer, where the Denoising Diffusion Model learns the mapping from the latent space to the signal space by removing noise that is sequentially added in a Markovian manner during the forward process, which corresponds to the noise added in the forward process [48], with the Transformer serving as the internal neural network within the diffusion model. The anomaly detection component assigns an anomaly score to each node in the time series based on the differences between the generated and original sequences, ultimately assessing the anomaly status of each node in the time series.

The traditional self-attention mechanism [28] computes the correlation between each position in a sequence by attending to itself and other positions, thereby capturing global contextual information. However, this characteristic also leads to the WIM issue in anomaly detection models [51]. In a scenario with full attention, each node is allowed to attend to itself, enabling it to reconstruct itself through simple self-replication during sequence reconstruction. Consequently, this approach struggles to uncover deep, intricate features within the time series data.

To address this challenge, we have devised a novel ADNM. In contrast to directly employing a self-attention mechanism, ADNM employs an Autoencoder to perform an initial reconstruction of the time series before applying the Transformer’s self-attention. Subsequently, it masks nodes with significant reconstruction errors and their neighboring nodes, effectively blocking self-information and information from closely adjacent neighbors. As a result, the model can focuses more on non-adjacent regions, making it more difficult to reconstruct anomalies during the process. The Fig. 3 illustrates the working process of ADNM.

For each node $i$, we introduce a mask scale $m_{i}\in N_{i}$, where $N_{i}$ represents the total number of neighboring nodes for node $i$, and $m_{i}$ is the number of neighboring nodes that need to be masked. Each node is associated with an reconstruction error, and based on this reconstruction error and the actual anomaly ratio, we can determine which points need to be masked. We then utilize the Pearson correlation coefficient between the masked node and its neighboring nodes to determine the size of the mask scale. If the correlation coefficient is greater than a certain threshold, the mask window can expand in both directions; otherwise, it stops. This approach allows mi¬ to adaptively adjust within the range $[0,N_{i}]$. Consequently, we obtain the mask matrix $M$. The mask matrix is dynamically generated based on the reconstruction errors of the time series and the required number of masked neighboring nodes $m_{i}$, which determines which nodes should be masked in the subsequent self-attention calculation process, as shown in Fig. 3.

ADNM adopts a simple autoencoder structure with the primary goal of disregarding information related to abnormal data patterns. While this autoencoder structure may not fully explore deep-level features within the time series, it excels at minimizing the impact of anomalous data. By selectively masking neighboring nodes, the ADNM elevates the model’s attention toward informative non-neighboring nodes while reducing the influence of the self-node on sequence reconstruction. This mechanism facilitates improved feature learning and enhances the model’s generalization ability.

Denoising Diffusion Model represents a class of generative models that have demonstrated state-of-the-art performance on a diverse range of domains, including images [24], audio [48], and video data [52], among others. Denoising Diffusion Model treats a stochastic process as a partial differential equation and solves this equation through numerical methods to obtain the probability distribution function of the stochastic process. As the noise gradually diffuses, the probability distribution function of the stochastic process evolves, reflecting the underlying dynamics. Denoising Diffusion Model consists of a forward diffusion process, denoted as $q(x_{t}|x_{t-1})$, and a learnable inverse process, denoted as $p_{\theta}(x_{t-1}|x_{t})$. The forward process gradually transforms data from the target distribution $p(x_{0})$ into a Gaussian distribution by adding noise, while the inverse process generates samples by transforming noise into $q(x_{0})$, as shown in Fig. 4.

The forward process of the Denoising Diffusion Model is a non-homogeneous Markov noise process. In this process, at each time step $t$, Gaussian noise with specific mean and variance is added to the data. This dynamic process can be described as follows:

where $q(x_{t}\mid x_{t-1})=\mathcal{N}\big{(}x_{t}\mid x_{t-1}\sqrt{1-\beta_{t}},\beta_{t}\mathbf{I}\big{)}$. The noise added at each step is determined by a variance table $\beta_{t}\in(0,1),t=1,...,T$. It can be defined as a small linear sequence. Based on the updates of $\beta$ as described in Ho et al. [22], we set $\beta_{1}=10^{-4}$ and linearly increase $\beta_{t}$ up to 0.02. By computing available $x_{t}=\sqrt{1-\beta_{t}}x_{t-1}+\sqrt{\beta_{t}}\epsilon_{t-1}$, for $\epsilon_{t-1}{}{\sim}N(0,\mathbf{I})$. The denoising process $p_{\theta}$ is learned by optimizing the model parameters $\theta$ and is defined as follows:

where $p_{\theta}(x_{t-1}\mid x_{t})=\mathcal{N}\big{(}x_{t-1}\mid\mu_{\theta}(x_{t},t),\tilde{\beta}_{t}\text{I}\big{)}$. As demonstrated by Ho et al. [22], it is possible to train the inverse process using the following objective function:

where $\mathcal{D}$ refers to the data distribution, and $\epsilon_{\theta}(x_{t},t)$ is parameterized using a neural network. For brevity, $L$ refers to the $\mathcal{L}_{\mathrm{simple}}$ in Ho et al. [22]. We have replaced the U-Net with Transformer as the neural network architecture inside the Denoising Diffusion Model to better handle deep local features and long-term dependencies within time series data. replacing the U-Net The original Denoising Diffusion Model used a U-Net as the internal neural network to learn the reverse process. In this paper, the convolutional and pooling layers of the U-Net in the Denoising Diffusion Model have been replaced with the encoder portion of the Transformer. This modification helps us capture both local and global relationships within the time series data. The core feature of the Transformer model is its self-attention mechanism, which allows the model to assign different weights to each element in the input sequence, taking into account information from other elements simultaneously. This design enables the model to handle long-range dependencies more effectively and benefits from the parallel computation. Furthermore, the multi-head attention mechanism allows the model to focus on different subspaces, enhancing its representational capacity. Feed-forward neural networks are used to process further intermediate representations, and residual connections and layer normalization techniques help mitigate the problem of gradient vanishing, thereby speeding up the training process. Fig. 5 illustrates the architecture of the Transformer encoder portion.

The implementation process of the DDT is illustrated in Algorithm 1 as follows:

To evaluate the performance of DDMT, we conducted extensive comparisons with 15 well-performing baseline models, including machine learning-based TSAD models: Isolation Forest [6], LOF [32], One-class Support Vector Machines (OC-SVM) [36], Deep Autoencoding Gaussian Mixture Model (DAGMM) [57], and Temporal Hierarchical One-Class (THOC) [56]. Deep learning-based anomaly detection models: Long Short-Term Memory-based Variational AutoEncoder (LSTM-VAE) [39], OmniAnomaly [53], UnSupervised Anomaly Detection (USAD) [58], InterFusion [59], Anomaly Transformer(AT) [7], Transformer-based Anomaly Detection model (TranAD) [60], TimesNet [61], Dual-Channel Feature Fusion (DCFF-MTAD) [62], Dilated Convolutional Transformer-based GAN (DCT-GAN) [63], and Multiscale wavElet Graph AE (MEGA) [64]. Table 2 presents a comprehensive performance comparison between our model and other baseline models. The best results are indicated in bold black, while the second-best results are represented with underlines.

Table 2 presents the accuracy, recall, and F1-score of all datasets on the DDMT and baseline models. From the table, it can be observed that in the machine learning-based anomaly detection models, THOC achieved superior results, attributed to its use of dilated recursive neural networks with skip connections to capture temporal dynamics at multiple scales and the incorporation of a self-supervised task in the time domain [56].On the other hand, the performance of DCT-GAN in the deep learning-based anomaly detection models is relatively poor, which may be primarily due to the utilization of PCA to transform multidimensional data into one-dimensional data for anomaly detection, which hinders the exploration of deep features and correlations among different dimensions in multidimensional data [63]. In addition to the DCT-GAN model, deep learning-based models generally outperform machine learning-based models in F1-score. Anomaly Transformer and TranAD achieved second-best results in five datasets twice and three times, respectively, while the MEGA model achieved the best result once.

Furthermore, we observed that DDMT demonstrated slightly better performance in F1-score compared to other baseline models, surpassing the best baseline results by 0.36%, 1.07%, 2.90%, and 0.15% in the MSL, SMAP, SWaT, and PSM datasets, respectively. For the SMD dataset, the MEGA model obtained the highest F1-score of 96.43.

To assess the differences between DDMT and other representative baseline models in both machine learning and deep learning-based anomaly detection, we selected 10 models for comparison. We used a t-test as the statistical method to calculate the p values for the differences in F1 scores between each pair of models. Following the convention, we set the significance level at 0.05. When the p value was less than 0.05, we consider the difference to be significant, indicating a notable distinction in F1 scores between different models. The results were then visualized in a heatmap (Fig. 6) to provide a more intuitive representation of the variations among the models.

In Fig. 6, the color intensity represents the correlation of the F1 scores, with lighter colors indicating a higher correlation. Additionally, we added corresponding p value markers to the heatmap to highlight the significance of the differences. Specifically, when the p value is less than 0.05, we used red asterisks to denote significant differences. From the heatmap, it can be observed that DDMT exhibits significant differences and holds statistical significance when compared to the LOF, DAGMM, THOC, LSTM-VAE, OmniAnomaly, and USAD models. These results demonstrate the distinctiveness of DDMT concerning F1 scores compared to the selected baseline models.

The experimental results above indicate that DDMT is feasible for anomaly detection in multivariate time series data. To investigate the effectiveness of each component of our method, we gradually removed the constituents of the model while keeping the settings unchanged. We observed the performance changes of the model on each dataset accordingly. Table 3 presents the experimental results comparison after eliminating various modules of the model.

Our Without DDT: In this study, we investigated the impact of the DDT module on the results by removing both the forward and backward processes of the Denoising Diffusion Model. From the table, it can be observed that after removing the DDT module, the model’s performance decreased. The DDT module contributed to an average F1-score improvement rate of 7.18% (87.19%→94.37%). The results demonstrate that the DDT module is highly effective and essential for the model’s overall performance.

Our Without ADNM: We further explored the impact of the ADNM module. In this study, we replaced the ADNM with the original self-attention module to investigate whether ADNM could enhance the anomaly detection performance of the model. From the table, it is evident that the ADNM module significantly improves the anomaly detection effectiveness. The proposed ADNM with the adaptive dynamic neighbor mask mechanism outperforms the original self-attention model with an average F1-score improvement of 3.99% (90.38%→94.37%). The results indicate that avoiding information leakage of each token and its neighbors during sequence reconstruction is crucial, and the masking mechanism in ADNM helps mitigate the identical shortcut problem during sequence reconstruction.

Baseline: To understand the effectiveness of the proposed method, we also compared it with a Transformer model as the baseline. In the Transformer model, the traditional self-attention mechanism was used to capture deep-level features of the time series by computing the attention map between all tokens for sequence reconstruction. The experimental results show that the average F1-score of the DDMT model was improved by 16.95% (77.42%→94.37%) compared to the Transformer model, demonstrating a significant superiority of the DDMT model over the Transformer model. In other words, the ADNM and DDT components in the DDMT model substantially enhance the anomaly detection performance.

In this section, we investigated the impact of various hyperparameters on the performance of the DDMT model. We primarily analyzed the effects of the window size on the results, as well as the influence of the mask scale in the ADNM module and the diffusion steps $\mathcal{S}$ in the DDT denoising process on the experimental outcomes.