LSTM-based Anomaly Detection for Non-linear Dynamical System

Anomaly detection is a proactive method that raises alerts when the system is still in the normal state but progressing to an anomaly state [10]. To prevent the anomaly or reduce the damage, anomaly detection is of great significance in many fields, such as geology, meteorology, and medicine.

For anomaly detection, the Markov model-based approaches are widely used. A Markov model is a stochastic model used to model randomly changing systems. It can be used to predict the state in the future using previous information. It is able to establish the transition probabilities relationship between states. In Markov chain model, the pattern of different metric values can be recognized and the next state of these values is predicted by the model. For example, Gu et al. presented a stream-based mining algorithm by combining Markov models and Bayesian classification methods for online anomaly prediction. Their anomaly prediction scheme can raise alerts for impending system anomalies in advance and suggests possible anomaly causes [11]. Sendi et al. proposed a framework to predict multi-step attacks before they pose a serious security risk. They used Hidden Markov Model (HMM) to extract the interactions between attackers and networks [12]. Paulo et al. applied a Markov chains-based method to characterize the stochasticity of droughts and predict the transition probability from one class of severity to another up to 3 months ahead [13]. In [14], Zhou et al. employed the Evidential Markov chain for the anomaly prediction of PlanetLab. A Belief Markov chain is proposed to extend the Evidential Markov chain and cope with noisy data stream.

Also, there are many anomaly detection approaches based on regression methods. In these approaches, the detection problems are converted into normal regression problems, then machine learning-based regression algorithms and models can be applied to anomaly detection. For instance, Hong et.al proposed a new anomaly detection approach based on principal component analysis (PCA), information entropy theory and support vector regression (SVR). It can be used in credit card fraud detection as well as intrusion detection in cyber-security [15]. In [16], Huang et al. presented a Recurrent Neural Networks (RNN) model for the anomaly prediction of component-based enterprise systems. Their RNN-based method has shown high prediction accuracy and time efficiency for large-scale systems.

In recent years, neural network-based deep learning approaches are widely applied to the prediction problems [10]. Among them, Recurrent Neural Network (RNN) and its advanced variants have shown a higher performance for prediction tasks than traditional methods. RNN is a type of artificial neural networks. It can capture the feature of input time series data by remembering its historical information. LSTM Network is a special RNN. LSTM is proved to outperform many other types of RNN in modeling sequential data and widely used in prediction tasks [17].

RNN is suitable for learning patterns, relationships and interconnections hidden in time series as well as modeling the temporal sequences. In [18], Zio et al. employed Infinite Impulse Response Locally Recurrent Neural Networks (IIR-LRNN) to forecast failures and make reliability prediction for systems’ components. It is innovative to use such dynamic modeling technique in reliability prediction tasks.

Especially, LSTM, which was proposed by Hochreiter et al. in 1997 [19], has emerged to be an effective and scalable type of RNNs for several learning problems related to sequential data [20]. By utilizing multiplicative gates that enforce constant error flow through the internal states of cells, LSTMs overcome the vanishing gradient problem in original RNNs [21].

Due to their superior ability for processing time-series data, LSTMs are widely applied to prediction tasks. In [22], an LSTM-based deep learning method is explored for travel time prediction on the dataset provided by Highways England. They have achieved high prediction accuracy for 1-step ahead travel time prediction error. In [23], an LSTM-based spatio-temporal learning framework is proposed for land cover prediction. The authors design a dual-memory structure to capture both long-term and short-term patterns in temporal sequences. Based on LSTM, the authors in [24] proposed an improved model to learn tweet representation from weakly-labeled data and make tweet classification with higher accuracy. In [17], LSTM network is first used to predict the performance of web servers as the URL requests are always sequential data. The logs of Nginx web servers are analyzed before the performance of web server is predicted.

Besides, Chen et al. utilized LSTM for the prediction task for China stock [25]. Altche et al. addressed a highway trajectory prediction approach based on LSTM network [26]. Qu et al. applied an approach based on PCA and LSTM to the field of wind power prediction [27].

LSTM layer receives a time sequence of the same length $N$ as input and outputs them to dropout layer to prevent overfitting. To collect samples for training, there are two different modes according to the requirement of prediction demand. As is shown in Fig. 2, there are two types of training modes according to the requirement of prediction demand. $T$ refers to the length of the input time series and $t$ means the prediction of the $t^{th}$ point after the current point.

In mode $a$, only the next point of current input time sequence is predicted in each prediction, as is illustrated in Fig. 2(a). A sample used for training is obtained by current state window. It contains the time sequence of fixed length and the value of the next timestep. By sliding over the long time series used for training, all samples are gathered into the training set. Mode $b$, as shown in Fig. 2(b), meets the requirement of muti-timestep prediction. For the sliding window in mode $b$, it equals to $T+t$ which contains the time sequence and a time interval between current value and the value to predict. In this mode, a training sample consists of time sequence of fixed length and the value of the $t^{th}$ timestep after the current value.

When all training samples are gathered into the training set, the training process begins. It randomly selects from the training set to remove correlations in the sequence and smooths the changes in data distribution.

Irrespective of different training modes, the Root Mean Square Error (RMSE) is utlized to evaluate the prediction performance of the proposed method,

Another important module in the architecture of LSTM-based anomaly detection method is the following anomaly detection. An algorithm, named Local Average with Adaptive Parameters (LAAP), is put forward. This algorithm uses adaptive parameters to compute local numerical features including average, standard deviation and slope. The threshold for determining anomaly depends on those features of predicted timestep.

As the sliding window moves forward, the predicted time series can be obtained. The length of the predicted future timesteps is assumed to be $d$. The predicted time series can be denoted as $D:s_{1},s_{2},\dots,s_{d}$. The window length and parameter adaptive rate are set to be $W$ and $\alpha$ respectively. The value of $W$ is usually determined by the pattern of most anomaly. The parameter adaptive rate can be adjusted according to how sensitive the detection system should be. It ranges from 0 to 1. If the parameter adaptive rate is relatively high, the anomaly detection system is more sensitive to the upcoming anomaly. Otherwise, the system may reduce the probability for predicting an upcoming anomaly.

In each window length of $W$, the local average ${\mu}_{i}$ is computed as

One kind of anomalies appears to be a local maximum while another kind of anomalies appears to be a local minimum. According to these two different situations, the result of anomaly detection $y_{i}$ is obtained and output by algorithm LAAP. To summarize the above procedure of our proposed method, the pseudocode of LAAP is shown in Algorithm 1.

To test and verify the LSTM-based multi-step prediction and the following anomaly detection, the data is generated by the temporal evolution of an incompressible Newtonian fluid in the plane Poiseuille (channel) geometry where the flow is driven by a constant volumetric flux, $Q$, at a Reynolds number of $R_{e}=1800$. The $x$, $y$, and $z$ coordinates are aligned in the streamwise, wall-normal, and spanwise directions, respectively. Periodic boundary conditions are imposed in the $x$ and $z$ direction with fundamental periods of $L_{x}$ and $L_{z}$. No-slip boundary conditions are imposed at the top and bottom walls $y=\pm h$, where $h=L_{y}/2$ is the half-channel height. Such a boundary condition necessitates that streamwise velocity is zero at both walls. In this study, the so-called minimal flow unit methodology is used with a domain size of $L_{x}\times L_{y}\times L_{z}=2\pi\times 2\times\pi$ [28]. A numerical grid system is generated on $N_{x}\times N_{y}\times N_{z}$ meshes, where Fourier-Chebyshev-Fourier spectral spatial discretization is applied to all variables. With a mesh convergence study, the resolution used is $\left(N_{x},N_{y},N_{z}\right)=(48,81,48)$. The time step used for forward integration of the system is $\Delta t=0.02$. Given these temporal and spatial resolutions, the corresponding errors are $O\left(10^{-6}\right)$ and $O\left(10^{-4}\right)$, respectively. For more details, the reader is referred to [29, 30, 31].

For this study, the one-dimensional data used for our experiments is the time series of the wall shear stress $\tau=\mu\partial u/\partial y$, where $\mu$ is the dynamic viscosity of the fluid and $u$ is the streamwise velocity. The wall shear stress is a measure of the resistance of fluid experiences and is a qualitative measure of the state of nonlinear dynamical system.

A segment of the raw data is visualized in Fig. 3. The data fluctuates around a certain value and has a definite boundary in normal state. According to the value, time series can be divided into four zones: those below 90% of the mean value (zone 1), those between 90% of the mean value and the mean value (zone 2), those between the mean value and 110% of the mean value (zone 3), and those over 110% of the mean value (zone 4). Data in zone 4 should be recognized because there is a great probability of a turbulence. Usually, there is a high transition probability from zone 1 to zone 4, and anomalies are more likely to appear in zone 1 and zone 4. Hence, the main objective of the anomaly detection is to predict the data in zone 1 and zone 4. Accurate prediction of these anomalies can effectively prevent some potential accidents from happening.

In this part, the experimental results are given and analyzed from two aspects: multi-step prediction and anomaly detection.