An Interpretable Systematic Review of Machine Learning Models for Predictive Maintenance of Aircraft Engine

The four sets of data in the CMAPSS Dataset are designated as FD001, FD002, FD003, and FD004, and each is broken down into Train, Test, and RUL data. The initial step is to create labels for the training data. These labels are rul (remaining usable life), Label1, and Label2, much like in the predictive maintenance template.

When calculating the RUL value for each cycle, the highest value of the cycle is subtracted from each cycle before being added to the RUL value for each machine id. Equation 1 looks like this:

In order to manually estimate each machine’s maximum life and the point of deterioration (reduction in machine performance), each machine id has a distinct number of cycles. The algorithm is as follows:
if n in $RUL\geq$Maxlife : newRUL = Maxlife else newRUL = n
We adjust the amount of the data dimensions while processing data to become RNN, LSTM, or GRU input since we are aware that deep learning requires high-dimension input data, particularly for LSTM and GRU. We must perform a dimensional transformation to a higher level in order to allow the artificial neural network, specifically the LSTM, GRU that was constructed, to incorporate more feature levels. By estimating the size of the data sequence to be 50, the process of converting this dimension from 2 dimensions to 3 dimensions is carried out. As a result, a two-dimensional data size of (50, 26) will be created, and this value can be used to choose the batch size for the training procedure. The overall methodology is shown in Fig. 1
For machine learning models, feature-wise data will be delivered to the Random Forest, KNN, Naive Bayes, and Gradient Boosting networks following min-max normalization in order to train the model.

We have built both machine learning and deep learning architectures, and a comparison of their performance is also discussed, to forecast airplane engine failure within a predetermined number of cycles.

• Machine Learning Models: An ensemble machine learning technique called Random Forest (RF) is well-suited for high-dimensional datasets, noisy data, and missing values, and can provide feature importance rankings. An efficient machine learning approach called K-Nearest Neighbors (KNN) predicts the class or value of an input sample based on the dominant class. Naive Bayes Classifier is a simple powerful probabilistic machine learning algorithm used for classification tasks, where it predicts the class of an input sample based on Bayes’ theorem. Gradient Boosting Classifier is used for classification tasks, where it combines multiple weak decision trees to create a strong classifier.

Since it is well known that lime cannot be utilized for merged data, machine learning models are used in this case as well as to see if they perform well or worse than deep models. The hyperparameters are also modified to produce higher accuracy than in earlier studies.

• Deep Learning Models: The outputs of one step are utilized as inputs for the subsequent step in a Recurrent Neural Network (RNN). RNNs’ Hidden state, which retains certain sequence-related information, is its primary and most significant property. The hidden state for each time step is simultaneously determined by Bidirectional RNNs using data from previous and succeeding time steps. LSTM is a type of recurrent neural network (RNN) that is designed to handle the vanishing gradient problem, which can occur in traditional RNNs when training on long sequences of data. Bidirectional long-short term memory (bi-lstm), which enables any neural network to retain sequence information in both ways (backward, from future to past, or forward), is a word used to describe this (past to future). GRU is increasingly being used in deep learning models for time series analysis. In a GRU, the cell states and hidden states are merged with the input gate and forget gate of an LSTM as well as the update gate.

According to past research [5] [6], deep learning models work effectively. For this reason, in the research of predictive maintenance, we are trying to improve upon those models.

Our models are evaluated using accuracy, precision, recall, and F1-score. How often the model was overall correct may be determined by its accuracy. How well the model can forecast a certain category is determined by its precision. The recall reveals how frequently the model was able to recognize a certain category. The F1-score combines accuracy and recall from a classifier into a single statistic by computing the harmonic mean of these two metrics.

CMAPSS Dataset was run through the previously mentioned machine learning and deep learning algorithms. Our datasets were split into train and test sets containing 80% and 20%. We used the sklearn package to use the algorithms. We used a combination of Python 3.6.5, Keras 2.3.1 and Tensorflow 2.0.0. For the machine learning model’s performance evaluation, we used Lime also. Google Colaboratory was used to implement our codes. In RNN the relu activation function is used in the first layer and the sigmoid activation function is used in the last layer. We employed a batch size of 64, an epoch of 15 for the training parameters, and a learning rate with an Adam optimizer of 0.001. Here, the validation split is 0.1 and verbose=1. The early stopping method is also used. In LSTM relu activation function is used in the first layer and softmax is used in the last layer. The early stopping method is also used here. We employed a batch size of 64, an epoch of 15, a dropout of 0.5, and the learning rate is 0.001. Here the validation split is 20% and verbose=0. Theoretically, the learning process will be deeper and the reach of feature levels will be larger the deeper the network is built. In GRU we employed a batch size of 200, a dropout of 0.2, a learning rate with an Adam optimizer of 0.001, and epochs of 20 for the training parameters investigated in this work.

We import Random Forest from sklearn. In KNN we import KNeighborsClassifier with neighbors size=3 from sklearn. We also import Naive Bayes from sklearn. In Gradient Boosting, the estimators are 20, and the learning rate is 0.5.

Results achieved by our models can be seen in this section having 2 different tables along with the analysis of the results. Table 3 shows the results of deep learning models and table 4 shows the performance of machine learning models.

Fig. 2 shows the loss and accuracy curve of GRU. The loss and accuracy curve for a GRU model trained for 20 epochs shows metrics such as validation loss, validation accuracy, accuracy, and loss. These metrics offer information on the model’s capacity for generalizing what it has learned from the training set of data to new, untried data. Overfitting of the model may be revealed if the validation loss starts to rise while the training loss keeps falling. Due to repeated Jacobian multiplications during backpropagation, RNNs have gradient vanishing and explosion issues. RNNs are unable to simulate long-term sequential dependencies because the gradient signal used for training is inconsistent. Compared to RNN, Bi-RNN networks have better accuracy. Long short-term memory networks (LSTM) prevent vanishing gradients by updating cell states in an additive, non-multiplicative manner. However, it can suffer from gradient explosions. Bi-LSTM also performs well and achieved the second-highest accuracy because the input flows in both directions and it is able to take data from both sides.

In their iterative layers, feedback loops are a feature of all RNNs. As a result, information can be kept in ”memory” for a long time. It can be challenging to train a normal RNN to handle issues that call for learning long-term temporal dependencies, though. This is due to the loss function’s gradient’s exponential decrease over time (called the vanishing gradient problem). A particular kind of RNN called an LSTM network employs both unique and conventional units. The timing of when data is output, stored in memory, and forgotten is controlled by a sequence of gates. Long-term dependencies can be learned using this design. GRU employs a structure that is simpler than LSTM but is similar to it. Although it does not require individual memory cells and makes use of fewer gates, it nevertheless uses a number of gates to regulate the information flow.

Long short-term memory (LSTM) training outcomes outperform recurrent neural network (RNN) training results, according to a comparison of the two. First, a simple RNN is used with two dense layers to get the output with 91.40% accuracy shown in Table 3. However, compared to RNN, which has an accuracy of 94.85%, Bi-RNN networks have better accuracy. Strategies such as gradient clipping are often used to mitigate this problem. So a 2-layer LSTM is used and gets 96.42% accuracy. And after using Bi-LSTM, the maximum accuracy has been discovered, which is 97.14%. The GRU continues to outperform other algorithms according to the total comparative data. The accuracy that has been achieved in this experiment was 97.80%, which is significantly higher than that of any other models. The benefits of employing GRU to forecast the projected value of RUL are in terms of processing effectiveness and research expenditure expenses since the RUL approach can be processed even with a medium-spec PC, making machine learning and deep learning processing more affordable in the future. input flows in both directions and it is able to take data from both sides.
Data are merged during training by using deep learning models LSTM/ RNN/ GRU, which is a common occurrence for sequential modeling, and the results that have been found are shown earlier. But while applying machine learning models, feature-specific data are used and there is no data merging. The accuracy for Random Forest, KNN, Naive Bayes, and Gradient Boosting are 96.1%, 97.3%, 93.0%, and 95.6% respectively in Table 4. But the corresponding Precision, Recall, and F1-score values are lower than the other deep learning models. Compared to deep learning models, machine learning models performed comparably lower. The Lime technique has been used to machine learning models to explain why they didn’t perform well.

The method known as Local Interpretable Model-agnostic Explanations (LIME) is used frequently to explain predictions made by black-box machine learning models. To explain specific predictions made by black box machine learning models, local surrogate models, which are interpretable, are used. Text classifiers, tabular models, and image classifiers may all be explained using LIME. In order to demonstrate how each sample is categorized depending on the values of the features, we implemented machine learning models to our dataset, including Random Forest, KNN, Naive Bias, and Gradient Boosting, and then applied lime to it.
We chose sample number 6 and applied lime on it after we trained our dataset using the four machine-learning models in the order indicated above. In the test set, the maintenance classified as 0 is represented by the sixth row. It defines, there is no maintenance required, and it was correctly classified by all models. Fig. 3 shows that the Random Forest model is 99% confident that it is under the label of 0 classification. The values of s11, s9, s12, s14, and s7 increase maintenance chance to be classified as 0.

According to Fig. 4, the KNN classifier is absolutely confident that it is classed as 0, and the values of id and s9 increase the likelihood that it will be given that classification.

Fig. 5 shows that the Naive Bayes model is 100% guaranteed that the given sample belongs in the category of 0. The likelihood of it being classed as 0 grows with the values of s14, s9, and s7. Once more, Fig. 6 shows that the Gradient Boosting model is 99 percent confident that the provided sample will be categorized as 0, which is completely correct. This is due to the higher values of s7, s14, s11, s8, s20, and s9.

Following that, we chose sample number 4, whose real label reads 1, indicating that maintenance is required. In this instance, the Random Forest model has a 30% confidence level that is categorized as 1. S7 values reduce the chance of correctly classifying it. The outcome is shown in Fig. 7. Again, based on Fig. 8, we can see that the KNN classifier misclassified it for the values of s9, s3, and s21.

On the other hand, in Fig. 9 the Naive Bayes model properly identified the sample as 1 in this situation. The sensors’ s12, s4, s20, s2, s21, s11, s8, s15, and s13 results are what allow it to be accurately classified. After applying Lime to the Gradient Boosting model for Sample 4 in Fig. 10, the model misclassified the data. Due to the higher values of s14 and s7 in class 0, the confidence level is only 41%.

It is clear from the lime implementation for sample 6 that the value of s4 is what caused the sample’s misclassification. For sample 4, the value of s7 increases the probability of its misclassification.
Due to the fact that the final processed data for this time series dataset are merged, we were unable to use lime on LSTM, GRU, or RNN. In order to understand why machine learning models didn’t perform well than the deep learning models, we applied lime to the models. Although it is known that Lime is only relevant for text and feature-based data, this is not how our final processed data are. An example of the outcome from the processed data is shown in Table 5.

A comparison of our proposed models with some Table 6 shows a comparison of our suggested approaches with some previously done work. The table clearly shows that the models we’ve provided are more accurate than those in past studies, and also further illustrates how the use of lime distinguishes our work from other studies in the earlier part.