Unsupervised Ensemble Methods for Anomaly Detection in PLC-based Process Control

Sun et al. [33] employed an unsupervised ensemble outliers detection framework based on IF algorithms to detect insider threats in an industry. They modified the isolation trees in the IF to support categorical data. Although they achieved significant detection recall values, a drawback of the approach in [33] is that the ensemble method consisted only of isolation trees. As a result, the isolation trees made the same assumptions about the dataset.

Haider et al. [34] proposed an ensemble method based on OCSVM and IF on data clusters. They employed human domain experts to identify false-positive results, which were used to update the ensemble algorithm progressively. They executed a k-means algorithm for class decomposition on top of the base detectors. Finally, they tuned the ensemble algorithm by oversampling the false-positive results to create an accurate decision boundary. Although this algorithm presents an interesting idea for unsupervised anomaly detection, it requires human intervention to achieve optimal performance.

In [35], the authors proposed an ensemble learning framework that combines an unsupervised learning method with a graph-based anomaly detection technique. The base detectors consisted of OCSVM and minimum description length (MDL)[36]. They also proposed a detection model solely based on multiple OCSVM and a streaming data mining approach to account for concept drift. Reported performance was was, suggesting that choice and number of base detectors in an ensemble algorithm are crucial to obtain the best results.

Yuan et al. [37] proposed an ensemble of deep learning algorithms consisting of long short-term memory and a convolutional neural network to identify anomalies in multiple scenarios. They achieved an area under the curve of $94\%$. Although their method worked well, their approach requires significant computational resources like graphical processing units during training and testing.

Inoue et al. utilized unsupervised ML algorithms for anomaly detection in water treatment systems [38]. They compared a deep neural network consisting of feedforward layers with multiple inputs and outputs with a One-class Support Vector Machine (OCSVM). The authors claimed that the deep neural network model produced fewer false positives than the OCSVM, although the OCSVM could detect more anomalies. The authors achieved recall values of less than 0.7 for both deep neural network and OCSVM.

In [39], the authors employed a fully connected neural network and an autoencoder to detect anomalies in IoT computer network traffic. Their results showed a higher detection rate and lowered false positive rate when compared with eight other modern anomaly detection techniques. Potluri et al. [40] proposed using artificial neural networks for identifying false data injection attacks in ICS. The classification report obtained in [40] shows a promising detection accuracy with artificial neural networks.

Ahmed et al. [41] presented an unsupervised learning IF approach to detect hidden data integrity assault on a smart grid communication network. They reported an average accuracy of 93%,using simulated experimental data. While the simulated data may not depict ICS accurately, The work in [41] suggests that IF has the potential for high performance in anomaly detection.

Although both supervised and unsupervised ML techniques have been applied in PLC anomaly detection [22, 24, 42], it is usually difficult to rely on a supervised learning approach as real-world ICS contain numerous sensor data that are tedious to label. An unsupervised ML technique called OCSVM was employed to detect anomalies in PLCs successfully [22]. In [22], the authors employed OCSVM, OCNN, and IF for PLC anomaly detection. Their experiment simulated a traffic light control system using a PLC. They captured relevant PLC memory addresses into a log file for real-time data recording of system operations. The captured data was normalized and used to train and evaluate their models. Aboah et al. concluded that IF outperforms OCNN and OCSVM on the TLIGHT system dataset.

While OCSVM, OCNN, and IF have been used as effective unsupervised techniques for anomaly detection, OCSVM performance is unsatisfactory on complex, high-dimensional datasets [43, 44, 45]. Although OCNN has been used for anomaly detection in a complex dataset [43, 22], the work in [22] shows that OCNN performs at par with OCSVM on several test cases of their TLIGHT dataset because OCNN and OCSVM are formulated from a similar optimization problem. The work in [22] also shows that some anomalous data points are detected by either OCSVM or OCNN, but IF fails to detect them. Therefore, this work hypothesizes that some base anomaly detectors will perform well on a particular subset of a dataset, whereas other base detectors will perform better on other subsets. This work proposes five unsupervised ensemble techniques based on OCSVM, OCNN, and IF to improve anomaly detection performance.

Base detectors refer to the individual components of an ensemble technique. The base detectors are strategically combined to make ensemble techniques’ output robust. The base detectors could consist of different kinds of learning algorithms of any number depending on the problem and adopted strategy [49, 50]. The base detectors adopted for all ensemble methods in this work are OCSVM, OCNN, and IF. OCSVM is an unsupervised anomaly detection technique that learns a decision function for separating the normal class from anomalies in a dataset. OCSVM has been employed for ICS anomaly detection in [51, 52, 22]. OCNN is an unsupervised machine learning algorithm for anomaly detection formulated on the foundation of OCSVM [22]. OCNN has been employed for anomaly detection in [43], although its application in ICS is very limited [24, 22]. IF is also an unsupervised machine learning technique that builds an ensemble of binary trees for a given dataset for anomaly detection [53, 54]. IF has been utilized to successfully detect anomalies in ICS [23, 55, 22]. The ensemble methods proposed in this work use different voting strategies for combining OCSVM, OCNN, and IF outputs.

Ensemble learning is a technique for combining the outputs of multiple algorithms to create an improved output. The three different anomaly detection techniques mentioned in the subsection above make independent assumptions about the nature of the dataset, which may not be the case for every dataset. Ensemble learning techniques have been successfully employed in various data mining and supervised machine learning applications such as classification and recommender systems [56, 57, 58]. However, unsupervised ensemble analysis is a recent field in anomaly detection, especially in ICS, with little work on combining anomaly detection algorithms [25]. This work seeks to extend the knowledge of ensemble-based detection approaches for anomaly detection in ICS. This section discusses five proposed ensemble-based anomaly detection techniques that combine the outputs of the three base detectors, namely, OCSVM, OCNN, and IF.

Ensemble learning is generally challenging in the context of anomaly detection because of the unsupervised nature or labeled data unavailability [25]. Another challenge is the coherent combination of base detectors’ decision scores, as base detectors may have different objective and scoring functions. Figure 2 shows a general representation of the proposed ensemble approach. Algorithm 1 and Algorithm 2 summarize the general approach to training and testing the proposed ensemble methods in this work. The major changes in each method relate to combining the scores of the base detectors and making final predictions, as described in the following subsections.

The majority-vote ensemble method performed well on test set 1 by having precision, recall, and F1-score of $90\%$, $91\%$, and $90\%$ respectively. The algorithm achieved a recall and F1-score of $88\%$ and $84\%$ respectively on test set 2, which are similar to the recall and F1-score values of test set 3. However, the algorithm’s high recall and F1-score rate on test set 2 resulted from its high anomaly detection rate on the normal data points at the expense of high false positive rate. The algorithm finds it challenging to detect anomalies involving anomalous scenarios 1 and 3. Performance of the algorithm decreases on test sets 4 and 5 as the recall values decreased to $82\%$ and $71\%$ respectively. Figure 4 shows the majority-vote ensemble method results of the normalized TP, TN, FP, and FN values on test sets 1 and 3.

Because of the algorithm’s binary prediction outcome, it is impossible to visualize and assess prediction confidence levels, unlike other ensemble approaches proposed in this work. Figure 4 shows that although the algorithm’s TN predictions are less than $50\%$, the levels of FN predictions are low at values below $10\%$. However, the algorithm has high levels of FP in the range above $50\%$ for test sets 1 and 2. The visualization of the algorithm performance on test sets 3-5 is similar to Figure 4.

Maximum-score ensemble method performance on test sets 1 and 2 are high with F1-scores of $90\%$ and $95\%$ respectively. This method has the same precision, and recall values as the majority-vote method on test set 1. However, this method significantly performs well on test set 2 as compared to the majority-vote method by having precision and recall values of $96\%$ and $94\%$ respectively. The algorithm’s performance on test sets 3 and 5 is similar to majority-vote method. F1-scores for test sets 3 and 5 are $85\%$ and $71\%$, respectively. This method has its worst performance on test set 4, with recall and F1-scores of $66\%$ and $58\%$ respectively. In effect, the maximum-score ensemble method has low detection performance on TLIGHT system anomalies involving timing bits manipulation. Figure 5 shows the maximum-score ensemble method results of the normalized TP, TN, FP, and FN values on test sets 1-4.

Maximum-score ensemble method has TP values above $60\%$ in test sets 1-4 and high TP scores confidence in test sets 1-3. The method detected all anomalies in test set 2 by having a TN value of $100\%$, which explains its high performance on test set 2. Figure 5 shows that the method misclassified over $95\%$ of test set 4 anomalies as normal instances, which resulted in high FP and low TN values. The visualization of the algorithm performance on test set 5 is similar to test set 4, and therefore it is not shown here.

Soft voting ensemble method achieved high performance on test set 1 with a precision, recall, and F1-score of $89\%$, $90\%$, and $89\%$ respectively. The method’s performance on test sets 1 and 2 are similar, with F1-scores of $83\%$ and $84\%$ respectively. However, the method has low performance on test sets 4 and 5, with F1-scores in both cases below $60\%$. The method’s low performance on test sets 4 and 5 signifies its inability to detect timing bits anomalies. Figure 6 shows the soft voting ensemble method results of the normalized TP, TN, FP, and FN values on test sets 1, 2, 3, and 5.

Soft voting ensemble method has a TP prediction confidence over $50\%$ on test sets 1 and 2. Again, the method correctly predicted the TLIGHT system’s normal behavior in all test cases by having TP predictions over $60\%$. However, the method misclassified all anomalies in test set 2 as normal instances leading to a high FP rate of $100\%$, which explains the low performance of the algorithm on test set 2. The method’s confidence levels of TP and FP predictions on test set 5 are below $50\%$. Also, over $80\%$ of anomalies in test set 5 are misclassified as normal instances leading to high FP rate. The visualization of the algorithm performance on test set 4 is similar to test set 5, and therefore it is not shown here.

The weighted voting ensemble method uses an ordinary least squares regression model for assigning weights to the base detectors. The weighted voting with ordinary least squares regression model WV-OLS has a similar performance to soft voting ensemble method on all datasets. The method achieved its highest performance on test set 1 with precision, recall, and F1-score of $89\%$. The method’s performance on test set 2 and 3 are similar, with F1-scores of $83\%$ and $84\%$ respectively. This method has its lowest performance on test sets 4 and 5, with F1-scores below $60\%$. The method’s low performance on test sets 4 and 5 signifies its inability to detect anomalies in timing bits. Figure 7 shows the WV-OLS ensemble method results of the normalized TP, TN, FP, and FN values on test sets 3 and 4.

The WV-OLS method correctly predicted the TLIGHT system’s normal behavior in all test sets with a TP score over 60%. The method’s confidence of TP and FP predictions on test set 3 is over $50\%$, whereas the confidence of the TP and FP predictions on test set 4 is below $50\%$. The visualization of the algorithm performance on test sets 1 and 2 are similar to test set 3, whereas the visualization of the algorithm performance on test sets 5 is similar to test set 4, and therefore they are not shown here. The method misclassified over $80\%$ of anomalies as normal in test set 4 leading to high FP. The method’s high FP value on test set 4 explains its low performance on test set 4.

WV-OLS ensemble method performance is lower than maximum score ensemble method. The WV-OLS ensemble method’s low performance results from the similarities between OCSVM and OCNN objective functions which makes their errors correlated. OCNN is formulated based on the foundation of OCSVM optimization problem [22]; hence WV-OLS method’s assigned weights are biased towards OCSVM and OCNN. Figure 8 shows the decision scores of WV-OLS base dectectors on 20 anomalous samples of test set 4. In each of the 20 anomalous samples, a negative score is desired to signify correct prediction. However, the OCSVM base detector misclassifies the anomalous test samples by producing positive scores for all 20 anomalous test samples. The OCNN base detector misclassifies more than 50% of the anomalous test samples. On the contrary, the IF base detector correctly detects more than 60% of the anomalous test samples. Similar misclassifications of anomalous test samples by the OCSVM and OCNN base detectors were observed across all test sets, explaining the low performance of the WV-OLS approach to weighted voting.

In an attempt to control the WV-OLS method’s high bias, the ordinary least squares linear regression model for assigning weights in Algorithm 3 can be replaced by a ridge regression model [64] or K-nearest neighbor model [65]. The resulting weighted voting approaches using these learning models are denoted WV-Ridge and WV-KNN, respectively. The proposed weighted voting method using different linear regression models are compared with the weighted voting approach found in [25]. WV-[25] uses RMSE for assigning weights to the base detectors. Table III shows accuracy, precision, recall, and F1-score for the four weighted voting approaches. The WV-OLS approach shows no significant improvement over WV-Ridge and weighted voting method in previous work [25]. WV-OLS, WV-Ridge, and WV-[25] all use linear models for assigning weights to base detectors’ output; therefore, the same fundamental limitation associated with the bias of OCSVM and OCNN base detectors holds. However, WV-KNN outperforms WV-[25], WV-OLS, and WV-Ridge on test sets 1, 2, and 5 across all performance metrics. WV-KNN achieved over 7% improvement in accuracy, precision, recall, and F1-score on test set 2 as compared to WV-[25], WV-OLS, and WV-Ridge. WV-KNN achieved high performance because it relies on distance in the feature space instead of making a linear assumption about the dataset.

Figure 9 shows WV-KNN ensemble method results of the normalized TP, TN, FP, and FN values on test sets 2 and 5. WV-KNN correctly detects all anomalies in test set set with over 25% prediction confidence. Also, WV-KNN correctly detects over 70% of normal data points with 100% prediction confidence leading to high TP value in test set 2. WV-KNN’s high anomaly detection on test set 2 explains why it achieves over 90% accuracy, precision, recall, and F1-score. WV-[25], WV-OLS, and WV-Ridge are unable to detect more than 20% of anomalies in test set 5. However, WV-KNN detects over 50% of anomalies in test set 5 which explains why WV-KNN has over 10% improved accuracy, precison, recall, and F1-score as compared to WV-[25], WV-OLS, and WV-Ridge. The visualization of WV-KNN performance on test 1 and 3 are similar to WV-OLS visualization on test set 3 in Figure 7. The visualization of WV-KNN performance on test set 4 is similar to the WV-OLS visualization on test set 4 in Figure 7, and therefore it is not shown here. Lastly, WV-[25] and WV-Ridge performance visualizations across all test sets are similar to those of WV-OLS in Figure 7, and therefore they are not shown here.

Stacking based ensemble method uses an IF meta-detector to consider the results of the base detectors. This method has excellent performance on all test sets. It achieved its highest performance on test sets 2 and 4 by having the same precision, recall, and F1-score of $97\%$, $96\%$, and $96\%$, respectively. Unlike other ensemble methods described in previous sections, the stacking-based ensemble has high performance on test sets 4 and 5 with recall values of $96\%$ and $92\%$ respectively. The method’s performance shows its ability to detect TLIGHT signals anomalies and its robustness in detecting timing bits anomalies. The method had its worst performance on test set 3 with a recall and F1-score of $87\%$ and $86\%$ respectively. The high performance of the stacking-based ensemble method is due to the IF meta-detector’s ability to extract significant information from the base detectors in an unsupervised learning manner. Figure 10 shows the stacking-based ensemble method results of the normalized TP, TN, FP, and FN values on test sets 1, 2, 4, and 5.

This method correctly predicted all anomalies in test 2 with high prediction confidence and TN value of $100\%$, which explains why it attained an F1-score of $96\%$. This method correctly predicted TLIGHT system normal behavior with TP value over $70\%$ at $100\%$ prediction confidence in all test sets. Again, in all test sets, over $70\%$ of TN predictions have prediction confidence over $50\%$. A closer assessment of Figure 10 confirms that not only is the stacking-based method excellent at detecting anomalies, but in all test cases, the method is confident about its decisions. The visualization of the algorithm performance on test sets 3 is similar to test set 1, and therefore it is not shown here.

Figure 11 shows the average performance of the ensemble methods with their respective standard deviations on the five test sets. Stacking-based ensemble and IF [22] have the least standard deviations $<$ 0.07 on all performance metrics across all test sets. WV [26], WV-OLS, WV-Ridge, and soft-voting methods have the high standard deviations $>$ 0.1 as a result of their poor performance on test sets 4 and 5. Table IV provides a performance summary of all ensemble methods explored in this work. It is evident from Table IV that stacking-based ensemble method has superior performance over all test sets.

An anomaly detection performance comparison between individual unsupervised machine learning detectors using the same dataset is presented in [22]. The unsupervised ensemble methods proposed here strategically aggregates the decision scores of the three methods examined in [22]. This section compares the average performance of the proposed unsupervised ensemble techniques with the reported results of the highest performing algorithm in [22] across all test sets.

First, the comparison is performed across test sets 1-3, which consisted of TLIGHT system signal anomalies, excluding timing bits anomalies. Figure 12 shows the average test sets 1–3 performance proposed ensemble methods in this work and the reported results of IF in [22]. Soft-voting, WV-OLS, WV-Ridge and WV-[25] methods have similar average accuracy, precision, recall, and F1-score. Majority vote and WV-KNN have similar average accuracy, precision, recall, and F1-score. The stacking ensemble method and IF [22] algorithm achieve the same average accuracy, precision, recall, and F1-score of $92\%$. Therefore, the stacking approach and IF [22] have similar detection performance on TLIGHT system anomalies, excluding timing bit anomalies. Stacking ensemble and IF [22] have superior performance to all other ensemble techniques.

Test sets 4–5 consist of timing bits anomalies unique to [22] and this work. Figure 13 shows the average performance comparison between the ensemble methods presented in this work and the reported results of IF algorithm in [22] on test sets 4–5. Soft-voting maximum-score, WV-[25], WV-OLS, and WV-Ridge methods have relatively similar average accuracy, and precision on test sets 4–5 which were inferior to WV-KNN and maximum vote ensemble methods. Stacking-based ensemble achieves the same average accuracy, precision, recall, and F1-score of $92\%$, whereas IF [22] achieves the same average accuracy, precision, recall, and F1-score of $82\%$. Overall, stacking-based ensemble method achieves outstanding performance on all evaluation metrics on test sets 4–5. The stacking-based ensemble method’s high performance on all test sets compared to previous work [22] upholds the motivation of this work: effective unsupervised ensembling of anomaly detection algorithms could result in a robust detection model capable of identifying anomalies in arbitrary datasets.

Stacking-based ensemble methods for classification purposes using supervised machine learning meta-detectors were examined in [70, 71, 72, 73, 74, 75]. However, a direct comparison between the proposed stacking-based ensemble method in this work and the works in [70, 71, 72, 73, 74, 75] is unrealistic because of the unsupervised ensemble approach adopted in this work for anomaly detection purposes.

In order to validate the results for the best performing algorithm, stacking-based ensemble method is compared with previous work trained and evaluated on the same TLIGHT system dataset by using statistical hypothesis test. Statistical hypothesis testing is employed to validate and ensure fair comparison between the stacking ensemble method and previous work. Table III compares the average performance of the best algorithm in [22] with the proposed stacking algorithm in this work. Statistical evidence about the best-performing technique is conducted using one-way analysis of variance (ANOVA) in order to determine if there is any significant difference between the average performance of the method in this work and that of previous work [22]. F1-score is selected as an evaluation metric in the hypothesis test because it measures the balance between precision and recall. The following assumptions are made about the ICS dataset

• •

  data points in each test sample identically distributed and independent; and

• •

  data points in each test sample are normally distributed.

Additionally, the hypotheses for the statistical test are

• •

  null hypothesis $(H_{0})$: The mean F1-scores of all anomaly detection models are equal; and

• •

  alternate hypothesis $(H_{a})$: One or more of the mean F1-score of the anomaly detection models are unequal.

Anomaly detection performance of the stacking-based ensemble in this work and the IF in [22] are evaluated on twenty different test samples of the exact sizes as test sets 1-5. Furthermore, each model’s F1-score is computed on the twenty different samples of each test set. Results from one-way ANOVA test shows an $F\ value$ of 42.54, and a p-value $<$ 0.001. Since the p-value is below 0.05, one-way ANOVA provides significant evidence to reject the null hypothesis. Rejecting the null hypothesis means there is a significant difference between IF [22] and stacking-based ensemble with $95\%$ confidence level. Because performance comparison is between two algorithms, one-way ANOVA results are sufficient to conclude that the stacking-based ensemble method outperforms IF [22].

The unsupervised ensemble methods presented in this work can be implemented in real-world PLC-based ICS infrastructure. One way is to adapt the experimental approach in this work, or the approach in [22] by serializing the trained ensemble models onto a separate computer with a data pipeline to the HMI historian and PLC memory addresses to receive data and perform real-time anomaly detection. Alternatively, the proposed methods can be implemented on both legacy and embedded PLCs by adapting a similar process outlined in [22]. That is, the trained ensemble models can be compiled to C code using open-source compilers [76, 77]. The resulting C code should be readily portable to dedicated embedded or general-purpose processors [78] for real-time anomaly detection.

Although this work proposes robust unsupervised ensemble techniques for anomaly detection in ICS, some limitations are associated with the proposed methods. First, ensembling OCSVM, OCNN, and IF base detectors make the proposed methods in this work expensive in terms of time and storage. That large amount of memory is required to store outputs of the base detectors and their ensemble output during training and testing. Moreover, the high time complexities of the proposed methods result from the effective summation of the base models’ time complexities and the time required for the voting function in each method to compute the final anomaly score. Despite the stacking-based method’s superior performance in all test cases, its IF meta-detector still requires hyperparameter tuning to achieve such robust performance. Lastly, the proposed WV-OLS has a fundamental limitation of high bias in situations where two or more base detectors have similar objective functions. The high bias effect is reduced by using a KNN learning algorithm for assigning the weights. In practice, a learning algorithm for the weighted voting method should be selected based on its ability to handle non-linear features and has low bias.