Neuroscience-Inspired Algorithms for the Predictive Maintenance of Manufacturing Systems

We focus on the specific task of PM on roller-element bearings due to their broad application and utility in manufacturing. We also evaluate Additive Manufacturing (AM) as it is a modern technique that presents unique challenges due to the dynamics of 3D printers. Here, we briefly discuss works related to PM for roller bearings and additive manufacturing.

Many PM methods use statistical models due to their simplicity and explainability. These approaches rely on extracted time and frequency domain features. For example, the energy entropy mean and root mean squared (RMS) values of wavelets were used to diagnose ball bearing faults in [5]. In another example, the spectral kurtosis (SK) of vibration and current signals was used to detect and classify the surface roughness of ball bearings in [6]. Using a particle filter method, Zhang et al. performed fault detection on bearings similar to those found in helicopter oil cooler fans [7].

In addition to statistical methods, ML techniques have been applied to a wide array of industrial prognosis tasks. One such method: AutoRegressive Integrated Moving Average (ARIMA), is one of the most popular techniques for time-series forecasting and was used to predict failures and identify quality defects in a slitting machine in [8]. In another approach, Tobon-Mejia et al. used Mixture of Gaussians HMMs and Wavelet Packet Decomposition to estimate the Remaining Useful Life (RUL) of roller-element bearings [9].

DL methods such as Long Short-Term Memory (LSTM) Networks and Convolutional Neural Networks (CNNs) have also been used extensively for PM. In one example, Feng et al. used an LSTM for detecting anomalies in industrial control systems [10]. Additionally, an RNN-LSTM was used to perform PM on an air booster compressor motor used in oil and gas equipment in [11].

Due to the increased complexity and relatively late adoption of AM systems, PM techniques for AM have not been studied in great detail. Proposed approaches often draw from research in related applications, such as PM for bearings. For example, Yoon et al. evaluated the feasibility of AM equipment fault diagnosis using a piezoelectric strain sensor and an acoustic sensor. In this work, features such as RMS value, kurtosis, skewness, and crest factor were used to detect faults [12]. Deep learning has also been used for AM anomaly detection, such as in [13] where a neural network was used to classify faults in 3D printer vibration data.

Despite the proliferation of statistical, ML, and DL approaches to PM for manufacturing, to the best of our knowledge, no HTM-based solutions have been proposed. However, the structural and temporal properties of HTM algorithms allow them to excel at cross-domain tasks that apply to manufacturing, such as anomaly detection [14]. Since the core objective of PM in manufacturing is detecting early symptoms of part failure, HTMs are a natural candidate for this task. HTMs were shown to match or surpass neural networks at detecting and classifying foreign materials on a conveyor belt in a cigarette manufacturing plant [15]. HTMs have also proven effective at detecting anomalies in crowd movements [16], traffic patterns [17], human vital signs [18], electrical grids [19], and computer hardware [20].

Overall, PM for manufacturing presents the following key research challenges:

1. 1.

   Identifying time-series anomalies in near real-time despite ambient noise.

2. 2.

   Learning efficiently from small training datasets to improve applicability to practical use cases.

3. 3.

   Developing a solution that can be generalized to many heterogeneous manufacturing systems without requiring extensive domain-specific tuning.

4. 4.

   Adapting to changes in data statistics (i.e., machine aging).

Despite the successes achieved by existing methods in the aforementioned applications, industrial manufacturing systems are diverse and complex, making it difficult to find solutions that generalize across applications. Consequently, PM systems require specialization, which necessitates specialized knowledge and cross-domain skills. This is especially true in the case of bearing-failure prognosis, as bearing design and lifetime management lies squarely in the mechanical and materials engineering domains.

It is difficult for any single technique to address all these research challenges effectively. For example, statistical methods such as thresholding based on kurtosis or spectral analysis are highly efficient and real-time capable but require explicitly defined health indicators and thresholds, which are machine- and application-specific. Also, stationary methods including RMS, kurtosis, and crest factor are only effective for stationary signals (signals with time-invariant statistical properties), but bearing vibration signals are generally cyclostationary (statistical properties vary cyclically) or non-stationary (statistical properties change depending on speed and load conditions)[21]. Spectral kurtosis is applicable to non-stationary and non-periodic signals but is sensitive to noise and outliers [22].

Classical ML algorithms such as AR Models, Support Vector Machines, Hidden Markov Models (HMM), Random Forests, and k-Nearest Neighbors have been demonstrated for PM in existing work, but require the extraction of explicit health indicators (features) from data [23]. These algorithms also require application-specific hyperparameter tuning, data preprocessing as they have poor noise robustness [3], and regular updates of model settings as they do not adapt to account for machine aging [23]. Moreover, both HMM and AR methods are ineffective on non-stationary signals [21].

In DL algorithms such as neural networks and LSTMs, health indicators can be learned implicitly by the network. However, a network trained for one machine cannot generalize to a new machine without retraining with a large amount of data for hundreds or thousands of epochs. Larger models may be able to generalize better, but the complexity of training and optimizing these models increases drastically with size [23]. This domain-specific training and tuning process can be expensive, time-consuming, and impractical for real-world use cases. Like the ML methods, DL algorithms also have poor noise robustness [24] and require high-quality data, or else performance can suffer significantly [3]. To address this, significant preprocessing steps are often needed to generate clean data for these models [3].

As stated in Section I-A, HTM-based anomaly detection methods have demonstrated success in several distinct fields. However, to the best of our knowledge, no prior work has comprehensively explored HTM’s ability to model vibration data or demonstrated its practical value for PM. Overall, all of these existing methods fall short of addressing one or more research challenges.

To address these key research challenges and improve on the PM performance demonstrated by previous works, our paper presents the following contributions:

1. 1.

   We demonstrate the ability of HTM-based anomaly detectors to detect early symptoms of bearing failure in several months’ worth of real-world vibration data. We show that HTM’s can efficiently learn with only a single training pass.

2. 2.

   We demonstrate the ability of HTMs to generalize across applications without much fine-tuning and their ability to continuously learn and adapt by evaluating their anomaly detection performance on a second, highly dynamic application: 3D printer vibration data. These characteristics of HTMs make them more practical for real-world use cases.

3. 3.

   We compare the performance of HTM anomaly detection methods against state-of-the-art anomaly detection techniques and traditional machine prognosis methods such as condition-based maintenance. Specifically, we evaluate each algorithm’s anomaly detection accuracy and robustness to noise.

4. 4.

   We demonstrate the efficiency and real-time capability of HTM-based prognosis by comparing its execution time with that of the other techniques.

Hierarchical temporal memory is a sequence learning framework modeled after the structure of the neocortex in the human brain [4].

The basic unit of HTM is a neuron modeled after those present in the neocortex (Fig. 1(b)). These neurons are stacked on top of one another to form a column like the ‘cortical column’ of the neocortex. The final HTM is a composition of many such columns. A single HTM neuron (Fig. 1(c)), is connected to two types of segments: (i) proximal segments (aggregation of feed-forward connections from the input) and (ii) distal segments (aggregation of lateral connections from neurons of the other columns). Each HTM neuron can be in three states: (i) inactive (the default state), (ii) predictive, and (iii) active. The predictive state of a neuron is determined by the activity of the distal segments, which in turn is determined by the activation state of the other neurons. A neuron becomes active at any time only if it was in the predictive state at the previous instant, with an exception that will be described in Section III-A. When the sequences of activations are viewed temporally, it is easy to see that the distal segments provide the temporal context for activation and thus capture the temporal relations. The column structure augments this capability of HTM by enabling them to store multiple such overlapping temporal sequences. Further details on the HTM-based anomaly detection methodology are discussed in Section III-A.

Roller-element bearings perform the critical task of reducing friction between rotating parts in machinery. Generally, catastrophic bearing failures present warning signs such as anomalous vibrations and/or noise. These anomalies can occur due to environmental factors (moisture or debris entering the bearing) as well as installation errors (misalignment, excessive loads, or poor/improper lubrication) [25]. Recently, sensor-based techniques that leverage vibration and temperature data to monitor bearing health have been proposed. For example, the NASA Bearing Dataset and the Pronostia Bearing Dataset contain vibration and temperature data for several bearings which were run until failure [26, 27]. In both datasets, anomalies in the vibration and temperature signals increase in size and frequency as the bearings approach failure, showing a strong correlation between the sensors’ readings and system state.

3D printing is a manufacturing process where a physical object is constructed from layers of material in an iterative process. Fused Deposition Modeling (FDM) is a standard technique where melted thermoplastic is extruded through a moving print head nozzle to build each layer. To ensure precision, stepper motors control the extrusion rate of the nozzle as well as the X, Y, and Z-axis movement of the print head. Since the motors, bearings, and belts are moving parts, they are prone to wear and must be regularly maintained to prevent component failures. As shown in [28], these components leak vibration information that can be used by PM systems. However, this leaked information is non-stationary since 3D printers move on multiple axes and change direction and speed often, presenting a challenge for conventional PM methods.

The end-to-end framework for the HTM-based detector is shown in Figure 2. Our methodology for anomaly detection consists of the following steps. First, the time-series vibration data $X(t)$ is taken as input and encoded into a Sparse Distributed Representation (SDR). Next, the SDR is passed through the spatial pooler. The spatial pooler’s output is fed into the temporal pooler, which then outputs a prediction for the next activation $\Pi(t_{n+1})$. Simultaneously, the prediction from the previous time step $\Pi(t_{n})$ is compared with the column activations in the current time step $A(t_{n})$ to give a prediction error value: a high error value indicates that this activation was not expected and may be anomalous. Finally, the anomaly detector uses the historical distribution of anomaly scores to calculate the anomaly likelihood $L(t_{n})$ for the current data point based on the prediction error value; if $L(t_{n})$ exceeds a set threshold, then $X(t_{n})$ is flagged as an anomaly. In the following paragraphs, we describe each of these components in detail.

We evaluate our proposed methodology on real-world bearing failure and simulated 3D printer failure datasets. Here, we discuss details about these datasets and the scoring system used for evaluation.

Table I shows the NAB results for the selected algorithms on the labeled bearing failure dataset as well as the total running time of each algorithm. The runtime was recorded over the complete dataset using a PC with an Intel Core i7-7700k processor. As shown in Table I, TM-HTM+HD achieved the highest anomaly detection score for the Standard and Low FN profiles while HTM+LP achieved the highest score for the Low FP profile. TM-HTM+HD scored 67.05, 73.33, and 56.57 for the Standard, Low FN, and Low FP profiles, respectively. The approach that scored closest to HTM was Windowed Gaussian, which achieved scores of 64.70, 70.50, and 57.35 for the same profiles, respectively. HTM and HTM+LP performed better than TM-HTM TM-HTM+LP, indicating that TM-HTM’s implementation only works well with the HD anomaly likelihood block.

As expected, the statistical methods (Windowed Gaussian, Threshold-Based, Relative Entropy) processed the dataset faster than the DL, ML, and HTM-based methods, albeit with lower performance. The HTMs using HD were 1.41x slower than the HTMs with no anomaly likelihood and 3.76x faster than the HTMs using LP on average. TM-HTM+HD processed the dataset 8.3x faster than LSTM.

To evaluate the qualitative performance of each anomaly detector, we plotted the anomaly scores over time for each detector for Test 1 of the Pronostia Bearing dataset and compared them to the labeled ground truth anomaly windows in Figure 6.

Table II shows our experimental results for the 3D printer dataset. HTM+HD achieved the highest score on the Low FN profile while LSTM achieved the highest score on the Standard and Low FP profiles. HTM+HD achieved scores of 63.03, 73.18, and 42.23 for the Standard, Low FN, and Low FP scoring profiles, respectively. LSTM scored 64.76, 71.43, and 51.34 at the same profiles, respectively. On both applications the HTM, TM-HTM, and TM-HTM+LP detectors performed worse than the HTM+HD, HTM+LP, and TM-HTM+HD detectors. Overall, the use of HD anomaly likelihood yielded the best HTM performance across applications. Each algorithm’s execution time is consistent with the results shown in Table I.

Interestingly, algorithms that performed well on the bearing dataset, such as EXPoSE and Etsy Skyline performed worse on the 3D printer dataset. Additionally, algorithms that performed worse on the bearing dataset, such as LSTM and BC performed much better on the 3D printer dataset. Our HTM-based methodology using HD anomaly likelihood achieved consistently high performance on both applications without any hyperparameter tuning, demonstrating that this configuration can generalize and adapt to different applications without domain-specific tuning. This result also suggests that HTMs significantly benefit from the inclusion of an HD anomaly likelihood block.

Also, HTM+LP was the best performing model on the Low FP profile for the bearing dataset. However, this performance was not replicated in the 3D printer dataset. Similarly, LSTM beat HTM on the Standard and Low FP profile for the 3D printer dataset while performing worse than HTM on the bearing dataset. Hence, our results suggest that LSTMs are highly data-dependent and need to be re-tuned for every machine and/or application. Thus, the LSTM approach is time-consuming, expensive, and impractical for real-world applications.

The benefits of HTM’s continuous learning capability are clearly shown in Figure 6: after identifying earlier anomalies, the HTM-based approaches learn the new baseline for the signal and can pinpoint the future anomalies despite higher signal amplitudes. CAD-OSE also appears to learn continuously, but not as well as the HTMs.

In addition to detection accuracy and precision, an optimal PM system should be able to detect failure symptoms in real-time to allow adequate time for repairs to be scheduled and performed.

However, part failures are infrequent and generally present progressive symptoms before failure, so a hard real-time requirement for processing raw sensor data may unnecessarily limit the complexity (and subsequently the performance) of anomaly detection methods. Thus, we evaluate the anomaly detectors in the context of ”soft real-time,” where we determine if each detector can process a subsampled data segment before the next subsampled data segment arrives. For example, 1 second of data can be recorded each minute as a data segment to reduce data size while still ensuring that a wide range of vibration frequencies are captured at frequent intervals.

Both HTM+HD and TM-HTM+HD were able to process the complete bearing failure dataset in under 100 minutes; since the bearing dataset contains several months’ worth of vibration data and minimal data preprocessing was performed (subsampling and timestamping), this demonstrates that HTMs can accurately detect failure symptoms in real-time, meaning that machine operators can be notified of degradation promptly. Other complex algorithms such as CAD-OSE, KNN-CAD, and EXPoSE had execution time on the same order of magnitude as HTMs and are thus also capable of real-time anomaly detection. Although HTM+LP, TM-HTM+LP, and LSTM took longer to process the dataset than HTM+HD and TM-HTM+HD, they can still be considered real-time due to the aforementioned dataset characteristics. However, the significant training time associated with the LSTM (over 12 hours on our hardware platform) and the need for application-specific hyperparameter tuning put LSTM at a disadvantage in terms of applicability to practical use cases.

Figure 6 clearly shows HTM’s ability to pinpoint anomalies while remaining robust to noise in the input. This is likely due to HTMs use of sparse encodings, making it unlikely that bit errors in the input due to noise will affect the bits corresponding to the input pattern, making them robust to noise. From the figure, it is also clear that the HTM implementations using anomaly likelihood blocks were more robust to noise outside of the anomaly windows than the HTM or TM-HTM alone. This is likely because the anomaly likelihood components filter out smaller detections to isolate only the most plausible anomalies. The HTM+HD and TM-HTM+HD detected anomalies earlier than the other configurations, albeit with slightly more false positives. The outputs of the different HTMs starkly contrast with the highly variable anomaly score outputs of Windowed Gaussian, EXPoSE, KNN-CAD, and BC, among others. These detectors record high anomaly scores even when there is relatively low noise in the input, meaning that they will likely suffer from false positives at higher noise levels.

A detector’s threshold can be tuned to account for higher noise levels; however, for detectors such as Windowed Gaussian, which used the maximum detection threshold of 1.0, the threshold cannot be increased further to reduce its sensitivity. In contrast, TM-HTM+HD used a threshold of 0.5497 on the Standard profile. Thus, although Windowed Gaussian outperformed TM-HTM+HD on the Low FP scoring profile, it lacks tunability and will likely perform much worse than this HTM configuration in more noisy environments.

LSTM appears to have good robustness to noise, as shown in Figure 6. However, it is clear from the figure that it missed some of the earlier anomaly windows completely. In the context of PM, this can mean that an observer will only be warned of degradation later and will not have much time to organize repairs. Overall, our methodology demonstrates significant noise-robustness, better tunability, and the ability to detect early anomalies as well as larger, late-stage anomalies.

Another related PM problem is Remaining Useful Life (RUL) estimation. In many cases, RUL and anomaly detection go hand in hand as part of a comprehensive PM system. Although we did not evaluate the performance of HTM at RUL estimation, the core architecture of HTM is good at sequence prediction and could likely be used to solve this problem. We leave this for future work.

Another limitation of our work is the use of synthesized 3D printer anomalies instead of real-world examples of 3D printer failures. Due to resource constraints, we opted not to perform these experiments and used synthetic failure data instead. The question of whether HTM’s performance on synthetic anomalies translates to real-world PM remains an open research problem.

The idea of predicting machine failures in advance is not brand new; many variants of PM systems have already been implemented in real-world manufacturing applications. However, based on our results, we believe that HTM is a better solution than current state-of-the-art methods. Our results demonstrate that HTMs are efficient enough to run on consumer-grade processors while learning and adapting continuously. Additionally, HTMs can be easily installed on existing PM systems as they only require time-series sensor inputs, which likely already exist in the system. As shown by our results, the industry-standard LSTM requires a significant amount of time for training (over 1000 epochs) as well as application-specific tuning. In contrast, HTMs do not require any application-specific parameter tuning and are essentially plug-and-play since they only need to be trained with a single pass on normal sensor data. These characteristics make HTMs an extremely viable, out-of-the-box solution for industrial PM.