Time Majority Voting, a PC-based EEG Classifier for Non-expert Users

u We reviewed and implemented several machine learning algorithms commonly used in the field [9, 10, 11, 13]. For examples, Linear Classifiers, Nearest Neighbors, Decision Trees, and Ensemble Methods.

Linear Classifiers: The Shrinkage Linear Discriminant Analysis (Shrinkage LDA) performed adequately on EEG datasets with simple tasks. The Support Vector Machine (SVM), effective in high dimensional spaces, performed reasonably well based on the previous research. These algorithms are simple to implement and are computationally efficient.

Nearest Neighbor: Such a classifier implements the K-Nearest Neighbor(KNN). KNN performs voting to determine an unseen dataset to one of the k nearest neighbors. The KNN performed pretty well compared to most other classifiers on the EEG dataset.

Decision Tree: The Decision Trees classifier is easy to understand, implement, and interpret. The decision tree creates a model that predicts the outcome of a data point based on decision rules. The computational cost is low and can handle both numerical and categorical data. However, it might overfit as trees are too complex.

Ensemble Methods: We used Random Forest and boosting. These are Ensemble Machine Learning algorithms that combine the predictions of several weaker learners and form more robust and more accurate predictions. These have been widely used in EEG-based experiments, and research [24, 33, 35].

Deep Learning: We implemented CNN with ReLu, and RNN, especially LSTM, mainly using toolsets from the PyTorch platform. [37, 15, 34]. These DLs performed very well on EEG datasets. However, we excluded these algorithms in this paper due to their high runtime.

This paper proposed a new voting approach based on the top two individual classifiers. Ensemble methods, especially boosting, bagging, and voting, have demonstrated excellent performance in previous research. [24, 37, 15, 36, 34] In EEG-based BCI classification research, the following voting methods have been investigated in several experiments [24, 33]: majority voting, weighted voting, and time continuity voting. Here we considered the advantages of both majority voting and time continuity voting and developed our new Time Majority Voting (TMV) algorithm.

Time Majority Voting (TMV).  
Figure 1 demonstrated the concept of the new algorithm. More details are in the Experiment section and the Result section. There are two phases in the new algorithm. First, we investigated the state-of-the-art machine learning algorithms [24, 37, 15, 34] and found the top two performers on average. In our experiments during phase 1, We tested Random Forest (RF), RBF and linear SVM, kNN, Decision Tree, and several boosting algorithms. We found Random Forest performed the best, and RBF SVM performed the second on average. Next, we entered phase two. For each subject, we picked the majority task predicted by the best performing classifier, the Random Forest classifier, for each time interval of each task from each session.

The next step is voting. We used the Random Forest and the RBF SVM to conduct the voting process. The best algorithm is the Random Forest, and the second algorithm is RBF SVM. The voting details are shown in the four examples at the bottom of Figure 1. If both the Random Forest and the RBF SVM algorithms agree with the results, as shown in the first two rows, the results reflect both algorithms’ results. For example, as the second row shows, if both the classifiers predicted the task 2, then the Time Majority Voting will yield task 2 as the result. If the two algorithms do not agree on the prediction results, the results will be labeled as the majority of tasks already determined by the Random Forest classifier, as shown in the last two rows. No matter what the two algorithms predict out of the five tasks, even if they predicted it as two different tasks other than the majority task, as the last row shows, Task 2 and Task 3, the TMV result is still set to be Task one. This concept is based on the temporal dependence time-series features in the previous research [25, 36].

Data cleaning: As mentioned in [35, 33], for each task during each session, the first 30% of the data, which is the first 18 seconds of each task and during the transition phase, will be removed. Thus, each one-minute task only had 42 seconds left. This has been proved reasonable during the data cleaning phase. Some electrodes may have temporarily lost contact with the subjects’ scalp during the EEG recording. The result was that multiple sequential spectral snapshots from one or more electrodes had the same value. In this paper, we decide to remove such anomaly when detected for a consecutive 1.4 seconds. Such a cleaning action caused a different level of loss of the data for each subject.

Subjects: We had a total of seventeen subjects. After the data cleaning actions, subjects who lost more than 65% of the total data will be excluded from the subsequent analysis. In the end, there were twelve subjects left to continue the analysis. Moreover, for the six sessions of each subject, if a session lost more than 65% of the data, then that session will also be excluded for further analysis.

Time-wise Cross-Validation: We adopted time-wise cross-validation. We divided each task into seven subsets, meaning each subset had six seconds, evenly and continuously. Then, we created a total of seven folds. Each fold contained six seconds of data for each task for each session. We checked any folds that lost more than 65% of the original data in each fold and discarded these folds for future analysis. Next, we used one fold for testing and the remaining non-discarded folds for training, and we cross-validated them.

We reported the average accuracy for all subjects and the runtime of each classifier for each subject we trained and tested during phase 1 in table 2. As we can see, Random Forest had the highest accuracy of 0.55 in our experiment, and the SVM with RBF kernel performed adequately on the TCR dataset with an average accuracy of 0.53. Though Nearest Neighbors did not perform as well as the Random Forest and the RBF SVM, it was one of the fastest algorithms on personal computers. Other ensemble methods such as Adaboost, RusBoost, and GradientBoosting performed relatively lower than these top three algorithms.

The individual difference may impact the accuracy of each subject. But we can still recognize a general pattern from Fig 2. We ordered all twelve subjects by prediction accuracy using Random Forest. Most of the algorithms demonstrated consistent patterns for the different algorithms. Random Forest and RBF SVM are above most of the other algorithms. We kept the threshold of maintaining the subject to 35% of the remaining data, as we believed that when a subject has little data left, the high accuracy from that subject contributes little to our research.

Figure 3 shows what the Random Forest and the RBF SVM in phase 1 predicted during the 42 seconds of subject 3’s task 1 for all six sessions. As we can see, both the Random Forest and the RBF SVM mainly produced results with relatively high accuracy in sessions one, four, and six. On the other hand, session two and three had much noise, and session five predicted task 3 for the majority of the time. To minimize the impact of noisy datasets, we calculated the accuracy compared to the ground truth based on the output of Random Forest for each session for each task in phase 1.

We excluded any sessions that yielded an accuracy of less than 50%. In the case of Figure 3, we excluded sessions two, three, and five for further machine learning analysis. We reported to subject three and started discussing what may have happened in these sessions. With the exclusion of noisy sessions, the new accuracy for the Random Forest in phase 2 is referred to as "Random Forests Phase 2" later in the paper. We also performed Time Majority Voting on this cleaner dataset.

As shown in Table 3, the Time Majority Voting (TMV) has achieved a higher accuracy for subject three, all the six sessions. A value of -1 means that we excluded that session for that task, as we discussed in the previous section. The Random Forest classifier also reached a higher accuracy after cleaning the noisy sessions. As Figure 4 and Figure 5 shows, the pattern is consistent across all the subjects. Figure 5 also shows a clear pattern that not only the Random Forest but also the RBF SVM also increased accuracy in phase 2 across all subjects. Table 4 shows the TMV achieved an 80% average accuracy with an average 74.3 seconds run time. The runtime consists of 39.1 seconds of running the Random Forest and 29.5 seconds of running the RBF SVM. The time for the actual voting is, on average, 5.7 seconds. The training process is the most time-consuming part of this analysis.

Figure 6 shows what the Time Majority Voting(TMV) predicted during the 42 seconds of subject 3’s task 1 for all six sessions. As you can see, sessions 2, 3, and 5 have values of -1, which means that they have been excluded. Sessions 1, 4, and 6 have less noise than sessions 2, 3, and 6, and the accuracy is relatively high.

The innovation of this method is mainly about temporal dependency. As [24, 37, 15, 36, 34] mentioned, EEG data has a significant temporal dependency. The signal of the same task takes about 12 to 18 seconds to switch to the next task. Using majority voting can catch this type of time continuity effect. If both classifiers recognize the same pattern, it is more likely to assure the results. If both of the classifiers recognize the same pattern that is different from the majority result, it is possible that the participants were doing other tasks during the data collection. If only one classifier detects some unusual behaviors, we label it as the majority task of the session. Thus we highlight the noise and keep the remaining data to reflect more on the time continuity nature of the EEG signals.

As Figure 7 shows, the run time directly correlated to the training data size. After the data pre-processing, we cleaned up the noise with a plateau longer than a threshold, as mentioned in [24, 32, 34]. We first identified more noisy sessions during the Time Majority Voting process based on the top two classifiers. During this step, more sessions were excluded for further analysis. The training process was the most time-consuming step during the coding running process. Thus the runtime changed together with the size of the training data.

Figure 3 and Figure 6 shows two examples of the feedback results we present to the end-users. In this figure 3, the six sessions of subject three, task one is listed as six horizontal charts. The first chart represents session one. Both the Random Forest and the RBF SVM identified the majority task as task one. And for the results of the different predictions, both of the classifiers agreed at some time spots, but not all of them. Our Time Majority Voting algorithm favorite the majority voting results.

We started with the sessions with good prediction results when demonstrating these figures to each subject. For example, in this figure 3, sessions one, four, and six show pretty consistent patterns. The majority of task prediction results were the designed task one. That implies that the subject may spend more time on task one as planned during these sessions. Session two and three had a lot of different prediction results from both classifiers. Thus we suspected some unexpected reason might cause this situation. We referred back to the experiment notes and reached out to subject three. After discussing with the end-user, we figured out that he had many issues with the sensor signals and was adjusting the EEG headset most of the time during the sessions 2 and 3. Thus we had more information to exclude these sessions from further data analysis. Session five is another situation. The data implied that the subject was doing task three, but the experiment notes were missing for that session, and the subject did not remember the details about that session. Thus, we left a question mark for that session, excluded the session for now, and came up with an improvement plan to keep better experiment notes. This type of machine learning result is explainable to the end-user.

Such interpretability could contribute to a better understanding of the results and better design for future experiments.