Adaptive Gaussian Fuzzy Classifier for Real-Time Emotion Recognition in Computer Games

Rules are created and updated depending on the behavior of the system over time. An eGFC rule, say $R^{i}$, is

Rules are created and evolved as data are available. A new granule, $\gamma^{c+1}$, and rule, $R^{c+1}$, are created if none of the existing rules $\{R^{1},...,R^{c}\}$ are sufficiently activated by $\textbf{x}^{[h]}$.

Let $\rho^{[h]}\in[0,1]$ be an adaptive threshold. If

A new $\gamma^{c+1}$ has membership functions $A_{j}^{c+1}$, $\forall j$, with

The class $C^{c+1}$ is initially undefined. If the corresponding output, $y^{[h]}$, associated to $\textbf{x}^{[h]}$, becomes available, then

In case a labeled sample activates a labeled rule, but their labels are different, then a new (partially overlapped) granule is created to represent new information. Partially overlapped Gaussian granules, tagged with different labels, tend to have their modal values withdrawn and dispersions reduced by the updating procedure (Section II-C). With this initial rule parameterization, preference is given to the design of granules balanced along its dimensions. eGFC realizes the principle of justifiable granularity [21], but allows Gaussians to find more appropriate places and dispersions.

The $i$-th rule is candidate to be updated if it is sufficiently activated by an unlabeled sample, $\textbf{x}^{[h]}$, according to

To include $\textbf{x}^{[h]}$ in $R^{i^{*}}$, the learning algorithm updates the modal values and dispersions of $A_{j}^{i^{*}}$, $\forall j$,

Let the activation threshold, $\rho^{[h]}$, be time-varying [16]. The threshold assumes values in the unit interval according to the overall average dispersion

Given $\textbf{x}^{[h]}$, rules’ activation degrees are compared to $\rho^{[h]}$ to decide between parametric or structural change of the eGFC. We suggest $\rho^{[0]}=0.1$ as starting value.

A distance measure between Gaussian objects is

A new granule, say $\gamma^{i}$, which results from $\gamma^{i_{1}}$ and $\gamma^{i_{2}}$, is built by Gaussians with modal values

A rule is removed from the eGFC model if it is inconsistent with the current environment. In other words, if a rule is not activated for a number of time steps, say $h_{r}$, then it is deleted from the rule base. However, if a class is rare, then it may be the case to set $h_{r}=\infty$, and keep the inactive rules. Removing rules periodically helps to keep the rule base updated.

See [13] or [19] for a pseudocode of the semi-supervised algorithm to construct and update an eGFC on the fly.

The eGFC is evaluated from EEG streams generated by individuals playing computer games. A game motivates predominantly a particular emotion, according to the quadrants of the Arousal-Valence circle [22]. On the left side of the valence dimension are negative emotions, e.g., ‘angry’, ‘bored’, and relative adjectives, whereas on the right side are positive emotions, e.g., ‘happy’, ‘calm’, and relative adjectives. The upper part of the arousal dimension characterizes excessive emotional arousal or behavioral expression, while the lower part designates apathy. We assign a number to the quadrants (Figure 1) according to our classification problem. The classes are: bored, calm, anger, and happy. As the players effectively change the outcome of the game, their mental activity is high. They cognitively process images, build histories mentally, and evaluate characters and situations. At first, emotions are not prone to any quadrant [2] [23].

The raw data is provided in [20]. The data were produced by 28 healthy individuals (experimental group) from 20 to 27 years old. An individual plays a game for 5 minutes (20 minutes in total) using the Emotiv EPOC+ EEG device and earphones. The male-female (M-F) order of players is: F M M M F F M M M M M F M M M M F F F F F M M M M M M M. A single user-independent eGFC model is evolved for the experimental group aiming at reducing individual uncertainty, and enhancing the reliability and generalizability of the system. Brain activity is recorded from 14 electrodes placed on the scalp according to the 10-20 System, namely, at Af3, Af4, F3, F4, F7, F8, Fc5, Fc6, T7, T8, P7, P8, O1, and O2 (Figure 2). A letter identifies the lobe. F stands for Frontal, T for Temporal, P for Parietal, and O for Occipital. Even and odd numbers refer to positions on the right and left brain hemispheres. The sampling frequency is 128Hz. Each individual produces 38,400 samples per game, and 153,600 samples in total.

The experiments are conducted in a dark and quiet room. A laptop with a 15.6 inch screen, with 16GB high-quality graphic rendering, is used. The games are played in the same order: ‘Train Sim World’, ‘Unravel’, ‘Slender The Arrival’, and ‘Goat Simulator’. Figure 3 illustrates their interfaces. According to a survey, the predominant class of emotion for the games are, respectively, boredom (Class 1), calmness (Class 2), angriness/nervousness (Class 3), and happiness (Class 4).

A fifth-order sinc filter is applied to the raw data to suppress movement artifacts [20]. Subsequently, feature extraction is performed. We generate 10 features from each of the 14 EEG channels. They are the maximum and mean values of 5 bands of the Fourier spectrum. The bands are known as Delta (1-4Hz), Theta (4-8Hz), Alpha (8-13Hz), Beta (13-30Hz), and Gamma (30-64Hz). In case 5-minute time windows are used to construct the frequency spectrum, and compose a processed sample to be fed to eGFC, then each individual produces 4 samples – one sample per game, i.e., one sample per class or predominant emotion. Therefore, the 28 participants generate 112 processed samples. We also evaluate 1-minute, 30-second, and 10-second time windows, which yield 560, 1120, and 3360 processed samples. Examples of spectra using 30-second time windows over data streams from frontal electrodes – Af3, Af4, F3 and F4 – are illustrated in Figure 4. Notice a higher level of energy in the Delta, Theta and Alpha bands, and that the maximum and mean values per band can be easily obtained to compose a 10-feature processed sample.

In a first experiment, the data of individual electrodes are analyzed (univariate time series analysis). 10-feature samples, $\textbf{x}^{[h]}=[x_{1}~{}...~{}x_{10}]$, $h=1,...$, are fed to the eGFC, which evolves from scratch. We perform the train-after-test approach, i.e., an eGFC estimate is given; then the true class $C=\{1,2,3,4\}$ of $\textbf{x}^{[h]}$ becomes available, and the pair $(\textbf{x},y)$ is used for a supervised learning step. Classification accuracy, $Acc\in[0,1]$, is obtained recursively from

A measure of model compactness is the average number of fuzzy granules over time,

Second, we consider the global 140-feature problem (multivariate time series analysis) – being 10 features extracted from each electrode. Thus, $\textbf{x}^{[h]}=[x_{1}~{}...~{}x_{140}]$, $h=1,...$, are considered as eGFC inputs. The classifier self-develops on the fly based on the train-after-test strategy. Dimension reduction is performed using the non-parametric Spearman’s Correlation-based Score method [24]. Fundamentally, a feature is better ranked if it is less correlated to other features, and more correlated to a class. The Leave $n$-Feature Out method ($n=5$) gradually eliminates lower ranked features.

We evaluate individual EEG channels to discover more propitious brain regions in terms of their relations to emotion patterns. Default hyper-parameters are used to initialize eGFC, i.e. $\rho^{[0]}=\Delta=0.1$, $h_{r}=200$ [13]. Input samples contain 10 entries. Table I shows the accuracy and compactness of eGFC models for different time window lengths.

From Table I we notice that the mean accuracy for 5-minute windows, $21.1\%$, does not reflect learning. This suggests that the filter effect on extracting features from 5-minute windows suppresses crucial details to distinguish patterns. Emotions tend to be sustained along shorter periods. Accuracies greater than $25\%$ arise for smaller windows. The average accuracy for 1-minute windows, $40.5\%$, is significant, especially due to the limitation of observing a single electrode. As window length reduces, accuracy increases using a more compact model. This is a result of the availability of a larger amount of processed samples (extracted from a smaller amount of time windows), and the ability of the learning algorithm to lead the eGFC to a more stable set up after merging and deleting granules. The difference between the average accuracy of the 30-second ($44.0\%$) and 10-second ($45.2\%$) windows becomes small, which suggests saturation around 45-46%. Windows smaller than 10 seconds tend to be needless. Asymmetries between the accuracy of models evolved for the left and right brain hemispheres can be noticed. Although the right hemisphere is known to deal with emotional interpretation, and be responsible to creativity and intuition; logical interpretation – typical of the left hemisphere – is also present as players seek reasons to justify decisions. Thus, we notice the emergence of patterns in both hemispheres.

In general, with focus on the 30- and 10-second windows, the pair of frontal electrodes, Af3-Af4, gives the best recognition results, especially Af3. The frontal cortex includes the premotor and primary motor cortices, which controls voluntary movements of body parts. Thus, patterns that arise from the Af3-Af4 streams may be strongly related to brain commands to move hands and arms – which is indirectly related to the emotions of the players. A spectator, instead of a game actor, may have a classification model based solely on Af3-Af4 with diminished accuracy. The pair Af3-Af4 is closely followed by the pairs O1-O2 (occipital lobe) and T7-T8 (temporal lobe), which is somehow consistent with the results in [25] [26]. The temporal lobe is useful for audition, language processing, declarative memory, and visual and emotional perception. The occipital lobe contains the primary visual cortex and areas of visual association. Neighbor electrodes, P7-P8, also offer relevant information for classification.

The multivariate 140-feature stream is fed to eGFC. We consider 10-second windows over 20-minute recordings per individual – the best set up secured in the previous experiment. Features are ranked from the Spearman’s Correlation Score. To give quantitative evidence, we sum the monotonic correlations between a band and the classes of emotions (a sum of 14 items that correspond to the 14 EEG channels). Figure 5 shows the result in dark yellow, and its decomposition per brain hemisphere. The global values are precisely: 2.2779, 1.7478, 1.3759, 0.6639, and 0.6564 for the Alpha, Delta, Theta, Beta, and Gamma bands, respectively. The higher, the better.

We apply the strategy of leaving $5$ features out to evaluate the user-independent eGFC. Each eGFC model delineates class boundaries in a different multi-dimensional space. Table II shows the results for the multi-channel streams. A quadcore laptop, i7-8550U, with 1.80GHz, 8GB RAM, and Windows 10 OS, is used. We testify that brain activity patterns are found in all frequency bands, viz., Delta, Theta, Alpha, Beta, and Gamma, as even lower ranked features positively affect the estimates. Features from any channel may assist the classifier decision. The greatest eGFC accuracy, $72.20\%$, occurs using all features extracted from the spectrum. In this case, a stream of 3,360 samples is processed in 6.032 seconds (1.8ms per sample). As a sample is generated at every 10s (10-second windows), and processed in 1.8ms, we have shown that eGFC can operate in real time with much larger rule bases and Big data, such as data generated by additional electrodes, and other physiological and non-physiological means.

The average number of fuzzy rules along the learning steps, around 5.5, is similar for all streams analyzed (Table II). An example of eGFC structural evolution (best case, $72.20\%$) is shown in Figure 6. Notice that the emphasis of the model and algorithm is to keep up with accuracy during online operation at the price of merging and deleting rules occasionally. If a higher level of memory is desired, then the default eGFC hyper-parameters can be changed by turning off the rule removing threshold, $h_{r}=\infty$; and setting the merging parameter, $\Delta$, to a lower value. Figure 6 also shows that the effect of male-female shifts due to the consecutive use of the EEG device does not imply a need of completely new granules and rules. Parametric adaptation of Gaussian granules is often enough to accommodate slightly different behaviors. Nevertheless, if a higher level of memory is allowed, rules for both genders can be kept separately. In general, the results using evolving fuzzy intelligence and EEG streams as unique source of physiological data are encouraging.