Unsupervised Motor Imagery Saliency Detection Based on Self-Attention Mechanism

Input embedding: Given a multivariate signal as input, the embedding layer captures the dependencies between different channels [10] in EEG signals and creates a representation vector for each time sample. We employ a 1D convolutional layer with $m$ numbers of $d$-dimensional kernels. This layer produces an $m$-dimensional vector for each time sample by looking at current and $d{-}1$ next time samples. We utilize zero padding to ensure that the input and output sequences have the same length.

LSTM encoder: LSTM as a variant of recurrent neural networks can capture long temporal dependencies in sequential data [11]. We employ LSTM units to build our encoder and decoder. The encoder processes the input sequence of length $T$ and summarizes the temporal information of each time sample in two $h$-dimensional vectors, the hidden state $h_{e}$ and the cell state $c_{e}$. The produced hidden states are then fed to a self-attention layer. After $T$ recurrences, the whole sequence is summarized in final encoder hidden state $h_{eT}$ and cell state $c_{eT}$. These two vectors are fed to the decoder’s initial hidden state $h_{d0}$ and cell state $c_{d0}$.

Self-attention: Attention is originally proposed in [7] to improve the performance of an encoder-decoder in a machine translation. It alleviates the bottleneck problem of using the fixed-length vector from which the decoder produces the output sentence by allowing the model to automatically seek source sentence parts that are significant for predicting a target word. Self-attention, also known as intra-attention, is an attention mechanism that links distinct points in a single sequence to calculate a representation of the sequence [12]. In this work, we adopt the self-attention mechanism to detect the salient parts of each MI-EEG trial. In our self-attention layer, the $i$-th encoder hidden state $h_{ei}{=}\{h_{ei,1},...\,,h_{ei,h}\}$ is mapped to an $n_{k}$-dimensional query vector $q_{i}$, an $n_{k}$-dimensional key vector $k_{i}$, and an $n_{v}$-dimensional value vector $v_{i}$ by multiplying $h_{ei}$ with three matrices $W_{q}$, $W_{k}$ and $W_{v}$, respectively:

$$q_{i}=W_{q}h_{ei}$$

(1)

$$k_{i}=W_{k}h_{ei}$$

(2)

$$v_{i}=W_{v}h_{ei}$$

(3)

These vectors are used to calculate a context vector in each position in the sequential data. Here, dot-product attention [12] is used to compute the context vectors. The context vector at $i$-th position is a weighted sum of value vectors in all positions

$$c_{i}=\sum_{t=1}^{T}\alpha_{i,t}v_{t},$$

(4)

where the attention weight $\alpha_{i,t}$ measures how well the $i$-th encoder hidden state aligns with the encoder hidden state at position $t$. The attention weights are computed as the inner product of the query with the corresponding key. After computing the weights associated with all keys, a softmax function is used to make the weights sum up to one. If all query vectors, key vectors, and value vectors are packed together into $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$ matrices, the context matrix $\mathbf{C}$ is calculated as follows

$$\mathbf{C}=softmax(\mathbf{Q}\mathbf{K}^{T})\mathbf{V}.$$

(5)

LSTM decoder and dense layer: The LSTM decoder accepts the final hidden state and cell state of the encoder as its initial hidden state and cell state. The context vectors generated by the self-attention layer are fed to the decoder inputs. The decoder outputs are then passed through a dense layer to increase the dimensionality of decoder outputs to the number of EEG channels. The decoder and dense layer learn to reconstruct the original MI trial by compelling the self-attention layer to pay more attention to certain parts of the MI trial that are more significant for reconstructing the input.

Training stage: To improve the performance of the proposed network in reconstructing the input EEG, $x$, we randomly select $p_{1}$ time samples of $x$ and set $p_{2}$ feature values to zero to obtain $\tilde{x}$. In the training stage, $\tilde{x}$ is used as the training trial, and its corresponding original EEG signal $x$ is used as the ground truth. The mean squared error is used as the loss function

$$Q=\frac{1}{Tn_{c}}\sum_{t=1}^{T}\sum_{i=1}^{n_{c}}(\hat{x}_{t}^{(i)}-x_{t}^{(i)})^{2},$$

(6)

where $n_{c}$, $\hat{x}$ and $x$ are the number of EEG channels, the reconstructed EEG, and the original EEG, respectively.

Testing stage: The decoder and dense layer are removed from the network for the testing stage. A test trial is passed through the embedding layer, the encoder, and the self-attention layer. The computed attention weights $\{\alpha\}$ are used to detect the salient parts of the test trial. Assume that the attention weights of a test trial are represented in a matrix $\Lambda_{T\times T}$. The attention at position $t$, $a_{t}$, is computed as follows

$$a_{t}=\frac{1}{T}\sum_{j=1}^{T}\Lambda_{j,t}.$$

(7)

The computed attentions form an attention vector $A=[a_{1},\,...\,,a_{T}]$. We segment the attention vector into $n$ intervals for separating salient intervals. $r$ out of $n$ intervals with the highest average attention are selected. These two hyperparameters are tuned in the cross validation procedure. The time samples corresponding to the selected attention intervals are then concatenated to form the pruned trial.

To evaluate the proposed method, we compare the performance of the CSP algorithm under two scenarios: 1) with and 2) without applying our method to the MI-EEG trials.

In the first scenario, we train our proposed architecture with the $4$-second training trials. After the training stage, we use the encoder and self-attention layer to extract the salient intervals from the training and testing trials. The spatial filters of the CSP algorithm are then designed using the pruned training trials. The accuracy of the CSP algorithm is computed using the pruned testing trials. In the second scenario, the $4$-second training trials and testing trials are directly used to design the spatial filters and obtain the accuracy of the CSP algorithm, respectively. In both scenarios, linear discriminant analysis is used as the classifier.

The results are shown in Fig. 3 for all the subjects. We observe that the classification accuracy is improved approximately by $14.2\%$, $19.4\%$, $6.6\%$, $31.9\%$, $24.7\%$, $9.2\%$, $10.4\%$, $1.5\%$, and $7.4\%$ for subject one through nine, respectively. On average, the proposed method enhances the classification accuracy of the CSP by about $13.9\%$ across all subjects.

In addition, the ratio of the length of the pruned trials to the length of the unpruned trials is $0.7$, $0.65$, $0.45$, $0.7$, $0.55$, $0.7$, $0.75$, $0.75$, and $0.75$ for subject 1 through 9, respectively. Hence, the proposed method reduces the computational burden required for processing EEG signals and increases the classification accuracy simultaneously.

This section evaluates the effect of different segment lengths $T/n$ used for splitting the attention vector to extract the salient intervals. Fig. 4 compares the results in terms of classification accuracy for subject one in the dataset versus the number of time samples in each interval, $T/n$. $r$ intervals with the highest average attention are concatenated to form a trial containing $\ell{=}rT/n{=}750$ (red dashed line), $\ell{=}550$ (green dashed line), $\ell{=}350$ (orange dashed line), and $\ell{=}250$ (blue dashed line) time samples. The black horizontal line shows the classification accuracy of the CSP algorithm without our method. We observe that restricting the concatenated signal to have a very short length ($\ell{=}350$ and $\ell{=}250$) results in lower accuracy compared to the CSP method alone. As shown the best curves correspond to $\ell{=}750$ and $\ell{=}550$. These curves prove that using the entire lengths of the trials does not always achieve the best obtainable accuracy.

In addition, the results suggest that shorter segment lengths $T/n$ can generally provide higher accuracy than longer segment lengths. In other words, using multiple time windows instead of a single time window [1, 6] results in detecting more salient intervals in a trial. The proposed method may be employed as a prepossessing step in MI-BCI studies for enhancing motor imagery classification.