SSVEP-DANet: Data Alignment Network for SSVEP-based Brain Computer Interfaces

Domain adaptation techniques aim to adapt the trained model from the source domain to the target domain by leveraging the available data from the target domain while utilizing the knowledge learned from the source domain. The goal is to improve the model’s performance and generalization capabilities across different users without the need for extensive retraining or user-specific calibration.

Domain adaptation methods in SSVEP-based BCIs are typically achieved through shared feature spaces or subspaces, as well as utilizing transformation relationships between the data. Additionally, based on the differences in domain adaptation methods, we can categorize these methods into two groups: 1) subspace alignment, which involves sharing feature spaces or subspaces, and 2) data alignment, which involves finding transformation relationships between samples [sarafraz2022domain, wan2021review, wu2020transfer].

Subspace alignment methods align the source and target domains to a common subspace to reduce the discrepancy between the two domains. Riemannian Procrustes Analysis (RPA) [rodrigues2018rpa] achieves this by applying simple geometric transformations (translation, scaling, and rotation) to symmetric positive definite matrices (SPD), aligning the source and target domains to the same subspace. Although this method can be applied across subjects and sessions, its practicality is limited due to the output being SPD matrices. Shared Latent Response (SLR) [masaki2020facil] uses common spatial filtering methods, including CCA and TRCA, to extract features from the training data and then uses least squares regression to obtain new spatial filters that project test data onto the same subspace as the training data. This approach is applicable to cross-subject, cross-session, and cross-device scenarios, with input data being common time series data, providing more flexibility in practical applications. ALign and Pool for EEG Headset domain Adaptation (ALPHA) [liu2022alpha] aligns spatial patterns through orthogonal transformations and aligns the covariance between different distributions using linear transformations, mitigating variations in spatial patterns and covariance. This method further improves upon obtaining new spatial filters and achieves better performance than SLR. Additionally, it can be applied to cross-subject, cross-session, and cross-device scenarios with input data being time series data, thereby providing more practical opportunities. Tangent Space Alignment (TSA) [bleuze2022tsa] shares similarities with RPA as it aligns different domains to the same subspace through translation, scaling, and rotation, but operates within the tangent space. The tangent space being Euclidean allows for faster decoding, and rotation can be achieved with a singular value decomposition (SVD), making it computationally efficient compared to RPA. TSA is also applicable to cross-subject, cross-session, and cross-device scenarios. However, it outputs tangent vectors, which impose limitations on subsequent classifiers, thus restricting its practicality.

Data alignment methods align source domain samples with target domain samples in order to mitigate the disparities between the two domains. The Least Squares Transformation (LST) approach [Chiang2021LST] finds a linear transformation relationship among the SSVEP data, effectively reducing the errors between the transformed data from the source SSVEP and the target SSVEP. This method is applicable in cross-domain scenarios (sessions, subjects, and devices). Furthermore, the output is time series data and differs from some subspace alignment methods [masaki2020facil, liu2022alpha] that can only identify target stimuli by computing correlation coefficients between spatial features in the same subspace. This method can be integrated into commonly used SSVEP decoding algorithms such as CCA, TRCA, and conCA, significantly enhancing their flexibility and feasibility in real-world applications. Therefore, we aim to design a data alignment method that can be applied to commonly used SSVEP decoding algorithms or future, more powerful SSVEP decoding models.

In recent years, several non-linear SSVEP models for data augmentation, such as SBSGAN [aznan2019simulating], S2S-StarGAN [kwon2022novel], and TEGAN, [pan2023short] have demonstrated promising results. This suggests that SSVEP signals can exhibit nonlinear characteristics. Additionally, SSVEP is a time-synchronous data; we will focus on spatial characteristics rather than temporal. Therefore, in this study, we assume that there exists a non-linear and channel-wise transformation of SSVEP signals between subjects. We propose a neural network-based transformation to transfer source SSVEP signals to target SSVEP signals. Figure 3 shows our network architecture, which consists of two modules: one spatial convolution and two channel-wise fully connected layer, a tangent hyperbolic (Tanh) activation function is used between two fully connected layers. The input to DANet is SSVEP data obtained in the source domain format to be $N_{C}\times N_{S}$ and the transformation target from the existing training format to be $N_{C^{\prime}}\times N_{S}$ is the SSVEP template obtained in the target domain, which is obtained by averaging multiple trials corresponding to specific stimulation from the target subject for noise suppression and signal improvement, where $N_{C}$ indicates the number of channels, $N_{C^{\prime}}$ indicates the number of channels, $N_{S}$ is the number of time samples.

Since SSVEP is time-synchronous data, several detection methods, such as CCA and TRCA, apply a spatial filter to find a linear combination of channel-wise SSVEP signal. These methods have been demonstrated to improve signal-to-noise ratio (SNR) [Johnson2006snr] and enhance SSVEP detection performance. SCCNet [wei2019sccnet] applies a network architecture based on spatial components for noise reduction. Therefore, in the first module, we use spatial convolution to project the original SSVEP data to latent spaces to obtain spatial features, where $N_{C}$ spatial filters with kernel size of $(N_{C},1)$. Then, the spatial features are permuted in order of $(2,1,3)$, and batch normalization is applied. Finally, we permute the features in order of $(1,3,2)$ as input for the next module.

In the second module, we perform two channel-wise fully connected layers and one activation function to find the non-linear channel relation between spatial features and target SSVEP templates. First, we use a channel-wise fully connected layer to integrate channel information at each time point and project the feature to a new latent space. Second, we utilize the activation function $tanh$ to make the data easier to fit our model. At last, we use a channel-wise fully connected layer to project the features into the target domain space and permute the data in order of $(1,3,2)$ as SSVEP data similar to the target domain data.

(i) cross-stimulus : The training approach involving cross-stimulus training enables the learning of model parameters across different stimulus frequencies, even when the available calibration data is limited, thereby ensuring a relatively reliable learning process. Therefore, during the training of SSVEP-DANet, all stimuli from all subjects are utilized to learn distinct mapping relationships between the target domain and the source domain, facilitating the attainment of a more stable domain adaptation model.

(ii) pre-trained: Many studies utilize the fine-tuning technique in transfer learning, which involves taking the pre-trained model and training some of its layers on new data, to improve the recognition performance for SSVEP-based BCI [ravi2020ssvepIDUD] [guney2021ssvepDNN]. Similarly, in order to get better model weights to transfer the existing domain to the target domain, we fine-tune the pre-trained model to produce different model weights using different subjects. The procedure of our method is shown in figure 4.

For each subject from the source domain, we utilize a fine-tuned model corresponding to a specific subject to transfer the subject data. We concatenate all transformation data from the source subjects and calibration data from the target subjects to form a larger training set. The new training set can be used to obtain a new spatial filter and SSVEP template using the TRCA-based method. Finally, we perform the SSVEP-decoding algorithm to classify the target stimuli.

We fit the network using Adam optimizer [kingma2014adam] with learning rate of 0.0005. And MSE (Mean Square Error) between target SSVEP data $X_{T}$ and output SSVEP data $X^{\prime}_{T}$ is used to measure a transformation loss $L_{trans}$:

The source subjects are splitted into the training set and the validation set in the ratio of 8:2 in this study. We train 500 epochs and save our model weights that produce the lowest validation set loss. In the fine-tune stage, we fine-tune 150 epochs for each source subject.

(i) Dataset I. : The benchmark dataset [wang2017benchmark] used in this study is a publicly available SSVEP dataset prepared by the Tsinghua group. In this dataset, the SSVEP-based BCI experiment involved 35 participants. Each participant participated in 6 blocks of the experiment, each block comprising 40 trials presented in random order. Visual stimuli were presented within a frequency range of 8 to 15.8 Hz with an interval of 0.2 Hz. The phase values of the stimuli ranged from 0, with a phase interval of 0.5$\pi$. EEG signals were recorded using the extended 10-20 system through 64 channels. We selected EEG data from eight channels (PO3, PO4, PO5, PO6, POz, O1, O2, Oz) in the analysis and performance evaluation. The EEG signals were down-sampled from 1000 to 250 Hz, and a notch filter at 50 Hz was applied to remove the common power-line noise. The data were extracted in [$L_{1}$ s, $L_{1}$ + $T_{w_{1}}$ s], where time zero denotes stimulus onset, $L_{1}$ is the latency delay ($L_{1}=$ 0.14 s) and $T_{w_{1}}$ indicates the time-window length ($T_{w_{1}}=$ 1.5 s).

(ii) Dataset II. : The wearable SSVEP BCI dataset [zhu2021wearalbe] used in this study is a publicly available SSVEP dataset prepared by the Tsinghua group. In this dataset, 102 healthy subjects participated in the wearable SSVEP-based BCI experiment. The experiment consisted of 10 blocks, each of which contained 12 trials in random order of 12 visual stimuli. Stimulation frequencies ranged from 9.25 to 14.75 Hz with an interval of 0.5 Hz. The phase values of the stimuli started at 0, and the phase difference between two adjacent frequencies was 0.5$\pi$. The 8-channel EEG data was recorded using wet and dry electrodes, and the electrodes were placed according to the international system 10-20. All channels (PO3, PO4, PO5, PO6, POz, O1, O2, Oz) of the EEG signals were used in data analysis and performance evaluation. The EEG signals were resampled at 250 Hz from 1000 Hz. To remove the common power-line noise, a 50 Hz notch filter was applied to the dataset. The data was extracted in [0.5 + $L_{2}$ s, 0.5 + $L_{2}$ + $T_{w_{2}}$ s], where 0.5 s denotes stimulus onset, $L_{2}$ indicates latency delay ($L_{2}$ = 0.14 s) and $T_{w_{2}}$ is the time-window length ($T_{w_{2}}=$ 1.5 s).

In this section, we experimentally validate the performance of the SSVEP-DANet scheme compared to three other schemes (Baseline, Concat. , and LST) when the number of calibration trials for each stimulus varies for the target subject. We also discuss the impact of different numbers of calibration trials for each stimulus on the effectiveness of our SSVEP-DANet scheme.

Figure 5 demonstrates the cross-domain performance using different schemes under varying numbers of calibration trials for each stimulus. The results indicate that our proposed method consistently outperforms the other three schemes in most cases, regardless of whether the calibration data for the target subject are sufficient. Moreover, as the number of calibration trials for the target subject increases, our proposed method consistently improves the performance of the SSVEP decoding algorithm. Interestingly, due to domain variability, the naive transfer learning (Concat. scheme) exhibits a negative impact on the TRCA-based method. Similarly, the LST-based method also adversely affects the TRCA-based method, as the accuracy of the transformation matrix is influenced by unstable SSVEP signals. Additionally, with a large amount of training data, we found that our proposed method does not rely on the data quality of a single source domain and can stably enhance the performance of the SSVEP decoding algorithm under different numbers of calibration trials for each stimulus.

This section validates the performance of the w/ SSVEP-DANet scheme compared to the other three schemes under different numbers of supplementary subjects. Additionally, it discusses the impact of varying numbers of supplementary subjects on the effectiveness of our w/ SSVEP-DANet scheme.

Figure 6 illustrates the cross-domain performance using different schemes under varying numbers of supplementary subjects. The results demonstrate that our proposed method outperforms the other three schemes in most cases, regardless of the number of supplementary subjects. Interestingly, the results indicate that both w/o SSVEP-DANet and the w/ LST schemes exhibit negative impacts on the TRCA-based method. We observed a significant drop in performance for the LST-based method as the number of supplementary subjects from dry electrode devices increases, indicating that the LST method transfers more low-quality SSVEP data for training TRCA. Furthermore, similar to the experiments mentioned earlier, our proposed SSVEP-DANet model, trained on a large amount of data, is capable of improving the performance of the SSVEP decoding algorithm consistently across different numbers of supplementary subjects, irrespective of the quality of the SSVEP signals. On the other hand, in both the dry-to-dry and wet-to-dry scenarios, since SSVEP-DANet is a training-based approach, incorporating more supplementary subjects enhances its data alignment capability and effectively transforms the data from the source domain.

We utilized t-SNE [van2008tSNE] to perform dimensionality reduction on EEG data, reducing it to 2D in order to compare SSVEP trials between two scenarios: SSVEP-DANet and Concat. , as well as among target subjects. Additionally, we investigated whether the use of SSVEP-DANet can effectively reduce differences among the subjects.

Figure 7 (a) presents the t-SNE visualization of the 13th target subject from the Benchmark dataset in a cross-subject scenario. Figure 7 (b) illustrates the t-SNE visualization of the first target subject wearing a dry electrode device in the Wet-to-dry scenario, using data from the Wearable dataset. From these figures, we observed that under the same stimulus, the clusters in the SSVEP-DANet scheme were smaller compared to the clusters in the Concat. scheme. Additionally, under different stimuli, the clusters in the SSVEP-DANet scheme showed stronger separation compared to the Concat. scheme. It is noteworthy that, under the same stimulus, the clusters in the SSVEP-DANet scheme did not consistently align closely with the target subject clusters. These findings suggest that the SSVEP-DANet method reduces inter-subject variability and increases inter-stimulus variability.

We utilized EEG spectrograms to investigate the impact of increased similarity among subjects on the power spectral density at the target frequency.

Figures 8 (a) depict the average spectra of the 12.6 Hz SSVEP signals for target subject 3 in the Benchmark dataset, under three schemes (Baseline, Concat. , and SSVEP-DANet), when the calibration trials for each stimulus are two. Similarly, Figures 8 (b) show the average spectra of the 14.75 Hz SSVEP signals for target subject 1 wearing a dry electrode device in the Wet-to-dry scenario from the Wearable dataset, with calibration trials two, for each stimulus under the three schemes (Baseline, Concat. , and SSVEP-DANet). In the Concat. scheme, we observed different outcomes in Figure 8 (a) and Figure 8 (b). The results in Figure 8 (a) indicate that the Concat. scheme on the standard dataset struggles to obtain stable spectra due to the high variability of SSVEP trials in the scheme, resulting in less concentrated spectral peaks at the target frequency. Conversely, the results in Figure 8 (b) demonstrate that on the wearable dataset, where the target participants have suboptimal SNR, the Concat. scheme can effectively improve SNR by incorporating a large amount of higher-quality data, leading to more stable spectra. In the SSVEP-DANet scheme, significant enhancements in peak amplitudes are observed at the target frequency and its harmonics in both Figure 8 (a) and Figure 8 (b). These findings suggest that our proposed SSVEP-DANet method effectively reduces inter-trial variability in SSVEP experiments, enabling the utilization of non-target subject trials and increasing the SNR. Moreover, these phenomena are reflected in the decoding accuracy (figure 5).

We conducted ablation experiments to evaluate the effectiveness of different training methods in SSVEP-DANet when target subjects have limited calibration data. Table III presents the results of SSVEP-DANet using different training methods. The Single + w/o FT scheme involves training the model using a single stimulus across subjects and using model transfer SSVEP signal from the corresponding stimulus in the source domain. The Subject scheme entails training distinct subject-specific models by employing various individual source subjects. These models are then employed to transfer SSVEP signals from corresponding subjects within the source domain. The All + w/o FT scheme, on the other hand, involves training the model using all the training data across subjects and applying the same model to transform the SSVEP signal from the source domain. The results indicate that our proposed training approach significantly outperforms both the Single + w/o FT scheme, the Subject scheme and the All + w/o FT scheme in most cases. This suggests that our proposed training scheme can effectively train a robust and stable model by utilizing a large amount of data. Fine-tuning the model using specific source domain subjects and target subjects allows for efficient transfer of source domain subject information to the target subjects, thereby significantly improving the accuracy of SSVEP recognition.