A contrastive-learning approach for auditory attention detection ^††thanks: This work was supported in part by the National Science Foundation under grant IIS-2235228, the Center for Cognitive and Brain Sciences, and by the Ohio Supercomputer Center.

Everyone struggles with understanding speech in noisy environments, such as at parties or sporting events. Many machine learning approaches, based on speech enhancement or speaker extraction, have been proposed to isolate the attended sound signal amongst competing noise or talkers, respectively. These approaches, however, assume that the attended signal is known beforehand, which is not the case in reality.

Auditory attention detection (AAD) addresses this problem, where statistical and machine learning approaches have been developed. The general idea is that the electroencephalographic (EEG) signal can be provided as input to a machine learning or statistical algorithm that uses the input to determine the attended sound source of the listener. The pivotal work in [1] introduced the idea of the dependency of brain processing in cocktail party scenarios on many factors including, the location of the voice, visual queues like lips-reading, and the speaker’s pitch, to name a few. In [2], it is shown that the spectral reconstruction of the cortical surface response of multiple speakers were highly correlated with the temporal and spectral features of the attended signal, where low correlation results for the unattended signal. Leveraging this finding, many early AAD approaches reconstruct the audio stimuli from the EEG signal (backward or reverse approaches) and identify the audio source with the highest correlation to the EEG signal as the attended source [3, 4, 5, 6, 7]. These approaches are known as linear methods, which also include correlation based learning, canonical correlation analysis (CCA) [8], and linear filter based methods (e.g. FIR) [9]. The simplicity and low computational cost of these models are, however, accompanied with low generalizability and accuracy, especially for small decision windows, which is undesirable.

More recently, approaches based on nonlinear mapping [10] and convolutional neural networks (CNNs) [11, 12, 13] have been developed for auditory attention detection. A comparison of linear based methods versus neural network based methods [13] shows a drastic improvement (about 15% increase in average accuracy) in detection for neural network based methods, even with a lower number of recording electrodes. A CNN based approach for similarity maximization of the latent space of the EEG and sound sources was used in [14], which tried to maximize the similarity with the attended speech. A recursive gated convolution method for left and right classification using only the EEG data was used in [15], where graph convolutional networks have also been used [16]. In [17], frequency and channel based attention is used for the EEG data, where a CNN network helps to predict AAD. Similarly, the work in [18] used a joint LSTM-CNN model. Many alternative neural network approaches have been proposed including, graph neural networks [19], which have the advantage of better signal explainability, but at the cost of accuracy; cross modal attention on the EEG and audio frequency bands [20]; attention [21]; and an unsupervised learning approach that showed a label-independent learning stage can be used to surpass a supervised only approach [22]. To the best of our knowledge, however, only one approach evaluates AAD performance on an unseen subject whose data is excluded from the training set [23], which is the optimal way to assess how well the approach generalizes.

Although convolutional and conventional neural network approaches have advantages over linear methods, these approaches are time dependant or they do not capture the essential differences between electrodes. Contrastive learning, as one of the many different methods for self supervised learning, learns the internal patterns and structures in the data, without a training label. Inspired by this approach, we propose a new method to learn the EEG patterns and make the learned representation more useful for the downstream task of attended source detection. As the base of our method, we use a pair of cross-modal attention-based auditory attention detection (CMAA) encoders, introduced in [24], which uses a cross modal attention module [25]. We devise a loss function that maximizes the similarity of the encoder representations out of the two CMAA paths, and minimize the binary cross entropy divergence for the classification loss. This approach addresses the need for better generalization in between experiments and between subject, one of the challenges in the biological signal processing. A contrastive approach has been proposed [26], but it uses a variational auto encoder (VAE) for latent representation learning, which even though it encourages more flexibility over an auto encoder, it is not a better choice over CMAA for a contrastive approach, due to the added stochasticity.

A depiction of our proposed approach is shown in figure 1. Common spatial pattern (CSP) features are first computed from the EEG signals [27], after pre-processing the EEG data (more in section III-B). The EEG recordings have a high correlation with nearby electrodes, which is an issue because it increases redundancy in the data and limits the useful information available to the neural network. CSP address this by maximizing the variance between different categories which in turn makes more distinctive inter-channel EEG data available. CSP has been used for feature processing for several approaches [12, 28], and it has helped improve performance. More details can be found in [27].

The audio envelope is extracted after bandpass filtering. The attended and competing audio sources are both processed in this manner. The processed audio signals are each paired with the procesesed EEG data, and provided to separate CMAA encoders, with the goal of getting meaningful contrastive and classification projections from their latent representations that can be useful for the AAD classification task. By utilizing self attention modules in the CMAA modules, we get better generalization in comparison with CNN blocks alone.

The network is based on the contrastive method. In [29], a contrastive method was proposed in which the latent representation produced from two augmentations of the same data instance should be close together, while representations for augmentations of different data instances should be farther apart. This method requires large batch sizes and a sizable dataset, which is not applicable to this work, mainly due to the limited size of available AAD datasets. In order to overcome this limitation, a new self supervised approach was adapted [30], where the base network was the same, but it uses different augmentations of the same input that are passed through different paths of the network that share parameters. In our experiments, it was shown that the augmentation was essential for the training.

The data augmentation for the EEG is challenging. Unlike the data augmentation applied to images, where we have multiple transformation options, which after they are applied, can generate valid and in distribution data examples, for EEG data we do not have an intuitive understanding of appropriate transformations that produce valid EEG signals. To mitigate this issue, we used standard normal Gaussian noise (e.g., zero mean and unit variance) as an augmentation for the audio and the CSP of the EEG data, across all channels and time samples, to improve generalization. Gaussian noise is chosen since it does not change the expectation of the signal, but increases its variance.

tionCMAA Encoder The CSP feataugmented ures are thenand audio provided as inputs to our network. The base network is composed of five components shown in figure 2. For each CMAA sub component, we layer normalize the audio and EEG input, and provide the normalized audio and EEG outputs to a multi-head attention (MHA) block within the cross attention module. The MHA output is added with the query input using a skip connection. Next the output is passed through a normalization layer, a fully connected (FC) layer, and then the FC output is added with the FC input using a skip connection. Another FC layer follows. Next to add the positional information, the final FC output is added to the positional encoding [25] to keep the network informed about the sequential ordering. This creates the output for the first EEG-audio pairing, but this process is repeated $N=5$ times here and for the other EEG-audio pairing. Also, eight attention heads were used. This is performed for each audio signal and EEG data pairing, where the two outputs are concatenated together, and then provided to a fully connected layer. We experimented with using cosine similarity between the different audio and EEG pairs instead of performing concatenation, but concatenation produced better results. The representations are then provided to pairs of probe and classifier heads, which are each three FC layers.

The data was divided into decision windows of 0.5, 2 and 5 second lengths with $50\%$ overlap, before being fed to the network for training, which are commonly-used window lengths. We ran two experiments. Eighty percent of the data was used for training and $20\%$ for validation. First we use 5-fold cross validation to train and validate the model for each subject and reported the average accuracy among them. Second, we use cross-subject validation, in which the network was trained on all subjects excluding one subject, and the trained network was then tested on the excluded subject.

Table I shows the per-subject accuracy of the network trained on the DTU dataset. We achieved the top accuracy on the 0.5s, 2s and 5 second windows among different methods. The comparison results are as reported in the corresponding papers. In [23], the authors report 63% accuracy for the 5 second window. We achieved better results for this window size and also achieved comparably high accuracy for the 2 seconds window. This generalization is attributed to the CLAAD loss, which helps the network learn data patterns rather than finding a close model for the classification task alone. The results for each subject using our proposed approach are shown in figure 3 (top), where performance is relatively consistent for each subject for all decision windows.

In figure 3 (bottom) the result for the exclusion case is plotted, in which one subject is reserved for validation and the data of the other subjects is used for training.

The figure shows that in some cases the model gets almost perfect validation accuracy, especially for the 2 and 5 second decision windows. The 0.5 decision produces the lowest results, which is expected since it is the most challenging scenario. Note that these results are still better than the 5 second decision window results from another approach that held data from one subject for evaluation (e.g. [23] with $63\%$ accuracy).

In this paper we proposed a self supervised approach based on contrastive learning to produce a more meaningful representation of the input data which can help in a downstream AAD task. To show that a self supervised approach performs better than conventional neural network with cross entropy loss, we adopted the CMAA network as the base encoder and showed with minimal change of parameters, we can have a much more effective network. To show the generalizability of our method, we tested the network on unseen training subjects, which shows even in presence of distribution shift in between subject, the network learns good generalizations in most cases.