Identifying Shared Decodable Concepts in the Human Brain Using Image-Language Foundation Models

The natural scenes dataset (NSD) is a massive fMRI dataset acquired to study the underpinnings of natural human vision. Eight participants were presented with 30,000 images (10,000 unique images over 3 repetitions) from the Common Objects in Context (COCO) naturalistic image dataset (Lin et al., 2014). A set of 1,000 shared images were shown to all participants, while the other 9,000 images were unique to each participant. Single-trial beta weights were derived from the fMRI time series using the GLMSingle toolbox (Prince et al., 2022). This method fits numerous haemodynamic response functions (HRFs) to each voxel, as well as an optimised denoising technique and voxelwise fractional ridge regression, specifically optimised for single-trial fMRI acquisition paradigms. Some participants did not complete all sessions, and three sessions were held out by the NSD team for the Algonauts challenge. Further details can be found in Allen et al. (2022).

To generate representations for each stimulus image, we use a model trained on over 400 million text-image pairs with the contrastive language-image pretraining objective (CLIP (Radford et al., 2021)). The CLIP model consists of a text-encoder and image-encoder that are jointly trained to maximize the cosine similarity of corresponding text and image embeddings in a shared low-dimensional space. We use the 32-bit Transformer model (ViT-B/32) implementation of CLIP to create 512-dimensional representations for each of the stimulus images used in the NSD experiment. We train our decoder to map from fMRI responses during image viewing to the associated CLIP vector for that same image.

We randomly split the per-image brain responses $\bm{X}$ and CLIP embeddings $\bm{Y}_{\text{CLIP}}$ into training $(\bm{X}^{\text{Train}},\bm{Y}^{\text{Train}}_{\text{CLIP}})$, validation $(\bm{X}^{\text{Val}},\bm{Y}^{\text{Val}}_{\text{CLIP}})$, and test $(\bm{X}^{\text{Test}},\bm{Y}^{\text{Test}}_{\text{CLIP}})$ folds. The validation and test folds were each chosen to have exactly 1,000 images. Some participants in the NSD did not complete all scanning sessions and only viewed certain images once or twice. We assign these images to the training set. Of the shared 1,000 images, 413 were shown three times to every participant across the sessions released by NSD. These 413 images appear in each participant’s testing fold.

The noise ceiling is often used to identify the voxels that most reliably respond to visual stimuli. The NSD fMRI data comes with voxelwise noise ceiling estimates, but they are calculated using the full dataset. These estimates can be used to extract a subset of voxels for decoding analyses, but this takes into account voxel sensitivity to images we later want to tune and test on, and is a form of double-dipping (Kriegeskorte et al., 2009). We therefore re-calculated the per-voxel noise ceiling estimates specifically on the designated training data only. Voxels with noise ceiling estimates above $5\%$ variance explainable by the stimulus were used as inputs to the decoding model, resulting in 10k-30k voxel subsets per participant (see Supplementary Info for exact per-participant voxel dimensions).

The decoding model $\bm{g}:\mathbb{R}^{v}\to\mathbb{R}^{512}$ is trained to map a vector of brain responses $\bm{X}=[\bm{x}_{1},\ldots,\bm{x}_{n}],\bm{x}_{i}\in\mathbb{R}^{v}$ to the CLIP embeddings of the corresponding stimulus images $\bm{Y}_{\text{CLIP}}=[\bm{y}_{1},\ldots,\bm{y}_{n}],\bm{y}_{i}\in\mathbb{R}^{512}$. Here $n$ is the number of training instances, and $v$ is the number of voxels. An illustration of the decoding procedure is given in 1(a). We define $\bm{g}$ to be a multi-layer perceptron (MLP) with a single hidden layer of size 5,000, following by a leaky ReLU activation (slope=0.01). The model is trained for 12 epochs (approximately 1,500 iterations) with the Adam optimizer and a batch size of 128. The learning rate is initialized to $1e^{-4}$ and decreased by a factor of 10 after epochs 3, 6, and 9. We train the brain decoder using the the InfoNCE loss function (van den Oord et al., 2018), which is defined as:

In the original CLIP setting, $q$ and $k$ both represent image and language embeddings, where the loss is minimized when these embeddings have a high similarity for the same images (positive class) and low similarity otherwise. In our implementation, we replace the language embeddings with the brain responses to images. We set $\tau=1$ in our implementation. We compare our model $\bm{g}$ to a baseline ridge regression model trained on the same data. We used grid search to select the best ridge regularization parameter $\lambda\in\{0.1,1,10,100,1000,10000,100000\}$ using the validation data.

We evaluate our models using top-k accuracy, which is computed by sorting in ascending order all true representations $\{y_{1}\ldots y_{n}\}$ by their cosine distance to a predicted representation $\hat{y}_{i}$. Top-k Accuracy is the percentage of instances for which the true representation $y_{i}$ is amongst the top-k items in the sorted list. Chance top-k accuracy is $\frac{100\cdot k}{n}\%$ where $n$ is the number of held-out data points used for evaluation. Figure 1(b) shows the results of this evaluation. The MLP model outperforms ridge regression across all values of $k$, motivating the need for this more complex model.

Recall that our end goal is to identify shared decodable concepts (SDC) in the brain. Our methodology for this task relies on the predicted CLIP vectors, and so an accurate deocoding model is of utmost importance.

The THINGS-Images database consists of 1854 object classes with 12 images per class (Hebart et al., 2023). These images were used to gather approximately 1.46 million responses to an odd-one-out task, during which MTurk workers were asked to select the one image (from a group of 3) that least belonged in the group. These responses were then used to learn THINGS object embeddings $[\bm{t}_{1},...,\bm{t}_{1854}]=\bm{T}$ , $\bm{t}_{i}\in\mathbb{R}^{49}$. Initially, $\bm{T}$ is randomly initialized. Then, for each triplet of classes $i,j,k$, and the human-chosen odd one out ($i$), the model is trained to maximize the dot products of the embeddings for the non-odd-one-out concepts ($\bm{t}_{j}\cdot\bm{t}_{k}$), and minimize the other two dot products ($\bm{t}_{i}\cdot\bm{t}_{j}$ and $\bm{t}_{i}\cdot\bm{t}_{k}$). The model is constrained ensure non-negativity and encourage sparsity in $\bm{T}$. After training, $\bm{T}$ contains 49 human-interpretable dimensions that are most important for performing the odd-one-out similarity judgments. These 49 dimensions were assigned semantic labels by hand.

We trained a function $\bm{h}_{\text{THINGS}}:\mathbb{R}^{512}\to\mathbb{R}^{49}$ to map from CLIP space to THINGS space. To train this mapping, all 12 images for each of the 1,854 Things object classes are passed into the CLIP image encoder to obtain $\bm{U}\in\mathbb{R}^{(12\cdot 1845)\times 512}$. We then averaged CLIP embeddings for all images within an object class to obtain $\bm{U}_{\text{avg}}\in\mathbb{R}^{1854\times 512}$. These averaged embeddings are used to fit a linear ridge regression model:

Because the THINGS-embeddings $\bm{T}$ are constrained to be non-negative, after training we introduced an additional ReLU operation, $\bm{h}_{\text{THINGS}}(\bm{Y})=\text{ReLU}(\bm{Y}\bm{W}_{\text{THINGS}}^{T})$, and further finetuned $\bm{W}_{\text{THINGS}}$. Empirical results for this fine tuning can be seen in the Supplementary Material. The CLIP-to-THINGS mapping function $\bm{h}_{\text{THINGS}}$ allows us to map CLIP embeddings derived from NSD stimulus images to THINGS space. In addition, we can use brain responses to those images passed through the decoder $\bm{g}(\bm{X})$ to derive decoded THINGS embeddings directly from the NSD fMRI.

Results for computing top-k accuracy in THINGS space appear in Figure 2 (green bar). Decoding performance in THINGS space is lower than in the original CLIP space. This implies that the THINGS concepts, originally derived from behavioral data, do not sufficiently capture the visual concepts available in CLIP space that are decodable from fMRI. This motivates the search for a new embedding space tuned specifically for the decodable concepts shared amongst all participants in the NSD dataset.

An initial question is whether the brain regions represented by $\bm{m}_{i,s}$ are truly specific to their corresponding concept vectors $\bm{w}_{i}$, or do they equally support the decodability of other concepts $\bm{w}_{j}$ where $j\neq i$? To help answer this, we measure the relative change in brain decodability for a concept $\bm{w}_{j}$ after applying a mask $\bm{m}_{i,s}$. The matrix of relative changes in brain decodability $\bm{D}^{s}$ for a participant $s$ is constructed as follows:

where $\odot$ denotes element-wise multiplication and $\text{pearsonr}(.)$ is the Pearson correlation coefficient. Then the participant-specific $\bm{D}^{s}$ are averaged across participants to obtain $\bm{D}$ which is displayed in Figure 7.

Next, we investigate whether the masks $\bm{m}_{i,s}$ exhibit consistency in their spatial locations within the brain across participants. The highly sparse and disjoint nature of the masks motivated the use of an ROI-based similarity measure. The HCP_MMP1 atlas parcellates the cortical surface into 180 distinct regions per hemisphere. We used this atlas to define ROI fraction vectors $f_{i,s}\in\mathbb{R}^{360}$ for each concept mask $m_{i,s}$. Each element in $f_{i,s}$ represents the number of voxels in $m_{i,s}$ that intersect a particular ROI in the atlas, divided by the total number of voxels in $m_{i,s}$. The similarity of mask regions can be compared across participants by taking the cosine similarity between ROI fraction vectors, i.e. $\text{cosine similarity}(f_{i,s},f_{j,t})$ where $i,j$ index concepts and $s,t$ index participants. Figure 5 outlines the method we use to identify shared decodable concepts.

Using our shared decodable concepts matrix $W_{SDC}$, (Fig. 5a), we use LASSO regularization to induce a participant-specific subset of voxels we call a voxel mask, such that each individual participant has a sparse voxel mask for each dimension of $W_{SDC}$ (Fig. 5b). The dimensionalities of the masks per-participant and per-mask are given in A.2. We demonstrate the procedure by selecting 3 synthetic example dimensions for purposes of illustration, $m1$, $m5$ and $m7$ for three participants in the fMRI dataset. In Fig-5c we show an example of how these voxel masks for 3 participants might be spatially organised along the inflated ventral surface (images generated using Freesurfer’s Freeview program (Fischl, 2012)). Dimension $m1$ (orange voxels) is highly consistent across participants, meaning that it frequently appears in ROI-1 and ROI-3 of each participant. Dimension $m5$ (magenta voxels) is consistent between participants 2 and 3, both appearing in ROI-2 but not in ROI-2 in Participant 1. This represents partial consistency. Finally, dimension $m7$ (blue voxels) represents a situation where the spatial organization of the mask does not intersect any defined ROI consistently across participants. By taking the 360 automatically-generated ROIs (180 from each hemisphere) from the Human Connectome Project (HCP) Atlas (Glasser et al., 2016), calculated by Freesurfer’s recon-all program, we count the number of voxels in each mask that intersect with each ROI and for each participant’s set of masks. This yields a 360-dimensional vector for each participant and each mask (Figure 5d). We then calculate the cosine similarity between these vectors in order to determine if there is a consistent shared representation of mask voxels across the ROIs as shown in Fig. 5e. For each dimension of $W_{SDC}$, we calculate a list of consistent (high cosine similarity) and inconsistent (low cosine similarity) scores according to the algorithms given in Fig. 5f.

By comparing the distributions of both sets of results, we identify dimensions that share stable mask distributions across participants (Fig. 5g). For some dimensions of our optimized matrix, we find greatly increased consistency across participants. We selected the top 7 dimensions and plot the 10 most associated images for that dimension in Figure 6. These shared decodable concept dimensions have associated images that are superficially very distinct but have a clearly consistent semantic interpretation.

The mask optimization procedure does not explicitly encourage the discovery of contiguous voxel subgroups, yet this is exactly what we find when generating masks for each of the decodable semantic concepts. This allows us to explore mask distributions across ROIs associated with specific semantically congruent stimuli, such as dimensions often used in fMRI localizer scans. We consistently find that semantic dimensions in our shared decoding space that contain prominent faces, bodies and places have contiguous mask locations in the expected ROIs. These voxel subgroups in our results occupy much smaller and fine-grained areas of ROIs found in localizer experiments, which may be related to processing the various fine-grained semantic distinctions. We expand on this analysis in Appendix A.1. Using the dimensions we found to be most consistent across participants (Figure 6: right), we further explore the voxel mask locations to identify which brain areas support the semantic dimensions of $W_{SDC}$ that are consistent across participants.