Research map / Architectures and efficiency / Attention and transformers
Transformer brain encoders explain human high-level visual responses
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Transformer Brain Encoder that uses learnable ROI or vertex queries to cross-attend over frozen backbone patch features and predict fMRI responses in NSD, outperforming ridge regression and spatial-feature factorized encoders with far fewer parameters.
Problem
Linear encoding models from DNN features to fMRI are parameter-heavy and ignore feature map structure, while existing spatial-feature factorized models learn static receptive fields that only suit early visual areas and cannot route information dynamically by image content in high-level visual corte
Approach
Use a single-layer transformer decoder whose learned queries (one per ROI or per vertex) cross-attend over frozen backbone patch tokens (with positional encodings) so each brain region dynamically routes content-relevant features, and its attention maps serve as interpretable routing signals.
Key results
- Transformer (ROI queries) with DINOv2 reaches 0.60/0.56/0.56/0.42 encoding accuracy for subjects 1/2/5/7 versus 0.56/0.52/0.50/0.37 for ridge regression, with ~28M vs ~900M parameters.
- Vertex-based queries improve to 0.63/0.59/0.57/0.44, and ensembling across DINOv2 backbone layers reaches 0.65/0.62/0.59/0.45.
- The same ranking holds with ResNet50, CLIP, and BERT caption backbones; the transformer beats ridge regression on BERT captions (e.g., 0.27 vs 0.19 for subject 1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ROI-query transformer routing instead of ridge regression | Brain encoding model with DINOv2 features · NSD, subjects 1, 2, 5, and 7 | encoding accuracy: 0.56 -> 0.60; 0.52 -> 0.56; 0.50 -> 0.56; 0.37 -> 0.42 | Helped |
| Use ROI-query transformer routing instead of ridge regression | Brain encoding model with ResNet50 features · NSD, subjects 1, 2, 5, and 7 | encoding accuracy: 0.49 -> 0.52; 0.48 -> 0.50; 0.47 -> 0.50; 0.37 -> 0.38 | Helped |
| Use ROI-query transformer routing instead of ridge regression | Brain encoding model with CLIP features · NSD, subjects 1, 2, 5, and 7 | encoding accuracy: 0.51 -> 0.53; 0.48 -> 0.49; 0.47 -> 0.50; 0.38 -> 0.38 | Helped |
| Use ROI-query transformer routing instead of ridge regression | Brain encoding model with BERT caption features · NSD, subjects 1, 2, 5, and 7 | encoding accuracy: 0.19 -> 0.27; 0.21 -> 0.27; 0.25 -> 0.33; 0.19 -> 0.27 | Helped |
| Use vertex queries instead of ROI queries (ablation) | Transformer brain encoder with DINOv2 features · NSD, subjects 1, 2, 5, and 7 | encoding accuracy: 0.60 -> 0.63; 0.56 -> 0.59; 0.56 -> 0.57; 0.42 -> 0.44 | Helped |
| Ensemble backbone layers (ablation) | Vertex-query transformer brain encoder with DINOv2 features · NSD, subjects 1, 2, 5, and 7 | encoding accuracy: 0.63 -> 0.65; 0.59 -> 0.62; 0.57 -> 0.59; 0.44 -> 0.45 | Helped |
| Increase training set size (ablation) | ROI-query transformer brain encoder with DINOv2 features · NSD S1 | encoding accuracy: 0.41 -> 0.46 -> 0.51 -> 0.56 -> 0.60 | Helped |
| Increase training set size (ablation) | ROI-query transformer brain encoder with DINOv2 features · NSD S2 | encoding accuracy: 0.45 -> 0.48 -> 0.52 -> 0.54 -> 0.56 | Helped |
| Increase training set size (ablation) | ROI-query transformer brain encoder with DINOv2 features · NSD S5 | encoding accuracy: 0.45 -> 0.49 -> 0.51 -> 0.54 -> 0.56 | Helped |
| Increase training set size (ablation) | ROI-query transformer brain encoder with DINOv2 features · NSD S7 | encoding accuracy: 0.34 -> 0.36 -> 0.39 -> 0.40 -> 0.42 | Helped |
| Use vertex queries instead of ROI queries (ablation) | Spatial-feature factorized encoder with DINOv2 features · NSD, subjects 1, 2, 5, and 7 | encoding accuracy: 0.49 -> 0.52; 0.46 -> 0.48; 0.48 -> 0.48; 0.37 -> 0.37 | Mixed |
| Use vertex-level routing for higher visual areas (ablation) | Transformer brain encoder · NSD high-level visual areas | encoding accuracy | No clear effect |
Surprising findings
- Gains over ridge regression come mainly from better prediction of higher-level categorical areas, while vertex-level routing improves mainly early visual areas.
- Text captions alone (BERT features from BLIP captions) predict high-level visual areas, with only high-level areas predicted from semantic information.
Limitations
- Evaluated only on NSD fMRI, so generality to other datasets, recording modalities (EEG, neurophysiology), and video or audio is untested.
- Vertex-wise routing does not reflect biological receptive field implementation via anatomical and wiring constraints.
- Reads out both early and high-level areas from a single backbone, without modeling connectivity between early and high-level areas.
Experimental setup
Datasets: Natural Scene Dataset (NSD) Algonauts split
Benchmarks: Natural Scene Dataset (NSD)
Metrics: encoding accuracy (fraction of explained variance, noise-cei Pearson correlation category selectivity of attention maps
Baselines: Ridge regression CLS + regression PCA + regression Spatial-feature factorized encoder Saliency based integration
Builds on: DETR transformer cross-attention spatial-feature factorized encoding models BrainDiVE
Closest papers by meaning
- A Convolutional Neural Network Interpretable Framework for Human Ventral Visual Pathway Representation (AAAI 2024 · similarity 0.89)
- In Silico Mapping of Visual Categorical Selectivity Across the Whole Brain (NeurIPS 2025 · similarity 0.89)
- Brain Mapping with Dense Features: Grounding Cortical Semantic Selectivity in Natural Images With Vision Transformers (ICLR 2025 · similarity 0.87)
- Unifying Top-down and Bottom-up Scanpath Prediction Using Transformers (CVPR 2024 · similarity 0.87)
- Meta-Learning an In-Context Transformer Model of Human Higher Visual Cortex (NeurIPS 2025 · similarity 0.87)
- Flexible Context-Driven Sensory Processing in Dynamical Vision Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.