Research map / Vision-language and multimodal / Human and embodied modeling
In Silico Mapping of Visual Categorical Selectivity Across the Whole Brain
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a transformer brain encoder (DINOv2 features with parcel-query cross-attention) trained on NSD and uses it, with ImageNet and BrainDIVE diffusion stimuli, to hypothesize and statistically validate whole-brain parcel category selectivity in silico.
Problem
Existing category-selectivity mapping relies on experimenter-chosen categories or linear encoding models, and fMRI stimulus sets are too small to discover fine-grained selectivity beyond visual cortex.
Approach
An encoder-decoder transformer with learnable parcel-specific queries cross-attending to frozen DINOv2 patch features predicts whole-brain fMRI nonlinearly; the encoder then ranks large image sets (ImageNet, BrainDIVE diffusion images) to produce per-parcel CLIP-space concept labels that are validated against held-out NSD data.
Key results
- DINOv2 ViT-B with transformer cross-attention mapping reaches encoding accuracy of 0.45, 0.43, 0.48, 0.43 on subjects 1, 2, 5, 7, outperforming the linear head (0.33–0.39).
- On NSD test, encoder-selected BrainDIVE labels yield the best Spearman rho in subjects 2 and 5 (0.190 and 0.154 vs 0.164 and 0.150 for NSD-train baseline), while differences in other subjects are small.
- Cross-subject test: encoder-ImageNet labels label 45/49 parcels in subject 2 vs 44/49 for the NSD-train baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use transformer cross-attention instead of a linear mapping head | DINOv2 ViT-B brain encoder · NSD held-out test, subjects S1, S2, S5, S7 | encoding accuracy: 0.33 -> 0.45; 0.34 -> 0.43; 0.39 -> 0.48; 0.33 -> 0.43 | Helped |
| Use DINOv2 rather than CLIP vision features with a transformer mapping | Brain encoder · NSD held-out test, subjects S1, S2, S5, S7 | encoding accuracy: 0.44 -> 0.45; 0.41 -> 0.43; 0.46 -> 0.48; 0.43 -> 0.43 | Mixed |
| Use DINOv2 ViT-G rather than ViT-B with transformer mapping | Brain encoder · NSD held-out test, subjects S1, S2, S5, S7 | encoding accuracy: 0.45 vs 0.45; 0.43 vs 0.43; 0.48 vs 0.48; 0.45 vs 0.43 | No clear effect |
| Use ImageNet-derived labels rather than NSD training-image labels | Within-subject parcel selectivity evaluation · NSD test S1-S7 | Spearman's ρ: 0.162 -> 0.168; 0.164 -> 0.163; 0.150 -> 0.142; 0.148 -> 0.133 | Mixed |
| Use BrainDIVE-derived labels rather than NSD training-image labels | Within-subject parcel selectivity evaluation · NSD test S1-S7 | Spearman's ρ: 0.162 -> 0.163; 0.164 -> 0.190; 0.150 -> 0.154; 0.148 -> 0.133 | Mixed |
| Use ImageNet-derived labels rather than NSD training-image labels | Cross-subject parcel selectivity evaluation · NSD training set S1-S7 | Spearman's ρ: 0.068 -> 0.072; 0.094 -> 0.105; 0.090 -> 0.105; 0.091 -> 0.093 | Mixed |
| Use BrainDIVE-derived labels rather than NSD training-image labels | Cross-subject parcel selectivity evaluation · NSD training set S1-S7 | Spearman's ρ: 0.068 -> 0.078; 0.094 -> 0.103; 0.090 -> 0.106; 0.091 -> 0.089 | Mixed |
| Use fewer top images to form the concept vector (K=1 rather than K=32) (ablation) | Within-subject NSD parcel-label evaluation · NSD test S1-S7 | Spearman's ρ: ImageNet S1: 0.168 ± 0.106 -> 0.148 ± 0.101; BrainDIVE S2: 0.190 ± 0.099 -> 0.165 ± 0.093 | Mixed |
| Use fewer top images to form the concept vector (K=1 rather than K=32) (ablation) | Cross-subject NSD parcel-label evaluation · NSD training set S1-S7 | Spearman's ρ: ImageNet S1: 0.072 ± 0.071 -> 0.073 ± 0.070; BrainDIVE S1: 0.078 ± 0.077 -> 0.081 ± 0.078 | Mixed |
| Vary the number of top images used to generate labels across K=64, 32, 16, 8, 4, 2, 1 (ablation) | Within-subject parcel-label evaluation · NSD test | parcels significantly better than chance: Results vary across K; e.g. NSD train S1: 150/181 at k=32 and 114/181 at k=1; ImageNet S1: 139/181 at k=32 and 133/181 a | Mixed |
| Vary the number of top images used to generate labels across K=64, 32, 16, 8, 4, 2, 1 (ablation) | Cross-subject parcel-label evaluation · NSD training set | parcels significantly better than chance: Results vary across K; e.g. ImageNet S2: 45/49 at k=32 and 44/49 at k=1; BrainDIVE S1: 37/49 at k=32 and 39/49 at k=1 | Mixed |
Surprising findings
- Head-to-head, the encoder-selected labels beat the NSD-train baseline in only 62/196 parcels for subject 7 with ImageNet, suggesting gains are subject-dependent.
- Attention maps follow expected stimuli less closely than in prior similar encoders, which the authors attribute to a priori parcel boundaries.
Limitations
- Superstimuli have not been validated in new fMRI experiments.
- Encoder trained only on NSD, so may inherit dataset biases (e.g., giraffe/zebra over-representation).
- Parcel selection relies on a labeled-area overlap criterion and percentile cutoffs that vary by subject.
Experimental setup
Datasets: Natural Scenes Dataset (NSD) ImageNet BrainDIVE generated images
Benchmarks: NSD test split NSD training set (cross-subject tests)
Metrics: Pearson correlation (noise-ceiling corrected) Spearman rank correlation fraction of parcels significantly above chance recall of top-32 images
Baselines: Linear mapping on DINOv2 CLS token CLIP vision transformer encoder RADIOv2.5-H encoder NSD-train top images baseline
Builds on: BrainDIVE DINOv2 Algonauts 2023 encoding models
Closest papers by meaning
- Brain Mapping with Dense Features: Grounding Cortical Semantic Selectivity in Natural Images With Vision Transformers (ICLR 2025 · similarity 0.90)
- BrainSCUBA: Fine-Grained Natural Language Captions of Visual Cortex Selectivity (ICLR 2024 · similarity 0.89)
- Transformer brain encoders explain human high-level visual responses (NeurIPS 2025 · similarity 0.89)
- Finding Shared Decodable Concepts and their Negations in the Brain (ICLR 2025 · similarity 0.88)
- CLIP-MSM: A Multi-Semantic Mapping Brain Representation for Human High-Level Visual Cortex (AAAI 2025 · similarity 0.88)
- Meta-Learning an In-Context Transformer Model of Human Higher Visual Cortex (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.