Research map / Vision-language and multimodal / Human and embodied modeling
BrainSCUBA: Fine-Grained Natural Language Captions of Visual Cortex Selectivity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BrainSCUBA, which projects a voxel's fMRI encoder weights onto natural-image CLIP embeddings and uses a frozen CLIPCap captioner to produce per-voxel natural-language captions of visual selectivity, then uses captions for text-guided image synthesis.
Problem
Existing brain-selectivity studies rely on hand-selected stimuli and decoding-style methods; there is no way to produce fine-grained, voxel-level natural-language descriptions of the preferred visual stimuli of higher visual cortex.
Approach
Train a linear fMRI encoder on frozen CLIP ViT-B/32 embeddings, then map each voxel's weight vector into the natural-image CLIP embedding space via a softmax-weighted sum over ~2M unpaired images (training-free modality-gap closing), and caption the projected embedding with a frozen CLIPCap model.
Key results
- BrainSCUBA-100 achieves mean zero-shot category match of 74.6% (S2) and 72.0% (S5) versus BrainDiVE-100 at 78.4% and 81.6%, close to the NSD top-1% real-image baseline of 63.2% and 61.4%.
- Captions mention person nouns in 83.1% of EBA and 88.6% of FFA captions, versus 8.7% RSC and 5.3% PPA; PCV and TPJ show 80.8% and 87.8% mean person density.
- In a human study on 10 raters, EBA-1 images were judged to contain multiple people (e.g., 'multi person' 60-100%) more often than EBA-2 images, and EBA-1 was rated 88% for people per image on S1 versus 5% for EBA-2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BrainSCUBA-100 rather than BrainDiVE-100 for generated images | BrainSCUBA image synthesis · Zero-shot 5-way classification, Faces, Places, Bodies, Words, and Food | category match (%): S2 Mean: 78.4 -> 74.6; S5 Mean: 81.6 -> 72.0 | Mixed |
| Use BrainSCUBA-100 rather than NSD top-100 images | BrainSCUBA image synthesis · Zero-shot 5-way classification, Mean | category match (%): S2: 63.2 -> 74.6; S5: 61.4 -> 72.0 | Helped |
Surprising findings
- Captions generated without gradient-based maximization achieved predicted activations comparable to BrainDiVE's gradient-based image synthesis while being roughly 10x faster per image.
Limitations
- Relies on a pre-trained captioning model (CLIPCap/GPT-2), so outputs reflect its biases.
- Unclear whether the most selective object in each region can be fully captured by language.
- Evaluation of brain selectivity uses a limited set of four NSD subjects and largely qualitative or rule-based analysis.
Experimental setup
Datasets: Natural Scenes Dataset (NSD) LAION-A v2 (6+ subset) Open Images NSD functional localizers
Benchmarks: NSD top-100 zero-shot category classification NSD functional localizer regions
Metrics: R^2 zero-shot 5-way classification accuracy percentage of captions containing person nouns predicted activation human study agreement percentage
Baselines: BrainDiVE NSD top-100 real images
Builds on: CLIP CLIPCap BrainDiVE GLMSingle UMAP
Closest papers by meaning
- Finding Shared Decodable Concepts and their Negations in the Brain (ICLR 2025 · similarity 0.90)
- In Silico Mapping of Visual Categorical Selectivity Across the Whole Brain (NeurIPS 2025 · similarity 0.89)
- Brain Mapping with Dense Features: Grounding Cortical Semantic Selectivity in Natural Images With Vision Transformers (ICLR 2025 · similarity 0.89)
- Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction (NeurIPS 2024 · similarity 0.88)
- Meta-Learning an In-Context Transformer Model of Human Higher Visual Cortex (NeurIPS 2025 · similarity 0.87)
- CLIP-MSM: A Multi-Semantic Mapping Brain Representation for Human High-Level Visual Cortex (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.