Research map / Vision-language and multimodal / Human and embodied modeling
Finding Shared Decodable Concepts and their Negations in the Brain
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains per-participant linear fMRI-to-CLIP decoders with a contrastive loss and clusters their voxel weight vectors with a cross-participant DBSCAN variant to find Shared Decodable Concepts (SDCs) and their negations in the NSD dataset.
Problem
Existing functional localizer work identifies only a few broad visuo-semantic categories in the brain, and cross-participant analyses usually require warping to a template brain, losing individual anatomy. The paper asks whether data-driven methods can find new or refined visuo-semantic regions shar
Approach
Treat each voxel's decoder weight vector as a CLIP-space concept, cluster these vectors across participants with a modified DBSCAN that requires core points to have neighbors from several other participants, and interpret each cluster by its most and least associated images (the negated centroid).
Key results
- The contrastive linear decoder outperforms ridge regression in top-k accuracy across all k on held-out NSD data for the 8 participants.
- Clustering yields 9, 14, 12, and 11 cross-participant clusters for epsilon 0.5, 0.55, 0.6, and 0.65 respectively.
- The method recovers a face cluster localized to FFA and EBA, a horizontal/vertical orientation cluster in V2/V3, and a numerosity cluster in right IPS/OPA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the contrastive decoder instead of ridge regression | CLIP decoding from fMRI · NSD held-out data | top-k accuracy | Helped |
| Average decoder parameter vectors across 50 retrainings | Modified DBSCAN SDC discovery · NSD | cluster quality and consistency | Helped |
| Add within-participant cluster expansion (ablation) | Modified DBSCAN SDC discovery · NSD | cluster location and semantic interpretation | No clear effect |
Surprising findings
- The negative images for the face cluster often show bodies with faces obscured or turned away, suggesting FFA dips when a face is expected but not visible.
- A food-related cluster's negative images are entirely grayscale, suggesting it tracks color rather than food.
Limitations
- Results are limited to the NSD stimulus set and may be biased toward over-represented categories.
- CLIP embeddings may not capture all signal in brain responses, and fMRI has limited temporal resolution.
- The clustering may merge concepts in regions of uniform density in CLIP space and miss some SDCs.
Experimental setup
Datasets: Natural Scenes Dataset (NSD) COCO
Benchmarks: NSD held-out test fold top-k accuracy for CLIP decoding
Metrics: top-k accuracy cosine distance
Baselines: ridge regression
Builds on: CLIP DBSCAN InfoNCE
Closest papers by meaning
- BrainSCUBA: Fine-Grained Natural Language Captions of Visual Cortex Selectivity (ICLR 2024 · similarity 0.90)
- CLIP-MSM: A Multi-Semantic Mapping Brain Representation for Human High-Level Visual Cortex (AAAI 2025 · similarity 0.89)
- In Silico Mapping of Visual Categorical Selectivity Across the Whole Brain (NeurIPS 2025 · similarity 0.88)
- Bridging the Semantic Latent Space between Brain and Machine: Similarity Is All You Need (AAAI 2024 · similarity 0.88)
- Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction (NeurIPS 2024 · similarity 0.88)
- Brain Mapping with Dense Features: Grounding Cortical Semantic Selectivity in Natural Images With Vision Transformers (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.