Research map / Vision-language and multimodal / Human and embodied modeling
CLIP-MSM: A Multi-Semantic Mapping Brain Representation for Human High-Level Visual Cortex
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds CLIP-based voxel-wise encoding models of human high-level visual cortex and applies CLIP Dissection with a softmax-normalized SoftWPMI score to map each voxel to multiple semantic labels (faces, bodies, places, words, food).
Problem
Existing interpretable encoding models assign each voxel of human high-level visual cortex only one category label, ignoring that multiple selectivities coexist within voxels; hypothesis-free multi-semantic mapping is lacking.
Approach
Apply CLIP Dissection (training-free, label-agnostic, CLIP-based) to voxel-wise CLIP encoding models and normalize the SoftWPMI semantic-mapping scores with a column-wise softmax so each voxel receives a graded distribution over multiple semantic labels.
Key results
- CLIP-based encoding models (ResNet50CLIP, ViT-32CLIP) predict NSD subject S5 voxel responses better than ImageNet-pretrained ResNet50 and AlexNet (P < 0.001, paired t-test).
- CLIP-MSM yields higher cosine similarity between label embeddings and voxel weights than Network Dissection in most selective regions for NSD subject S5.
- CLIP-MSM shows significantly higher Pearson correlation with floc ground truth than CLIP Dissection for places, food, faces, bodies, and words in NSD, and for faces, bodies, and words in NOD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CLIP image embeddings instead of ImageNet-pretrained ResNet50 or AlexNet features | Voxel-wise encoding models · NSD | R² | Helped |
| Use CLIP Dissection instead of Network Dissection | CLIP-MSM semantic mapping · NSD selective regions | cosine similarity | Helped |
| Use CLIP-MSM instead of CLIP Dissection with maximum semantic mapping scores | ResNet50CLIP · NSD: places, food, faces, bodies, and words | Pearson correlation coefficient | Helped |
| Use CLIP-MSM instead of CLIP Dissection with maximum semantic mapping scores | ResNet50CLIP · NOD: faces, bodies, and words | Pearson correlation coefficient | Helped |
Limitations
- Analysis focused on high-level visual cortex and five categories (faces, bodies, places, words, food).
- Ground-truth comparisons rely on floc localizer data and a subset of shared images for food, with limited subjects in the main analyses.
- Full subject results are only in the appendix; main results shown for a single subject per dataset.
Experimental setup
Datasets: Natural Scenes Dataset (NSD) Natural Object Dataset (NOD) Broden
Benchmarks: Natural Scenes Dataset (NSD) Natural Object Dataset (NOD)
Metrics: R2 (coefficient of determination) cosine similarity between label text embeddings and voxel we Pearson correlation with floc ground-truth responses noise ceiling
Baselines: Network Dissection CLIP Dissection ImageNet-pretrained ResNet50 AlexNet
Builds on: CLIP Dissection Network Dissection CLIP
Closest papers by meaning
- CLIP-MUSED: CLIP-Guided Multi-Subject Visual Neural Information Semantic Decoding (ICLR 2024 · similarity 0.90)
- Finding Shared Decodable Concepts and their Negations in the Brain (ICLR 2025 · similarity 0.89)
- In Silico Mapping of Visual Categorical Selectivity Across the Whole Brain (NeurIPS 2025 · similarity 0.88)
- Brain Mapping with Dense Features: Grounding Cortical Semantic Selectivity in Natural Images With Vision Transformers (ICLR 2025 · similarity 0.88)
- Multigranular Evaluation for Brain Visual Decoding (AAAI 2026 · similarity 0.88)
- Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.