Research map / Vision-language and multimodal / Human and embodied modeling
Brain Mapping with Dense Features: Grounding Cortical Semantic Selectivity in Natural Images With Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BrainSAIL, a learning-free method that distills dense CLIP/DINOv2/SigLIP patch features via augmentation averaging and maps voxel-wise fMRI encoder weights onto them to localize which image regions drive category, scene, and low-level selectivity in NSD.
Problem
Existing brain-encoding analyses treat whole images as stimuli, so it is unclear which spatially distributed semantic components of complex natural scenes drive selectivity in higher visual cortex; dense ViT features from contrastive models are also artifact-laden.
Approach
Average dense patch features across augmentations (shifts and flips) without training, which the authors show equals the MSE-optimal embedding, then apply frozen voxel-wise linear adapters to these clean dense features to produce spatial relevance maps for brain voxels.
Key results
- Learning-free distillation improves NACLIP open-vocabulary segmentation: ADE20k mIoU 17.69 to 18.19 and COCO Object 33.14 to 34.15 (zero-shot).
- CLIP text-alignment on NSD assigns the correct category to 76% (places), 50% (bodies), 46% (faces) of top-100 images for the matching ROI under the paper's category caption sets (S1).
- Distilling takes under 0.2 s per image on GPU, about 100x faster than Denoising Vision Transformers per the authors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add learning-free feature distillation to NACLIP dense features (ablation) | NACLIP with CLIP · ADE20k | mIoU: 17.69 -> 18.19 | Helped |
| Add learning-free feature distillation to NACLIP dense features (ablation) | NACLIP with CLIP · ADE20k | Pearson: 0.425 -> 0.443 | Helped |
| Add learning-free feature distillation to NACLIP dense features (ablation) | NACLIP with CLIP · COCO Object | mIoU: 33.14 -> 34.15 | Helped |
| Add learning-free feature distillation to NACLIP dense features (ablation) | NACLIP with CLIP · COCO Object | Pearson: 0.418 -> 0.435 | Helped |
| Add learning-free feature distillation to NACLIP dense features (ablation) | NACLIP with CLIP · COCO Stuff | mIoU: 22.58 -> 23.08 | Helped |
| Add learning-free feature distillation to NACLIP dense features (ablation) | NACLIP with CLIP · COCO Stuff | Pearson: 0.393 -> 0.405 | Helped |
| Add learning-free feature distillation to NACLIP dense features (ablation) | NACLIP with CLIP · VOC20 | mIoU: 77.09 -> 79.09 | Helped |
| Add learning-free feature distillation to NACLIP dense features (ablation) | NACLIP with CLIP · VOC20 | Pearson: 0.473 -> 0.489 | Helped |
Surprising findings
- CLIP and SigLIP relevance maps agree more with each other than with DINO, which shows greater sensitivity to low-level color and texture rather than semantic parts.
- Neural data does not consistently separate faces and bodies, and food-region voxels attribute selectivity to small or high-saturation objects.
Limitations
- Encoders still need to be trained on NSD, so dataset biases can shape the learned representations.
- Results are limited to four NSD subjects, and food localizer data was not independently available for these subjects.
- Relevance maps depend on hand-written caption sets for category alignment.
Experimental setup
Datasets: Natural Scenes Dataset (NSD) ADE20k COCO Object COCO Stuff VOC20
Benchmarks: ADE20k COCO Object COCO Stuff VOC20 NSD test set
Metrics: mIoU Pearson correlation R^2 percent category assignment
Baselines: SCLIP NACLIP MaskCLIP Denoising Vision Transformers
Builds on: MaskCLIP NACLIP SCLIP Denoising Vision Transformers BrainSCUBA
Closest papers by meaning
- In Silico Mapping of Visual Categorical Selectivity Across the Whole Brain (NeurIPS 2025 · similarity 0.90)
- BrainSCUBA: Fine-Grained Natural Language Captions of Visual Cortex Selectivity (ICLR 2024 · similarity 0.89)
- Finding Shared Decodable Concepts and their Negations in the Brain (ICLR 2025 · similarity 0.88)
- CLIP-MSM: A Multi-Semantic Mapping Brain Representation for Human High-Level Visual Cortex (AAAI 2025 · similarity 0.88)
- Transformer brain encoders explain human high-level visual responses (NeurIPS 2025 · similarity 0.87)
- Flexible Context-Driven Sensory Processing in Dynamical Vision Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.