Research map / Vision-language and multimodal / Human and embodied modeling
Toward Generalizing Visual Brain Decoding to Unseen Subjects
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that a single unified fMRI-to-CLIP retrieval network trained on whole-brain HCP movie-viewing data generalizes to unseen subjects, with accuracy rising as the number of training subjects grows (up to 167).
Problem
Visual brain decoding models are subject-specific and typically use per-subject heads or tokenizers on few subjects, so it is unclear whether decoding generalizes to unseen individuals.
Approach
Build an image-fMRI dataset from HCP movie-viewing (177 subjects) and train one subject-agnostic network on whole-brain fMRI (voxels upsampled to a common size) with CLIP contrastive alignment, with no subject-specific heads; study how generalization scales with subject count, architecture, and subject similarity.
Key results
- Top-1 retrieval accuracy on unseen subjects rises from 2% (1 training subject) to 45% (167 training subjects) on 100 test pairs; 50% with BiMixCo+SoftCLIP.
- With 167 training subjects, MLP, 1D CNN, 3D CNN and Transformer backbones reach 45%, 42%, 40% and 34% top-1 on unseen subjects.
- Models trained on 20 most similar subjects reach 21% top-1 on Subj 1 versus 8% for 20 most dissimilar subjects.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase training subjects from 1 to 167 | HCP MLP visual brain decoder · HCP unseen subjects (Subjs 1-10) | TOP1 Acc.: 2% -> 45% | Helped |
| Use 1D CNN instead of MLP backbone (ablation) | HCP visual brain decoder trained on 167 subjects · HCP unseen subjects (Subjs 1-10) | TOP1 Acc.: 45% -> 42% | Hurt |
| Use 3D CNN instead of MLP backbone (ablation) | HCP visual brain decoder trained on 167 subjects · HCP unseen subjects (Subjs 1-10) | TOP1 Acc.: 45% -> 40% | Hurt |
| Use Transformer instead of MLP backbone (ablation) | HCP visual brain decoder trained on 167 subjects · HCP unseen subjects (Subjs 1-10) | TOP1 Acc.: 45% -> 34% | Hurt |
| Add BiMixCo+SoftCLIP training (ablation) | HCP MLP visual brain decoder trained on 1 subject · HCP unseen subjects (Subjs 1-10) | TOP1 Acc.: 2% -> 6% | Helped |
| Add BiMixCo+SoftCLIP training (ablation) | HCP MLP visual brain decoder trained on 167 subjects · HCP unseen subjects (Subjs 1-10) | TOP1 Acc.: 45% -> 50% | Helped |
| Use the proposed normalized whole-brain data format instead of MindEye1's original format (ablation) | NSD individual-subject decoding · NSD Subj 1 | TOP1 Acc.: 35% -> 46% | Helped |
| Use the proposed normalized whole-brain data format instead of MindEye1's original format (ablation) | NSD individual-subject decoding · NSD Subj 7 | TOP1 Acc.: 23% -> 29% | Helped |
| Train on six subjects instead of one (ablation) | NSD whole-brain decoder · NSD Subj 1 | TOP1 Acc.: 46% -> 49% | Helped |
| Train on six subjects instead of one (ablation) | NSD whole-brain decoder · NSD Subj 7 | TOP1 Acc.: 29% -> 35% | Helped |
| Train on six NSD subjects and test on unseen subjects (ablation) | NSD decoder · NSD unseen Subjs 2,5 | TOP1 Acc.: 1% | Hurt |
| Train on 50 males rather than 50 females (ablation) | HCP gender-group decoder · Unseen male Subj 1 | TOP1 Acc.: 27% -> 36% | Helped |
Surprising findings
- Training on only 20 highly dissimilar subjects yields almost no generalization (2% top-1 on Subj 2), and a gender-split training set shows a clear bias on unseen subjects.
- Multi-subject whole-brain training on NSD (6 subjects) slightly outperforms single-subject models on whole-brain data, while on NSDGeneral data multi-subject models are slightly worse.
Limitations
- Only 8 subjects in NSD is too few for generalization to unseen subjects, so the NSD results show weak generalization.
- Retrieval only; reconstruction and grounding are not evaluated.
- Small test set of 100 image-fMRI pairs from 4 films limits the evaluation.
Experimental setup
Datasets: HCP movie-viewing fMRI (177 subjects) NSD
Benchmarks: HCP movie-viewing image-fMRI retrieval (100 test pairs, unseen subjects 1-10) NSD retrieval (1,000 test images, 300-pair subsets)
Metrics: top-1 accuracy top-3 accuracy
Baselines: MindEye1 MindEye2 UMBRAE CLIP-MUSED
Builds on: MindEye1 CLIP BiMixCo SoftCLIP
Closest papers by meaning
- See Through Their Minds: Learning Transferable Brain Decoding Models from Cross-Subject fMRI (AAAI 2025 · similarity 0.92)
- MoRE-Brain: Routed Mixture of Experts for Interpretable and Generalizable Cross-Subject fMRI Visual Decoding (NeurIPS 2025 · similarity 0.91)
- Brain decoding: toward real-time reconstruction of visual perception (ICLR 2024 · similarity 0.90)
- $i$MIND: Insightful Multi-subject Invariant Neural Decoding (NeurIPS 2025 · similarity 0.90)
- ZEBRA: Towards Zero-Shot Cross-Subject Generalization for Universal Brain Visual Decoding (NeurIPS 2025 · similarity 0.89)
- SIM: Surface-based fMRI Analysis for Inter-Subject Multimodal Decoding from Movie-Watching Experiments (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.