Research map / Vision-language and multimodal / Human and embodied modeling
CLIP-MUSED: CLIP-Guided Multi-Subject Visual Neural Information Semantic Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLIP-MUSED, a Transformer-based multi-subject fMRI semantic decoder with learnable low-level and high-level subject tokens, guided by RSA to CLIP stimulus similarity structure.
Problem
Decoding visual semantic information from fMRI across multiple subjects is hard due to individual differences in functional topography, limited per-subject data, and parameter growth with subject count; existing multi-subject methods have restricted mapping expressiveness and do not capture cross-su
Approach
Use learnable subject-specific low-level and high-level tokens in a shared Transformer fMRI encoder, with RSA losses aligning token-representation similarity matrices to CLIP stimulus similarity matrices (shallow layer for low-level, multimodal deep feature for high-level) plus an orthogonality constraint between the two token types, so parameters scale only with added tokens.
Key results
- On NSD, CLIP-MUSED reaches mAP .258, AUC .877, Hamming .030, versus best multi-subject baseline MS-EMB at mAP .220, AUC .825.
- On HCP, CLIP-MUSED reaches mAP .373 and AUC .581, outperforming MS-SMODEL-ViT (.367 mAP) and MS-EMB (.368 mAP), and single-subject SS-ViT (.360 mAP).
- Ablation on NSD shows removing either the RSA guidance or the orthogonal constraint substantially lowers mAP (best without all three terms: .201 to .215 vs .258).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multi-subject CLIP-MUSED instead of MS-EMB | HCP decoding · HCP | mAP: .368\pm.004 -> .373\pm.002 | Helped |
| Use multi-subject CLIP-MUSED instead of SS-MLP | NSD decoding · NSD | AUC: .854\pm.004 -> .877\pm.021 | Helped |
| Add the orthogonal constraint to low- and high-level RSA guidance (ablation) | CLIP-MUSED · NSD | mAP: .201\pm.006 -> .258\pm.017 | Helped |
| Add low- and high-level RSA guidance to the orthogonal constraint (ablation) | CLIP-MUSED · NSD | mAP: .209\pm.002 -> .258\pm.017 | Helped |
| Add high-level RSA guidance to orthogonal constraint plus low-level guidance (ablation) | CLIP-MUSED · NSD | mAP: .215\pm.012 -> .258\pm.017 | Helped |
| Add low-level RSA guidance to orthogonal constraint plus high-level guidance (ablation) | CLIP-MUSED · NSD | mAP: .209\pm.016 -> .258\pm.017 | Helped |
| Use RSA loss instead of mapping-based loss (ablation) | CLIP-MUSED · NSD | mAP: .247\pm.026 -> .258\pm.017 | Helped |
| Apply CLIP guidance to the single-subject SS-ViT (ablation) | SS-ViT · NSD | mAP: .238\pm.005 -> .234\pm.002 | Hurt |
| Use multimodal CLIP features rather than CLIP image-only guidance (ablation) | CLIP-MUSED · NSD | mAP: .238\pm.006 -> .258\pm.017 | Helped |
| Use multimodal CLIP features rather than CLIP text-only guidance (ablation) | CLIP-MUSED · NSD | mAP: .223\pm.006 -> .258\pm.017 | Helped |
Surprising findings
- On NSD, the single-subject SS-MLP (mAP .258) matches CLIP-MUSED's mAP, with CLIP-MUSED winning mainly on AUC and Hamming.
- CLIP guidance applied to the single-subject SS-ViT (CLIP-SS-ViT) does not improve over SS-ViT in mAP (.234 vs .238), suggesting the guidance benefit comes mainly in the multi-subject setting.
Limitations
- Evaluated only on semantic classification with two fMRI datasets; visual reconstruction is left to future work.
- Only 9 HCP subjects used out of 158; generalization to new subjects not tested (requires learning new subject tokens).
- Substantial variance on NSD (e.g., mAP .258 ± .017) and single-subject SS-MLP is comparable on mAP.
Experimental setup
Datasets: HCP (Human Connectome Project movie fMRI) NSD (Natural Scenes Dataset) MSCOCO WordNet labels
Benchmarks: HCP NSD
Metrics: mAP AUC Hamming distance
Baselines: SS-CNN SS-ViT SS-MLP MS-SMODEL-CNN MS-SMODEL-ViT MS-SMODEL-MLP MS-EMB SRM
Builds on: ViT CLIP hyperalignment shared response model
Closest papers by meaning
- CLIP-MSM: A Multi-Semantic Mapping Brain Representation for Human High-Level Visual Cortex (AAAI 2025 · similarity 0.90)
- $i$MIND: Insightful Multi-subject Invariant Neural Decoding (NeurIPS 2025 · similarity 0.89)
- MindTuner: Cross-Subject Visual Decoding with Visual Fingerprint and Semantic Correction (AAAI 2025 · similarity 0.89)
- See Through Their Minds: Learning Transferable Brain Decoding Models from Cross-Subject fMRI (AAAI 2025 · similarity 0.88)
- Multigranular Evaluation for Brain Visual Decoding (AAAI 2026 · similarity 0.88)
- Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.