Research map / Vision-language and multimodal / Human and embodied modeling
See Through Their Minds: Learning Transferable Brain Decoding Models from Cross-Subject fMRI
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STTM, a cross-subject fMRI brain decoding framework with shallow subject-specific adapters and a shared deep decoder, combining high-level CLIP-aligned decoding with high-level-guided pixel-wise reconstruction, and transfers to new subjects and imagery on GOD.
Problem
Brain decoding of visual content from fMRI is limited by scarce, noisy per-subject data, and subject-specific models generalize poorly across subjects and datasets.
Approach
Shallow subject-specific adapters map each subject's fMRI into a unified feature space, feeding a shared deep decoder that outputs CLIP embeddings (high-level) and Stable Diffusion VAE latents (low-level, guided by high-level features); new subjects are added by training new adapters with the shared modules frozen.
Key results
- On NSD, STTM-L (low-level pipeline with guidance) reaches PixCorr 0.383 and SSIM 0.488, outperforming MindEye2 (Low-level) on most metrics, though MindEye2 is pre-trained on 7 subjects.
- On GOD subject 3 zero-shot classification, STTM-H achieves 25.6% top-1 and 68.0% top-5 on visual stimuli, exceeding BrainCLIP-VAE (20.0 / 58.0) and CoOp (21.33 / 64.7), and reaches 12.0% top-1 on imagery decoding.
- Cross-subject training (Condition III) improves STTM-L PixCorr from 0.417 (single-subject) to 0.445 and STTM-H CLIP accuracy relative to single-subject training in ablations on NSD subject 1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the decoder with cross-subject fMRI data rather than single-subject data (ablation) | STTM-L · NSD subject 1 | PixCorr: .417 -> .445 | Helped |
| Train the decoder with cross-subject fMRI data rather than single-subject data (ablation) | STTM-H · NSD subject 1 | PixCorr: .150 -> .221 | Helped |
| Add high-level perception guidance to the low-level pipeline (ablation) | STTM-L · NSD subject 1 | PixCorr: .440 -> .445 | Helped |
| Add high-level perception guidance to the low-level pipeline (ablation) | STTM-L · NSD subject 1 | Alex(2): 84.5% -> 87.4% | Helped |
| Add global visual-linguistic contrastive learning (ablation) | STTM-H · NSD subject 1 | PixCorr: .214 -> .221 | Helped |
| Add global visual-linguistic contrastive learning (ablation) | STTM-H · NSD subject 1 | Eff: .606 -> .599 | Helped |
| Transfer the visually trained decoder to imagery-induced brain activity | STTM-H · GOD subject 3 zero-shot classification | top-1 accuracy: 25.6±1.96 -> 12.0±1.79 | Hurt |
| Use STTM rather than MindEye2 | NSD · NSD | PixCorr: .322 -> .333 | Helped |
| Use STTM rather than MindEye2 | NSD · NSD | Image retrieval accuracy: 98.8% -> 92.8% | Hurt |
| Use STTM rather than CMVDM | GOD subject 3 · GOD subject 3 stimulus reconstruction | CLIP: 82.0% -> 87.0% | Helped |
Surprising findings
- Decoders trained only on visually-evoked fMRI transfer to imagery-induced fMRI, reaching 12.0% top-1 zero-shot accuracy, though reduced.
- Shared/unified cross-subject training improved both high-level and low-level decoders in ablations.
Limitations
- Each subject requires a unique adapter during pre-training, limiting the number of subjects due to accelerator memory.
- Imagery decoding performance is reduced relative to visual stimulus decoding, and imagery reconstructions have no ground truth.
- Only four NSD subjects with full sessions are used for pretraining.
Experimental setup
Datasets: Natural Scenes Dataset (NSD) Generic Object Decoding (GOD) GOD-Cap COCO captions
Benchmarks: NSD GOD
Metrics: PixCorr SSIM AlexNet(2) AlexNet(5) Inception CLIP similarity EfficientNet-B1 distance SwAV distance top-1 image retrieval accuracy top-5 text retrieval accuracy
Baselines: MindEye MindEye2 Brain-diffuser BrainCLIP Mind-Reader MinD-Vis CMVDM IC-GAN BraVL MindBridge DREAM PSYCHOMETRY
Builds on: MindEye BrainCLIP Versatile Diffusion Stable Diffusion DALL-E 2 diffusion prior BiMixCo SoftCLIP
Closest papers by meaning
- Toward Generalizing Visual Brain Decoding to Unseen Subjects (ICLR 2025 · similarity 0.92)
- Brain decoding: toward real-time reconstruction of visual perception (ICLR 2024 · similarity 0.91)
- MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data (ICML 2024 · similarity 0.91)
- MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited fMRI Data (ICML 2025 · similarity 0.91)
- MoRE-Brain: Routed Mixture of Experts for Interpretable and Generalizable Cross-Subject fMRI Visual Decoding (NeurIPS 2025 · similarity 0.90)
- $i$MIND: Insightful Multi-subject Invariant Neural Decoding (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.