Research map / Image generation / Autoregressive and latent generation
MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
MindEye2 pretrains a shared-subject fMRI-to-image model across 7 NSD subjects, then fine-tunes on 1 hour of a new subject's data, mapping fMRI to OpenCLIP ViT-bigG/14 and reconstructing images with a fine-tuned SDXL unCLIP model.
Problem
Brain-to-image reconstruction models are per-subject and need dozens of hours of fMRI data per person, limiting practical use.
Approach
A shared-subject functional alignment (subject-specific linear ridge layers into a shared 4096-dim latent, then a shared MLP backbone and diffusion prior) pretrained across subjects and fine-tuned on a held-out subject, with the CLIP-to-pixel decoder being SDXL fine-tuned to accept CLIP image tokens instead of text, plus retrieval and low-level submodules.
Key results
- With full 40-session data, MindEye2 reports SOTA on nearly all reconstruction and retrieval metrics (e.g., PixCorr 0.322, SSIM 0.431, image retrieval 98.8% and brain retrieval 98.3% out of 300).
- Fine-tuned on only 1 hour of subject 1 data, MindEye2 reaches 79.0% image retrieval and 84.2% AlexNet(2) two-way identification, approaching the 40-hour single-subject MindEye1 level in many metrics.
- Human raters preferred refined reconstructions 71.94% of the time over unrefined ones.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the initial MLP with dropout by a linear voxel-to-latent mapping with L2 regularization (ablation) | MindEye2 · NSD test set, subject 1, 10 sessions | PixCorr: 0.225 -> 0.292 | Helped |
| Map fMRI to OpenCLIP ViT-bigG/14 instead of CLIP ViT-L/14 (ablation) | MindEye2 · NSD test set, subject 1, 10 sessions | PixCorr: 0.243 -> 0.292 | Helped |
| Add the low-level submodule to the prior-only model (ablation) | MindEye2 · NSD test set, reduced-capacity ablation | PixCorr: 0.155 -> 0.281 | Helped |
| Add the retrieval submodule to the prior-only model (ablation) | MindEye2 · NSD test set, reduced-capacity ablation | PixCorr: 0.155 -> 0.233 | Helped |
| Add the prior and low-level submodules to the retrieval-only model (ablation) | MindEye2 · NSD test set, reduced-capacity ablation | Fwd retrieval: 96.5% -> 98.0% | Helped |
| Fine-tune after multi-subject pretraining using 1 hour of target-subject data | MindEye2 · NSD test set | Image retrieval | Helped |
| Use predicted brain captions rather than empty text prompts for refinement (ablation) | MindEye2 · NSD test set, subject 1, 1 session | Inception two-way identification: 82.67% -> 83.56% | Helped |
| Use predicted brain captions rather than empty text prompts for refinement (ablation) | MindEye2 · NSD test set, subject 1, 1 session | CLIP two-way identification: 81.21% -> 80.75% | Hurt |
| Reduce pretraining from the full available subject set to subjects 2, 5, and 7 (ablation) | MindEye2 fine-tuned on subject 1 with 1 hour · NSD test set, subject 1 | PixCorr: 0.235 -> 0.234 | No clear effect |
| Skip base SDXL refinement (ablation) | MindEye2 · NSD test set | Image reconstruction quality | Mixed |
| Replace weighted averaging with VAE-embedding conditioning (ablation) | MindEye2 · NSD test set | Low-level image metrics | Hurt |
Surprising findings
- Unrefined MindEye2 reconstructions scored better on several high-level metrics and brain correlation despite looking visibly distorted, suggesting standard metrics may not reflect subjective quality.
- Non-pretrained single-subject models trained on 10 or 30 minutes of data showed instability and possible mode collapse, with SSIM rising for nonsensical outputs.
Limitations
- Decoding is sensitive to head movement and subject compliance with the task.
- Only shown to work on natural scenes like COCO; other image distributions would need more data or specialized generators.
- OpenCLIP bigG embeddings retrieve less well than CLIP L for large image pools.
Experimental setup
Datasets: Natural Scenes Dataset COCO 2017 LAION-5B
Benchmarks: Natural Scenes Dataset shared1000 test set COCO 2017 (FID for unCLIP)
Metrics: PixCorr SSIM AlexNet two-way identification Inception CLIP two-way identification EfficientNet-B1 distance SwAV distance top-1 retrieval accuracy brain correlation caption metrics (METEOR, ROUGE, CLIP, Sentence)
Baselines: MindEye1 Brain Diffuser Takagi et al. UniBrain Versatile Diffusion
Builds on: MindEye1 Stable Diffusion XL OpenCLIP ViT-bigG/14 DALL-E 2 diffusion prior
Closest papers by meaning
- See Through Their Minds: Learning Transferable Brain Decoding Models from Cross-Subject fMRI (AAAI 2025 · similarity 0.91)
- MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited fMRI Data (ICML 2025 · similarity 0.90)
- NSD-Imagery: A Benchmark Dataset for Extending fMRI Vision Decoding Methods to Mental Imagery (CVPR 2025 · similarity 0.89)
- MindTuner: Cross-Subject Visual Decoding with Visual Fingerprint and Semantic Correction (AAAI 2025 · similarity 0.89)
- MindBridge: A Cross-Subject Brain Decoding Framework (CVPR 2024 · similarity 0.89)
- SIM: Surface-based fMRI Analysis for Inter-Subject Multimodal Decoding from Movie-Watching Experiments (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.