Research map / Image generation / Autoregressive and latent generation
Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors
Paul Steven Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Cohen Ethan, Aidan James Dempster, Nathalie Verlinde, Elad Yundler, David Weisberg, Kenneth Norman, Tanishq Mathew Abraham
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add diffusion prior to MLP backbone + projector (ablation) | MindEye · NSD Subject 1 | CLIP two-way identification: 77.3% -> 95.0% | Improved |
| add separate MLP projector for contrastive loss (ablation) | MindEye retrieval · NSD Subject 1 | image retrieval: 0.934 -> 0.972 | Improved |
| add projector (decoupling losses) to backbone-only model (ablation) | MLP backbone · NSD Subject 1 | image retrieval: 0.133 -> 0.888 | Improved |
| replace InfoNCE with bidirectional CLIP loss (ablation) | MindEye retrieval · NSD Subject 1 | image retrieval: 0.237 -> 0.837 | Improved |
| add MixCo augmentation to CLIP loss (BiMixCo) (ablation) | CLIP loss · NSD Subject 1 | image retrieval: 0.837 -> 0.884 | Improved |
| combine BiMixCo with SoftCLIP (ablation) | BiMixCo · NSD Subject 1 | image retrieval: 0.884 -> 0.896 | Improved |
| switch from BiMixCo to SoftCLIP after one-third of training (ablation) | MindEye · NSD Subject 1 | CLIP two-way identification: 94.4% -> 95.0% | Improved |
| add skip connections to 4-block MLP (ablation) | MLP backbone · NSD Subject 1 | image retrieval: 0.836 -> 0.896 | Improved |
| map to all 257 CLIP tokens instead of CLS only (7x more parameters) (ablation) | MLP backbone · NSD Subject 1 | image retrieval: 0.611 -> 0.896 | Improved |
| add residual blocks (0 to 4) to MLP backbone (ablation) | MLP backbone · NSD Subject 1 | image retrieval: 0.880 -> 0.896 | Improved |
| use high-level pipeline only (drop low-level img2img init) (ablation) | MindEye · NSD test | PixCorr: .309 -> .194 | Hurt |
| MindEye vs Ozcelik et al. on SSIM | MindEye · NSD test | SSIM: .356 -> .323 | Hurt |
Closest papers by meaning
- MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data (ICML 2024 · similarity 0.94)
- NSD-Imagery: A Benchmark Dataset for Extending fMRI Vision Decoding Methods to Mental Imagery (CVPR 2025 · similarity 0.90)
- MindBridge: A Cross-Subject Brain Decoding Framework (CVPR 2024 · similarity 0.88)
- Mind Artist: Creating Artistic Snapshots with Human Thought (CVPR 2024 · similarity 0.88)
- Brain decoding: toward real-time reconstruction of visual perception (ICLR 2024 · similarity 0.88)
- See Through Their Minds: Learning Transferable Brain Decoding Models from Cross-Subject fMRI (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
