Research map / Image generation / Fast diffusion sampling
Controllable Mind Visual Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CMVDM, which adds a ControlNet-style control model with silhouette and residual conditions to a fMRI-finetuned latent diffusion model to reconstruct images with better semantics and silhouette from fMRI.
Problem
Reconstructing visual stimuli from fMRI with diffusion models yields photo-realistic images that often mismatch the stimulus in semantics and spatial layout (silhouette/position).
Approach
Combine a semantic alignment loss (CLIP image features) on the fMRI encoder with a silhouette decoder pretrained on fMRI-to-image and self-supervised ImageNet reconstruction, and feed the silhouette and a residual fMRI branch as ControlNet conditions into a frozen LDM.
Key results
- On GOD, CMVDM reaches 30.112% Acc, 0.76751 PCC, 0.63167 SSIM, versus MinD-Vis at 26.644%, 0.53159, 0.52669.
- On BOLD5000, CMVDM reaches 27.791% Acc, 0.55691 PCC, 0.53459 SSIM, versus MinD-Vis at 25.918%, 0.54486, 0.52379.
- Ablation: adding the control model to MinD-Vis raises PCC from 0.53159 to 0.75730 on GOD, the largest single gain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CMVDM instead of MinD-Vis | fMRI-to-image reconstruction · GOD | Acc (%): 26.644 -> 30.112 | Helped |
| Use CMVDM instead of MinD-Vis | fMRI-to-image reconstruction · GOD | PCC: 0.53159 -> 0.76751 | Helped |
| Use CMVDM instead of Gaziv | fMRI-to-image reconstruction · GOD | SSIM: 0.64857 -> 0.63167 | Hurt |
| Use CMVDM instead of MinD-Vis | fMRI-to-image reconstruction · BOLD5000 | SSIM: 0.52379 -> 0.53459 | Helped |
| Add semantic alignment loss (ablation) | MinD-Vis · GOD | Acc (%): 26.644 -> 27.362 | Helped |
| Add semantic alignment loss (ablation) | MinD-Vis · GOD | SSIM: 0.54489 -> 0.52628 | Hurt |
| Add control model (ablation) | MinD-Vis · GOD | SSIM: 0.54489 -> 0.63404 | Helped |
| Add control model instead of only semantic alignment loss (ablation) | MinD-Vis · GOD | PCC: 0.56686 -> 0.75730 | Helped |
| Add residual block (ablation) | CMVDM · BOLD5000 | Acc (%): 25.393 -> 27.791 | Helped |
| Add residual block (ablation) | CMVDM · BOLD5000 | PCC: 0.54184 -> 0.55691 | Helped |
| Add residual block (ablation) | CMVDM · GOD | SSIM: 0.63894 -> 0.63167 | Hurt |
| Add residual block (ablation) | CMVDM · GOD | PCC: 0.75837 -> 0.76751 | Helped |
Surprising findings
- Without the residual block, CMVDM achieves higher SSIM on GOD (0.63894 vs 0.63167), so the residual module does not help on every metric.
Limitations
- Performance differs substantially between GOD and BOLD5000, with weaker silhouette extraction on BOLD5000 due to fewer voxels and more complex scenes.
- Evaluated only on two small fMRI datasets, so generalization to larger paired data is unverified.
- Cross-domain generalization and robustness across fMRI acquisition settings remain unexplored.
Experimental setup
Datasets: GOD BOLD5000 HCP ImageNet
Benchmarks: GOD BOLD5000
Metrics: N-way top-1 accuracy Pearson correlation coefficient SSIM
Baselines: MinD-Vis IC-GAN Gaziv Beliy
Builds on: MinD-Vis ControlNet latent diffusion model CLIP
Closest papers by meaning
- NSD-Imagery: A Benchmark Dataset for Extending fMRI Vision Decoding Methods to Mental Imagery (CVPR 2025 · similarity 0.85)
- MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data (ICML 2024 · similarity 0.85)
- In Silico Mapping of Visual Categorical Selectivity Across the Whole Brain (NeurIPS 2025 · similarity 0.84)
- DEEM: Diffusion models serve as the eyes of large language models for image perception (ICLR 2025 · similarity 0.84)
- CLIP-MSM: A Multi-Semantic Mapping Brain Representation for Human High-Level Visual Cortex (AAAI 2025 · similarity 0.84)
- SIM: Surface-based fMRI Analysis for Inter-Subject Multimodal Decoding from Movie-Watching Experiments (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.