Research map / Vision-language and multimodal / Human and embodied modeling
MindSimulator: Exploring Brain Concept Localization via Synthetic fMRI
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MindSimulator, a diffusion-based generative fMRI encoding model trained on NSD that synthesizes fMRI for concept-oriented images to localize concept-selective brain regions.
Problem
Localizing concept-selective cortical regions via fLoc experiments is limited by scarce fMRI data, biased and unnatural stimuli, and restricted concept categories.
Approach
A generative fMRI encoder: an fMRI autoencoder aligned to CLIP image space via SoftCLIP, a Transformer diffusion estimator that predicts fMRI latents conditioned on image tokens, and an inference sampler using multi-trial averaging with correlated Gaussian noise; synthetic fMRI on MSCOCO images is then used for statistical region localization.
Key results
- MindSimulator (Trials=5) reaches Pearson 0.355 and MSE 0.385 averaged over 4 subjects, beating linear regression (0.334/0.394) and Transformer encoding (0.337/0.387) on voxel-level metrics.
- For bodies-selective region localization (Subj01, top 100 images), MindSimulator attains 91.7% accuracy and 0.674 F1 versus 45.8% and 0.570 for linear regression.
- On places-selective localization, MindSimulator improves accuracy from 32.0% to 52.6% over linear regression for top 100 images (Subj01).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use five trials instead of one for multi-trial averaging (ablation) | MindSimulator · NSD test (4-subject average) | MSE: 0.404 -> 0.385 | Helped |
| Use five trials instead of one for multi-trial averaging (ablation) | MindSimulator · NSD test (4-subject average) | CLIP: 90.3% -> 91.2% | Helped |
| Add the Voxel Decoder (ablation) | MindSimulator · NSD test (Subj01) | Pearson: 0.287 -> 0.322 | Helped |
| Add the joint fMRI-image latent space (ablation) | MindSimulator · NSD test (Subj01) | Pearson: 0.215 -> 0.322 | Helped |
| Add the fMRI Autoencoder (ablation) | MindSimulator · NSD test (Subj01) | Pearson: 0.203 -> 0.322 | Helped |
| Add correlated noise (ablation) | MindSimulator · NSD test (Subj01) | Pearson: 0.316 -> 0.322 | Helped |
| Increase residual blocks from 2 to 4 (ablation) | fMRI Autoencoder (hidden dim 256, end-to-end) · Autoencoder reconstruction (voxel input) | MSE: 0.227 -> 0.240 | Hurt |
| Increase hidden dimension from 256 to 512 (ablation) | fMRI Autoencoder (2 blocks, end-to-end) · Autoencoder reconstruction (voxel input) | MSE: 0.227 -> 0.252 | Hurt |
| Disable end-to-end training (ablation) | fMRI Autoencoder (hidden dim 256, 2 blocks) · Autoencoder reconstruction (voxel input) | MSE: 0.227 -> 0.401 | Hurt |
Surprising findings
- Multi-trial averaging improved both voxel-level MSE and semantic metrics, contrary to the expectation that higher reproducibility dilutes global pattern accuracy; the authors attribute this to correlated noise.
- Localization accuracy for places-selective regions falls as more images are selected (e.g., 52.6% at top 100 to 33.5% at top 1000).
Limitations
- Decoded synthetic fMRI is noisy, and the decoder was trained on ground-truth fMRI, so it struggles with synthetic inputs.
- Localized novel regions are validated only by voxel ablation and qualitative comparison, not by neuroscience experiments.
- Evaluation is limited to four NSD subjects and a single fMRI dataset.
Experimental setup
Datasets: Natural Scenes Dataset (NSD) MSCOCO CIFAR-10 CIFAR-100
Benchmarks: NSD fLoc empirical regions CIFAR-10 CIFAR-100
Metrics: Pearson MSE PixCorr SSIM Alex(2) Alex(5) Incep CLIP Eff SwAV
Baselines: Linear Regression encoding Transformer encoding model
Builds on: SoftCLIP fLoc MindEye2 GLMSingle
Closest papers by meaning
- In Silico Mapping of Visual Categorical Selectivity Across the Whole Brain (NeurIPS 2025 · similarity 0.87)
- Flexible Context-Driven Sensory Processing in Dynamical Vision Models (NeurIPS 2024 · similarity 0.87)
- NEvo: Neural-Guided Evolutionary Video Synthesis for Dynamic Visual Selectivity (NeurIPS 2026 · similarity 0.86)
- A Local-Ascending-Global Learning Strategy for Brain-Computer Interface (AAAI 2024 · similarity 0.86)
- SIM: Surface-based fMRI Analysis for Inter-Subject Multimodal Decoding from Movie-Watching Experiments (ICLR 2025 · similarity 0.85)
- Finding Shared Decodable Concepts and their Negations in the Brain (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.