Research map / Vision-language and multimodal / Human and embodied modeling
SynBrain: Enhancing Visual-to-fMRI Synthesis via Probabilistic Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SynBrain, a probabilistic generative framework (BrainVAE plus a one-step Semantic-to-Neural mapper) that synthesizes subject-specific and few-shot cross-subject fMRI responses from visual stimuli, and uses the synthetic fMRI as augmentation for fMRI-to-image decoding.
Problem
Visual-to-fMRI encoding is inherently one-to-many (trial, attention, and subject variability), but existing regression and deterministic generative methods (e.g., MindSimulator) fail to model this variability while keeping functional consistency, and synthetic fMRI has limited cross-subject transfer
Approach
Model fMRI responses as semantic-conditioned continuous latent distributions with a conv+attention VAE (BrainVAE) trained with MSE, KL and SoftCLIP losses, and map frozen CLIP embeddings to the latent center in one step with a Transformer (S2N Mapper) instead of multi-step diffusion.
Key results
- Subject-specific synthesis: SynBrain (1 trial) reaches MSE 0.139 and Pearson 0.687 vs MindSimulator (5 trials) 0.385 and 0.357 on the reported 4-subject average (text reports a different MSE for ablation subject 1).
- Few-shot 1-hour adaptation to novel subjects keeps CLIP two-way accuracy at 84.7%-88.1%, while synthetic-fMRI retrieval reaches up to 76.5% vs raw 13.2%-19.5%.
- Using SynBrain synthetic fMRI as augmentation (MindEye2 1h+DA 1h) raises CLIP two-way accuracy from 80.8% to 84.7% and Inception from 83.6% to 85.1% on Subject 1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SynBrain instead of MindSimulator (Trials=1) | Subject-specific visual-to-fMRI synthesis · NSD subject-specific, 4-subject average | MSE: .403 -> .139 | Helped |
| Use SynBrain instead of MindSimulator (Trials=5) | Subject-specific visual-to-fMRI synthesis · NSD subject-specific, 4-subject average | CLIP: 91.2% -> 94.3% | Helped |
| Add 1h of SynBrain-generated synthetic fMRI data | MindEye2 (1h) · Subject 1 few-shot fMRI-to-image decoding | CLIP: 80.8% -> 84.7% | Helped |
| Increase synthetic data augmentation from 1h to 4h | MindEye2 (1h) · Subject 1 few-shot fMRI-to-image decoding | CLIP: 84.7% -> 85.0% -> 84.0% | Mixed |
| Add variation sampling (ablation) | SynBrain · Subject 1 subject-specific | CLIP: 86.7% -> 95.9% | Helped |
| Add contrastive learning (ablation) | SynBrain · Subject 1 subject-specific | Syn retrieval: 0.4% -> 99.3% | Helped |
| Add S2N Mapper (ablation) | SynBrain · Subject 1 subject-specific | CLIP: 75.0% -> 95.9% | Helped |
| Use S2N Mapper instead of diffusion transformer (ablation) | BrainVAE · Subject 1 subject-specific | CLIP: 93.5% -> 95.9% | Helped |
| Replace CLIP visual encoder with MAE (ablation) | SynBrain · Subject 1 subject-specific, visual encoder comparison | CLIP: 95.9% -> 90.5% | Hurt |
| Replace CLIP visual encoder with DINOv2 (ablation) | SynBrain · Subject 1 subject-specific, visual encoder comparison | CLIP: 95.9% -> 91.8% | Hurt |
| Increase BrainVAE downsampling blocks from 1 to 2 (ablation) | SynBrain · Subject 1 subject-specific, latent dimensionality | CLIP: 95.9% -> 95.0% | Hurt |
| Increase BrainVAE downsampling blocks from 1 to 3 (ablation) | SynBrain · Subject 1 subject-specific, latent dimensionality | CLIP: 95.9% -> 94.5% | Hurt |
Surprising findings
- Removing the S2N mapper causes the largest semantic drop (CLIP 95.9% to 75.0%), and removing contrastive learning collapses raw-retrieval accuracy to about 0.5%.
- Augmenting with more synthetic data (2h, 4h) plateaus or slightly declines on several decoding metrics, and image retrieval gets worse with augmentation.
Limitations
- Reliance on pre-trained vision models (CLIP) may inherit representational biases not aligned with neural processing.
- Variance sources such as attentional state and neuromodulatory effects are not modeled.
- Evaluation is limited to four NSD subjects and a visual cortex ROI; semantic metrics depend on the MindEye2 decoder.
Experimental setup
Datasets: Natural Scenes Dataset (NSD) COCO
Benchmarks: NSD
Metrics: MSE Pearson correlation cosine similarity Inception two-way accuracy CLIP two-way accuracy EfficientNet distance SwAV distance image retrieval top-1 accuracy PixCorr SSIM
Baselines: MindSimulator MindEye2 MindAligner GNet LinearReg BrainVAE-DiT
Builds on: MindSimulator MindEye2 CLIP SoftCLIP
Closest papers by meaning
- Brain decoding: toward real-time reconstruction of visual perception (ICLR 2024 · similarity 0.87)
- Meta-Learning an In-Context Transformer Model of Human Higher Visual Cortex (NeurIPS 2025 · similarity 0.87)
- Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction (NeurIPS 2024 · similarity 0.87)
- CSBrain: A Cross-scale Spatiotemporal Brain Foundation Model for EEG Decoding (NeurIPS 2025 · similarity 0.87)
- MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data (ICML 2024 · similarity 0.87)
- In Silico Mapping of Visual Categorical Selectivity Across the Whole Brain (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.