Research map / Image generation / Text-to-image diffusion
BrainBits: How Much of the Brain are Generative Reconstruction Methods Using?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BrainBits, a linear information-bottleneck sweep that measures how much neural signal generative fMRI-to-image/text reconstruction methods actually need, and finds that very small bottlenecks (about 50 dimensions) recover most reconstruction performance.
Problem
Stimulus reconstruction from fMRI with powerful generative models may reflect the generative prior and metric artifacts rather than improved extraction of information from the brain, so reported fidelity overstates how much of the neural signal is used.
Approach
Insert a linear bottleneck of varying dimension between fMRI input and each existing reconstruction pipeline, trained jointly with the original objective, to measure how much brain information is needed to reach a method's performance; complement this with a method-specific random baseline (noise input) and a ceiling (ground-truth latents).
Key results
- BrainDiffuser with a 50-dimensional bottleneck achieves 75%, 95%, 100%, and 89% of original performance on DreamSim, CLIP cosine similarity, SSIM, and pixel correlation, using a reduction of roughly 300x from about 14,000 voxels.
- For language decoding, a 1000-dimensional bottleneck recovers 50% of the original BERTScore performance averaged across subjects, with 26% of BLEU and 20% of METEOR performance.
- Random-noise baselines achieve high scores for vision methods because generative priors dominate; the reconstruction ceiling with perfect latents is surprisingly low on some metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| restrict BrainDiffuser to a 50-dimensional bottleneck (ablation) | BrainDiffuser · NSD visual reconstruction | DreamSim performance relative to full method: 75% | Hurt |
| restrict BrainDiffuser to a 50-dimensional bottleneck (ablation) | BrainDiffuser · NSD visual reconstruction | CLIP cosine similarity relative to full method: 95% | Hurt |
| restrict BrainDiffuser to a 50-dimensional bottleneck (ablation) | BrainDiffuser · NSD visual reconstruction | SSIM relative to full method: 100% | No clear effect |
| restrict BrainDiffuser to a 50-dimensional bottleneck (ablation) | BrainDiffuser · NSD visual reconstruction | Pixel correlation relative to full method: 89% | Hurt |
| restrict Tang et al. to a 1000-dimensional bottleneck (ablation) | Tang et al. · Language reconstruction, averaged across subjects and tasks | BERT performance relative to full method: 50% | Hurt |
| restrict Tang et al. to a 1000-dimensional bottleneck (ablation) | Tang et al. · Language reconstruction, averaged across subjects and tasks | BLEU performance relative to full method: 26% | Hurt |
| restrict Tang et al. to a 1000-dimensional bottleneck (ablation) | Tang et al. · Language reconstruction, averaged across subjects and tasks | METEOR performance relative to full method: 20% | Hurt |
| restrict Tang et al. to a 1000-dimensional bottleneck (ablation) | Tang et al. · Language reconstruction, averaged across subjects and tasks | WER | No clear effect |
| Use synthetic standard-normal inputs instead of neural recordings for the random baseline | Tang et al. · Language reconstruction | BERT score: approximately 0.79 | No clear effect |
Surprising findings
- Random-input baselines from the generative prior already produce images with high metric scores, so the performance range attributable to the brain is narrow.
- Effective dimensionality of a 50-dimensional BrainDiffuser bottleneck is only about 16, and language bottlenecks use roughly 5 to 20 effective dimensions.
Limitations
- Requires rerunning decoding for many bottleneck sizes and joint integration into each method, so it is not plug-and-play.
- Only linear bottlenecks are studied; sub-one-float bottlenecks are hard to train and vector quantization is left to future work.
- Near-chance performance (e.g., language metrics) makes the apparent bottleneck size exaggerated because metrics lack calibrated scales.
Experimental setup
Datasets: Natural Scenes Dataset Tang et al. 2023 language fMRI data (perceived/imagined speech, movie, multi-speaker) MS COCO
Benchmarks: Natural Scenes Dataset reconstruction evaluation Tang et al. 2023 language decoding evaluation
Metrics: DreamSim CLIP cosine similarity SSIM pixel correlation WER BLEU METEOR BERTScore effective dimensionality (95% variance PCA) identification accuracy
Baselines: BrainDiffuser Takagi & Nishimoto Tang et al. 2023
Builds on: BrainDiffuser Takagi & Nishimoto 2023 Tang et al. 2023 VDVAE Versatile Diffusion
Closest papers by meaning
- Brain decoding: toward real-time reconstruction of visual perception (ICLR 2024 · similarity 0.84)
- NeuroClips: Towards High-fidelity and Smooth fMRI-to-Video Reconstruction (NeurIPS 2024 · similarity 0.84)
- Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion (NeurIPS 2024 · similarity 0.84)
- MonkeySee: Space-time-resolved reconstructions of natural images from macaque multi-unit activity (NeurIPS 2024 · similarity 0.84)
- Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction (NeurIPS 2024 · similarity 0.84)
- MoRE-Brain: Routed Mixture of Experts for Interpretable and Generalizable Cross-Subject fMRI Visual Decoding (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.