Research map / Probabilistic and scientific modeling / Dynamical systems
BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BrainVista, a causal multimodal autoregressive transformer that forecasts naturalistic fMRI circuit tokens from past brain states and stimulus history, using network-wise tokenizers, a Stimulus-to-Brain mask, and a Spatial Mixer Head.
Problem
Forecasting naturalistic fMRI brain dynamics from multimodal movie stimuli is hard because fast stimulus streams must be aligned with slow hemodynamic BOLD signals without leaking future sensory information, while cortical networks have heterogeneous dynamics.
Approach
Combine per-network MLP autoencoder tokenizers that compress parcel fMRI into circuit tokens, an interleaved stimulus-brain token sequence with a stimulus-to-brain causal mask that blocks look-ahead, and a self-attention mixer over network tokens to model cross-network coupling for open-loop rollout.
Key results
- On Algonauts 2025 at H=20 rollout, BrainVista reaches r=0.2183 and p_corr=0.2090 versus 0.1867/0.1466 for the strongest baseline Brain Harmony.
- On HAD at H=20, BrainVista p_corr=0.2950 versus 0.2688 for Brain Harmony; BrainVista is best at all horizons H=1, 10, 20.
- Spatial Mixer Head ablation on Algonauts 2025 gives p_corr 0.2010 at H=20 versus 0.1706 without the mixer (about +17.8%).
Surprising findings
- Video-only, audio-only and text-only stimulus conditioning all underperform the multimodal model, and the paper reports that removing video hurts most.
- Tokenizer ablation shows a plain MLP autoencoder (r=0.834) outperforms the VAE variant (r=0.507) for reconstruction.
Limitations
- Performance degrades substantially with rollout horizon due to compounding error, with weak correlation in association networks.
- Baselines are re-implemented by freezing pretrained representations and training a past-only MLP, so comparisons depend on this protocol.
- Inconsistencies in the paper: the H=20 numbers in the appendix tables differ across tables and several H=1/10 entries are described as placeholders.
Experimental setup
Datasets: Algonauts 2025 CineBrain HAD
Benchmarks: Algonauts 2025 CineBrain HAD
Metrics: Pearson r pattern correlation p_corr Cls@Top10 MSE Spearman rho 2v2
Baselines: TRIBE MindSimulator BrainLM BrainJEPA Brain Harmony
Builds on: TRIBE Brain Harmony Brain-JEPA BrainLM SynBrain
Closest papers by meaning
- MIRAGE: Adaptive Multimodal Gating for Whole-Brain fMRI Encoding (NeurIPS 2026 · similarity 0.88)
- BrainWorld: A Structural-Prior-Conditioned Generative Model for Whole-Brain 4D fMRI Dynamics (NeurIPS 2026 · similarity 0.87)
- BrainLM: A foundation model for brain activity recordings (ICLR 2024 · similarity 0.87)
- Brain Harmony: A Multimodal Foundation Model Unifying Morphology and Function into 1D Tokens (NeurIPS 2025 · similarity 0.87)
- Brain-OF: An Omnifunctional Foundation Model for fMRI, EEG and MEG (NeurIPS 2026 · similarity 0.87)
- BrainOmni: A Brain Foundation Model for Unified EEG and MEG Signals (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.