Research map / Vision-language and multimodal / Human and embodied modeling
Neuroformer: Multimodal and Multitask Generative Pretraining for Brain Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Neuroformer reframes spiking-population analysis as autoregressive spatiotemporal generation, pretraining a multimodal GPT-style transformer on simulated and two-photon mouse datasets, then fine-tuning for behavior (speed) decoding.
Problem
Analyzing large-scale, cellular-resolution multimodal systems neuroscience data (spiking activity, stimuli, behavior) with tools that handle multiple modalities and transfer to downstream tasks.
Approach
Treat neuron spikes as tokens (neuron ID plus time-bin) in a causally masked, block-structured autoregressive transformer that fuses current/past neural states with video and behavior via cascaded cross-attention, with a CLIP-style contrastive alignment front end; the pretrained model is then fine-tuned with a mean-pooled head for behavior decoding.
Key results
- On the Visnav behavior (speed) prediction task, Neuroformer reached Pearson r=0.95 (lateral) and 0.97 (medial), above Bidirectional GRU (0.83, 0.88), MLP (0.83, 0.85) and GLM (0.69, 0.81).
- Few-shot: a pretrained model fine-tuned on 1% of speed data reached r=0.51, versus r=0.33 for a non-pretrained model trained on 10% of the data.
- On V1+AL neural population prediction, Neuroformer's predicted responses correlated better with ground truth than a GLM (t-test p=0.0196).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Neuroformer instead of conventional behavior-decoding baselines | Visnav behavior prediction · Visnav lateral and medial | Pearson correlation (r): Lateral: 0.83 -> 0.95 (Bidirectional GRU); Medial: 0.88 -> 0.97 (Bidirectional GRU) | Helped |
| Pretrain the model before fine-tuning on behavior data | Neuroformer · Visnav behavior prediction, few-shot fine-tuning | Pearson correlation (r): r_{1%, ft}=0.51 vs. r_{10%, no pre}=0.33 | Helped |
| Add Past State features to the random-feature baseline (ablation) | Speed probing · Visnav | R²: 0.0521 -> 0.2299 | Helped |
| Add Stimulus features to Past State (ablation) | Speed probing · Visnav | R²: 0.2299 -> 0.5341 | Helped |
| Add Current State features to Past State + Stimulus (ablation) | Speed probing · Visnav | R²: 0.5341 -> 0.6357 | Helped |
| Add Past State, Video, behavioral modalities, and contrastive alignment progressively (ablation) | Neuroformer · Neural response generation across datasets and brain regions | F1 score | Helped |
| Add eye-position behavior input (ablation) | Neuroformer · Lateral dataset | Neural response predictive performance | Helped |
| Add contrastive alignment (ablation) | Neuroformer · Neural response prediction | Predictive performance | Helped |
| Use CEBRA for behavior representation learning and decoding | CEBRA · Behavior prediction | Generalization | Hurt |
Surprising findings
- Pretrained model fine-tuned on 1% of behavior data outperformed a randomly initialized model fine-tuned on 10% of the data.
- CEBRA failed to generalize on the behavior task despite extensive hyperparameter search, showing heavy overfitting.
Limitations
- Reliability of attention-based connectivity and receptive-field inference varied across simulated datasets.
- Poor results when applied to representation-learning baselines such as CEBRA; direct comparisons with LFADS are missing.
- Unclear why the transformer outperforms GLM and other baselines; the appendix attributes performance gains to unclear factors.
Experimental setup
Datasets: Hub-net simulated spiking network (Brian2) V1+AL two-photon calcium imaging (386 neurons) Visnav lateral visual cortex two-photon dataset (2022 neurons) Visnav medial two-photon dataset (1905 neurons)
Benchmarks: Hub-net simulation V1+AL two-photon dataset Visnav lateral and medial behavior prediction
Metrics: Pearson correlation F1 R^2 cross-entropy loss MSE
Baselines: Lasso Regression GLM MLP Bidirectional GRU CEBRA LFADS Granger causality partial correlation
Builds on: GPT CLIP Perceiver AdamW
Closest papers by meaning
- Towards a "Universal Translator" for Neural Dynamics at Single-Cell, Single-Spike Resolution (NeurIPS 2024 · similarity 0.89)
- Identifying interactions across brain areas while accounting for individual-neuron dynamics with a Transformer-based variational autoencoder (NeurIPS 2025 · similarity 0.89)
- Population Transformer: Learning Population-level Representations of Neural Activity (ICLR 2025 · similarity 0.88)
- A Scalable, Causal, and Energy Efficient Framework for Neural Decoding with Spiking Neural Networks (NeurIPS 2025 · similarity 0.88)
- Leveraging unlabelled data for generalizable neural population decoding (NeurIPS 2026 · similarity 0.88)
- Modeling Dynamic Neural Activity by combining Naturalistic Video Stimuli and Stimulus-independent Latent Factors (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.