Research map / Vision-language and multimodal / Human and embodied modeling
Neural Encoding and Decoding at Scale
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces NEDS, a multimodal transformer pretrained with multi-task masking over neural activity and behavior that performs both neural encoding and decoding, outperforming POYO+ and NDT2 on IBL repeated-site data.
Problem
Large-scale neural models typically handle only encoding (behavior to neural activity) or only decoding (neural activity to behavior), not both bidirectionally across animals.
Approach
A single transformer trained with multi-task masking that alternates neural, behavioral, within-modality, and cross-modality masking, so the same pretrained model can encode neural activity from behavior or decode behavior from neural activity; neuron embeddings also encode brain region without explicit supervision.
Key results
- Multi-session NEDS outperforms NDT2 by 11-37% and POYO+ by 1-13% across decoding tasks on 10 held-out animals.
- Multi-session NEDS improves encoding by 24% over single-session NEDS and reaches 83% brain-region classification accuracy from neuron embeddings.
- Single-session multimodal NEDS beats unimodal NEDS, improving encoding by 5% and decoding by 2-7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pretrain multimodal NEDS across 74 sessions rather than train on one session | NEDS · IBL repeated site, 10 held-out animals | encoding and decoding performance: decoding improving by 4–11%; encoding improving by 24% | Helped |
| use multimodal, multi-task NEDS instead of POYO+ | neural decoding on IBL · IBL repeated site, 10 held-out animals | decoding performance: 1–13% | Helped |
| use multimodal, multi-task NEDS instead of NDT2 | neural decoding on IBL · IBL repeated site, 10 held-out animals | decoding performance: 11–37% | Helped |
| add cross-modal masking (ablation) | single-session NEDS · IBL held-out test sessions | wheel decoding R²: 0.50\pm0.02 -> $0.57 \pm0.03$ | Helped |
| add within-modality masking (ablation) | single-session NEDS · IBL held-out test sessions | encoding (bps): $0.10 \pm0.02$ -> $0.20 \pm 0.00$ | Helped |
| increase model size from 3 million to 12 million parameters (ablation) | single-session NEDS · IBL held-out test sessions | encoding (bps): $0.20 \pm 0.00$ -> $0.09 \pm0.06$ | Hurt |
| add modality embeddings (ablation) | single-session NEDS · IBL repeated site, 10 test sessions | whisker decoding R²: \num{0.459} $\pm$ \num{0.086} -> \num{0.523} $\pm$ \num{0.097} | Helped |
| add temporal embeddings (ablation) | single-session NEDS · IBL repeated site, 10 test sessions | wheel decoding R²: \num{0.523} $\pm$ \num{0.082} -> \num{0.568} $\pm$ \num{0.089} | Helped |
| add temporal and modality embeddings (ablation) | single-session NEDS · IBL repeated site, 10 test sessions | encoding (bps): \num{0.246} $\pm$ \num{0.069} -> \num{0.203} $\pm$ \num{0.062} | Hurt |
| use multimodal instead of unimodal NEDS | single-session NEDS · IBL single-session tasks | encoding and decoding: encoding by 5%; decoding by 2–7% | Mixed |
| use multimodal NEDS rather than MLP | finger velocity prediction · MC-RTT monkey reaching dataset | single-trial R²: 0.66 -> 0.72 | Helped |
| use all behavioral task variables instead of any one variable | multi-session NEDS encoding · IBL repeated site | encoding (bps): 0.27 ± 0.03; individual variables: 0.06 ± 0.02, 0.10 ± 0.02, 0.12 ± 0.02, 0.11 ± 0.02 | Helped |
Surprising findings
- Neuron embeddings from NEDS predict brain region well despite no brain-region supervision during training.
- A 12M-parameter model overfits in single-session training, performing worse than a 3M model.
Limitations
- Relies exclusively on trial-aligned data, limiting pretraining data volume.
- Limited hyperparameter tuning for the 73-animal pretraining due to compute constraints.
- Encoding performance is low in absolute bps (0.27), and the MC-RTT test is single-session only.
Experimental setup
Datasets: IBL repeated site dataset MC-RTT primate motor task dataset
Benchmarks: IBL repeated site dataset MC-RTT
Metrics: bits per spike (co-bps) classification accuracy single-trial R^2
Baselines: POYO+ NDT2 RRR linear regression unimodal NEDS MLP
Builds on: POYO NDT2 masked autoencoder Ray Tune
Closest papers by meaning
- Towards a "Universal Translator" for Neural Dynamics at Single-Cell, Single-Spike Resolution (NeurIPS 2024 · similarity 0.88)
- Inpainting the Neural Picture: Inferring Unrecorded Brain Area Dynamics from Multi-Animal Datasets (NeurIPS 2025 · similarity 0.88)
- BaRISTA: Brain Scale Informed Spatiotemporal Representation of Human Intracranial Neural Activity (NeurIPS 2025 · similarity 0.87)
- Modeling Dynamic Neural Activity by combining Naturalistic Video Stimuli and Stimulus-independent Latent Factors (NeurIPS 2025 · similarity 0.87)
- Dynamical modeling of nonlinear latent factors in multiscale neural activity with real-time inference (NeurIPS 2025 · similarity 0.87)
- MEIcoder: Decoding Visual Stimuli from Neural Activity by Leveraging Most Exciting Inputs (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.