Research map / Probabilistic and scientific modeling / Dynamical systems
Inpainting the Neural Picture: Inferring Unrecorded Brain Area Dynamics from Multi-Animal Datasets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces NeuroPaint, a masked transformer autoencoder trained across animals with partially overlapping recorded areas to inpaint latent dynamics of unrecorded brain areas, evaluated on synthetic RNN data and IBL and MAP Neuropixels datasets.
Problem
Multi-area neural interaction analysis is limited because simultaneous Neuropixels recordings rarely cover all areas of interest in one session, so dynamics of unrecorded areas cannot be directly observed.
Approach
Mask-and-reconstruct training across multi-animal sessions with area-specific latent factors (including learned mask tokens for unrecorded areas), a cross-attention read-in stitcher, and a consistency loss on cross-area embedding correlations, so unrecorded-area latents can be inferred from other animals.
Key results
- On IBL and MAP held-out areas, NeuroPaint outperforms LFADS by a large margin in DFE, per the paper's main comparison.
- On the synthetic dataset, the reconstruction+consistency NeuroPaint variant is closest to the upper bound and beats GLM and LFADS baselines.
- NeuroPaint infers area-specific latents for unrecorded areas whose trial-to-trial correlations and decoding remain task-relevant, and shows similarity between areas rising during the response period in MAP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use consistency loss rather than regularization loss alongside reconstruction loss (ablation) | NeuroPaint · Synthetic dataset | DFE | Helped |
| Train with reconstruction, consistency, and regularization losses together (ablation) | NeuroPaint · High-firing-rate synthetic dataset | DFE | Hurt |
| Add consistency loss (ablation) | NeuroPaint · Synthetic dataset | DFE | Helped |
| Use NeuroPaint area-specific latent factors instead of LFADS latent factors | held-out-area prediction · IBL | DFE | Helped |
| Use NeuroPaint area-specific latent factors instead of LFADS latent factors | held-out-area prediction · MAP | DFE | Helped |
| Add consistency loss (ablation) | NeuroPaint · Low-firing-rate synthetic dataset | DFE | Helped |
| Include all three losses rather than an ablated loss-term variant (ablation) | NeuroPaint · Low-firing-rate synthetic dataset | DFE | Helped |
| Remove regularization loss (ablation) | NeuroPaint · IBL and MAP | DFE | Hurt |
| Remove consistency loss (ablation) | NeuroPaint · IBL and MAP | DFE | Hurt |
| Use GLM instead of NeuroPaint for held-out-area prediction | held-out-area prediction · IBL | mean DFE | Hurt |
| Use GLM instead of NeuroPaint for held-out-area prediction | held-out-area prediction · MAP | mean DFE | Hurt |
Surprising findings
- The GLM baseline gives extremely negative DFE on IBL (-3.8e32) and MAP (-5.1e23) datasets, indicating severe overfitting or mismatch.
- Combining all three losses degraded performance on the higher-firing-rate synthetic data, but was best on the low-rate synthetic and real datasets.
Limitations
- Training is computationally expensive since self-attention scales quadratically with brain areas and time steps.
- Latent dimensionality chosen via a heuristic procedure with some arbitrariness; selected dims often exceed intrinsic dimensionality.
- Evaluated on a limited subset of brain areas and sessions; scaling to hundreds of areas needs architectural work.
Experimental setup
Datasets: IBL brain-wide map MAP (Mesoscale Activity Project) synthetic RNN-generated spike trains
Benchmarks: synthetic RNN dataset IBL brain-wide map MAP dataset
Metrics: deviance fraction explained (DFE)
Baselines: GLM LFADS NeuroPaint-Linear
Builds on: LFADS Perceiver-IO masked autoencoders multi-task-masking mDLAG
Closest papers by meaning
- Towards a "Universal Translator" for Neural Dynamics at Single-Cell, Single-Spike Resolution (NeurIPS 2024 · similarity 0.88)
- Neural Encoding and Decoding at Scale (ICML 2025 · similarity 0.88)
- Accurate Identification of Communication Between Multiple Interacting Neural Populations (ICML 2025 · similarity 0.87)
- Identifying interactions across brain areas while accounting for individual-neuron dynamics with a Transformer-based variational autoencoder (NeurIPS 2025 · similarity 0.87)
- Learning dynamic representations of the functional connectome in neurobiological networks (ICLR 2024 · similarity 0.86)
- CREIMBO: Cross-Regional Ensemble Interactions in Multi-view Brain Observations (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.