Research map / Label-efficient and robust learning / Self-supervised and contrastive
Towards a "Universal Translator" for Neural Dynamics at Single-Cell, Single-Spike Resolution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes multi-task-masking (MtM), a self-supervised pretraining scheme for multi-region neural spiking data that alternates among causal, neuron, intra-region and inter-region masking with a learnable prompt token, evaluated on the IBL repeated-site dataset with NDT1-stitch and NDT2.
Problem
Building a foundation model for single-neuron, single-spike population recordings across many brain regions and animals; existing neural population transformers pretrain on sensorimotor data and treat the population as homogeneous, ignoring brain region structure.
Approach
Alternate among four masking schemes (causal, neuron, intra-region, inter-region) during self-supervised training, with a learnable prompt token that selects the masking objective and can be switched at test time for task-specific adaptation; masking is applied to inputs so the method is architecture-agnostic.
Key results
- MtM (prompted) NDT1-stitch multi-session co-smoothing 1.11 bps vs 0.79 for temporal masking; intra-region 0.85 vs -0.08.
- Single-session NDT1 MtM forward prediction 0.57 bps vs 0.42 temporal baseline, with intra-region 0.43 vs -0.20.
- Choice decoding is comparable between MtM and temporal masking (0.88 vs 0.89 multi-session), while whisker motion energy R2 slightly improves (0.68 vs 0.67).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace temporal masking with prompted MtM | NDT1-stitch, single-session fine-tuning · 5 held-out sessions | co-smoothing: 0.55 -> 1.00 | Helped |
| Replace temporal masking with prompted MtM | NDT1-stitch, single-session fine-tuning · 5 held-out sessions | forward prediction: 0.17 -> 0.28 | Helped |
| Replace temporal masking with prompted MtM | NDT1-stitch, single-session fine-tuning · 5 held-out sessions | intra-region prediction: -0.49 -> 0.70 | Helped |
| Replace temporal masking with prompted MtM | NDT1-stitch, multi-session fine-tuning · 5 held-out sessions | choice accuracy: 0.89 -> 0.88 | Hurt |
| Add neuron masking only (ablation) | NDT1 · single-session selected session | co-smoothing: 0.84 -> 1.04 | Helped |
| Add causal masking only (ablation) | NDT1 · single-session selected session | forward prediction: 0.42 -> 0.48 | Helped |
| Add intra-region masking only (ablation) | NDT1 · single-session selected session | intra-region prediction: -0.20 -> 0.32 | Helped |
| Add inter-region masking only (ablation) | NDT1 · single-session selected session | inter-region prediction: 0.57 -> 0.90 | Helped |
| Add prompting to MtM (ablation) | NDT1 · single-session selected session | forward prediction: 0.54 -> 0.57 | Helped |
| Switch among neuron, causal, intra-region, and inter-region prompts at inference (ablation) | NDT1-stitch MtM · 5 held-out sessions | whisker motion energy R²: 0.68; 0.68; 0.68; 0.68 | No clear effect |
| Vary MtM masking ratio from 0.1 to 0.6 (ablation) | NDT1 · single-session selected session | co-smoothing: 1.31 -> 0.93 | Hurt |
Surprising findings
- NDT2 underperformed NDT1 across metrics, which the authors attribute to NDT2's spatiotemporal patching not handling spike-sorted multi-region recordings well.
- For choice decoding, MtM did not outperform temporal masking, and the prompt choice had little effect on behavior decoding.
Limitations
- Data diversity (39 mice, five brain regions) is far smaller than the full IBL brain-wide map.
- Uses simple NDT1/NDT2 time-series transformer architectures with basic tokenization.
- NDT2 used a fixed 128-neuron patch size due to computation, which likely hurt its performance and scaling.
Experimental setup
Datasets: IBL repeated site dataset
Benchmarks: IBL repeated site dataset co-smoothing forward prediction intra-region co-smoothing inter-region co-smoothing choice decoding whisker motion energy decoding
Metrics: bits per spike (co-bps) accuracy R-squared
Baselines: NDT1 temporal masking NDT2 random token masking NDT1-stitch temporal masking single-masking-scheme ablations
Builds on: Neural Data Transformer (NDT1) NDT2 masked autoencoding coordinated dropout POYO
Closest papers by meaning
- Neuroformer: Multimodal and Multitask Generative Pretraining for Brain Data (ICLR 2024 · similarity 0.89)
- A Scalable, Causal, and Energy Efficient Framework for Neural Decoding with Spiking Neural Networks (NeurIPS 2025 · similarity 0.89)
- Leveraging unlabelled data for generalizable neural population decoding (NeurIPS 2026 · similarity 0.88)
- Neural Encoding and Decoding at Scale (ICML 2025 · similarity 0.88)
- POCO: Scalable Neural Forecasting through Population Conditioning (NeurIPS 2025 · similarity 0.88)
- Inpainting the Neural Picture: Inferring Unrecorded Brain Area Dynamics from Multi-Animal Datasets (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.