Research map / Vision-language and multimodal / Human and embodied modeling
Retrospective for the Dynamic Sensorium Competition for predicting large-scale mouse primary visual cortex activity from videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Retrospective of the SENSORIUM 2023 competition that benchmarked dynamic models predicting single-neuron responses in mouse V1 to videos, releasing a 78,853-neuron dataset and summarizing winning architectures.
Problem
Lack of a standardized, withheld-test-set benchmark for dynamic (video-driven) predictive models of neuronal responses in the mouse primary visual cortex, including out-of-distribution stimulus generalization.
Approach
Establishes a public competition with a large calcium-imaging dataset from ten mice, two evaluation tracks (natural-video main track and OOD bonus track), and a single-trial/average correlation metric; winners use transformer cores or enhanced 3D-factorized CNN cores with modified readouts.
Key results
- Winning DwiseNeuro reached single-trial correlation 0.291 on the main track vs. 0.197 for the ensemble baseline, a ~48% improvement.
- On the OOD bonus track, the winner reached 0.221 single-trial correlation vs. 0.129 for the ensemble baseline (~70% improvement).
- 163 model submissions from 22 teams were received during the four-month competition.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DwiseNeuro instead of the ensemble baseline | Mouse V1 response prediction · Main track | single-trial correlation (ρst): 0.197 -> 0.291 | Helped |
| Use DwiseNeuro instead of the ensemble baseline | Mouse V1 response prediction · Bonus track (OOD) | single-trial correlation (ρst): 0.129 -> 0.221 | Helped |
| Add core techniques, tune hyperparameters, and train on ten mice instead of five (ablation) | DwiseNeuro early model · Main track live-test | score: ≈0.19 -> 0.25 | Helped |
| Remove normalization (ablation) | DwiseNeuro · Main track live-test | score: 0.25 -> 0.27 | Helped |
| Add the cortex module and CutMix (ablation) | DwiseNeuro · Main track live-test | score: 0.27 -> 0.276 | Helped |
| Tune the Softplus β value (ablation) | DwiseNeuro · Main track live-test | score: 0.276 -> 0.294 | Helped |
| Adjust drop rate and batch size (ablation) | DwiseNeuro · Main track live-test | score: 0.294 -> 0.3 | Helped |
| Add CutMix data augmentation (ablation) | DwiseNeuro · Main track | single-trial correlation: +0.006 | Helped |
| Tune the Softplus nonlinearity (ablation) | DwiseNeuro · Main track | single-trial correlation: +0.018 | Helped |
| Replace 3D convolution with a 3D swin transformer core (ablation) | Dynamic-V1FM · Main track live-test | R2: 0.188 -> 0.233 | Helped |
| Replace Gaussian readout with Hierarchical Gaussian Readout (ablation) | Dynamic-V1FM · Main track live-test | R2: 0.233 -> 0.251 | Helped |
| Add feature enhancement modules to the low-resolution features (ablation) | Dynamic-V1FM · Competition evaluation | performance | No clear effect |
Surprising findings
- Data augmentation (CutMix) gave only a small gain (+0.006) and tuning the softplus nonlinearity contributed about +0.018 for the top solution.
- Models trained with seed variation show std of about 0.009 in single-trial correlation across 14 ensemble seeds, indicating seed sensitivity.
Limitations
- Modern training demands (multi-GPU, long wall times) exclude many participants from freely available compute such as Colab or Kaggle.
- Only excitatory neurons in layers 2-5 of V1 under passive, head-fixed conditions with limited behavioral variables are covered.
- No error bars reported for the main results; seed analysis is provided only in the appendix.
Experimental setup
Datasets: SENSORIUM 2023 dataset (natural movies from MICrONS/Sports-1M/cinematic movies, OOD stimuli)
Benchmarks: SENSORIUM 2023 main track SENSORIUM 2023 bonus (OOD) track
Metrics: single-trial correlation correlation to average
Baselines: GRU baseline Factorized baseline Ensemble baseline SENSORIUM 2022 winner
Builds on: Gaussian readout factorized 3D convolution core Swin transformer ViViT
Closest papers by meaning
- Modeling Dynamic Neural Activity by combining Naturalistic Video Stimuli and Stimulus-independent Latent Factors (NeurIPS 2025 · similarity 0.89)
- NEvo: Neural-Guided Evolutionary Video Synthesis for Dynamic Visual Selectivity (NeurIPS 2026 · similarity 0.87)
- Benchmarking Out-of-Distribution Generalization Capabilities of DNN-based Encoding Models for the Ventral Visual Cortex. (NeurIPS 2024 · similarity 0.86)
- STSBench: A Large-Scale Dataset for Modeling Neuronal Activity in the Dorsal Stream of Primate Visual Cortex (NeurIPS 2025 · similarity 0.86)
- Long-Range Feedback Spiking Network Captures Dynamic and Static Representations of the Visual Cortex under Movie Stimuli (NeurIPS 2024 · similarity 0.86)
- Time-Evolving Dynamical System for Learning Latent Representations of Mouse Visual Neural Activity (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.