Research map / Probabilistic and scientific modeling / Dynamical systems
Modeling Dynamic Neural Activity by combining Naturalistic Video Stimuli and Stimulus-independent Latent Factors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a probabilistic video-based encoding model that adds a stimulus-independent latent state (VAE-style, from neuron subsets) to a factorized 3D-CNN core with a zero-inflated gamma likelihood, evaluated on SENSORIUM mouse V1 recordings.
Problem
Dynamic encoding models of mouse V1 responses to video either give point estimates without response distributions or ignore internal brain state; no dynamic video-based model explicitly combines a latent state with the full neuronal response distribution.
Approach
Modify a deterministic core-readout model (factorized 3D CNN + Gaussian readout) into a probabilistic latent-variable model: a GRU encoder infers a dynamic latent from a subset of neuron responses, which modulates a zero-inflated gamma response distribution alongside the video features; trained by ELBO, with the latent marginalized from the prior when no neuronal input is given.
Key results
- Latent ZIG reaches log-likelihood -0.30 bits per neuron and time vs -0.98 for video-only ZIG, with correlation 0.182 vs 0.183 (Poisson baseline 0.195).
- Conditioning on half of the neurons raises correlation on held-out neurons to about 0.27 vs 0.18 for the video-only prediction.
- Latent factors correlate strongly with treadmill speed and pupil dilation despite no behavioral data in training; on OOD stimuli, conditioned correlation averages 0.17 vs 0.08 for Poisson.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add stimulus-independent latent factors to the video-only ZIG model | Video-only ZIG · SENSORIUM mice 1–5, non-conditioned | Log-likelihood: -0.98 { $\pm$0.04} -> -0.30 { $\pm$0.05} | Helped |
| Add stimulus-independent latent factors to the video-only ZIG model | Video-only ZIG · SENSORIUM mice 1–5, non-conditioned | Pearson correlation: 0.183 { $\pm$0.003} -> 0.182 { $\pm$0.004} | No clear effect |
| Learn per-neuron response distributions with ZIG instead of point predictions | Poisson Baseline · SENSORIUM bonus track OOD stimuli | Correlation: 0.08{ $\pm$0.009} -> 0.11{ $\pm$0.008} | Helped |
| Condition on responses from half the neurons | Latent ZIG · SENSORIUM bonus track OOD stimuli | Correlation: 0.11{ $\pm$0.007} -> 0.17{ $\pm$0.009} | Helped |
| Predict latent feature vectors from cortical positions instead of freely learning them (ablation) | Latent ZIG · SENSORIUM mice 1–5, conditioned on half of neurons | Log-likelihood: -0.34 -> -0.69 | Hurt |
| Share the same latent feature vector across neurons, differing only by per-neuron scale (ablation) | Latent ZIG · SENSORIUM mice 1–5, conditioned on half of neurons | Correlation: 0.24 -> 0.18 | Hurt |
| Use cortical coordinates to map latent features for held-out neurons | Position-based latent mapping · SENSORIUM mice 6–10, 500 held-out neurons | Correlation: 0.213{ \, $\pm$0.005} -> 0.210{ \, $\pm$0.004} | No clear effect |
| Train without pretrained video-component weights (ablation) | Latent ZIG · SENSORIUM mice 6–10, non-conditioned | Correlation: 0.18 -> 0.10 | Hurt |
| Add a GRU or CNN smoother to latent variables (ablation) | Latent ZIG · SENSORIUM mice 1–5, 12-dimensional latent | Log-likelihood: -0.30 -> -0.74 (GRU); -0.30 -> -0.77 (CNN) | Hurt |
| Train latent ZIG with Poisson loss rather than ZIG loss (ablation) | Latent ZIG, conditioned on half of neurons · SENSORIUM mice 1–5 | Pearson correlation: 0.230 -> 0.266 | Helped |
| Replace the ZIG response distribution with a Gaussian response model (ablation) | Latent ZIG · SENSORIUM mice 6–10 | Log-likelihood: -0.88 -> -2.1 | Hurt |
| Add a learned flow to the Gaussian response model (ablation) | Gaussian latent model · SENSORIUM mice 6–10 | Log-likelihood: -2.1 -> -1.2 | Helped |
Limitations
- Residual stimulus-driven content in the latents cannot be fully excluded (contrast R^2 up to 0.27 for one mouse).
- The model uses more parameters per neuron than the video-only baseline.
- Built on the SENSORIUM competition baseline core rather than state-of-the-art cores; ViT core results were below Poisson baseline.
Experimental setup
Datasets: SENSORIUM 2022 mouse V1 two-photon recordings (5 mice, extended to mice 1-10) SENSORIUM bonus track OOD pink-noise videos
Benchmarks: SENSORIUM SENSORIUM bonus OOD stimuli
Metrics: Pearson correlation log-likelihood in bits per neuron and time CCA correlation with behavior R^2 of contrast regression
Baselines: SENSORIUM Poisson baseline Video-only ZIG model CEBRA Vision Transformer core model Gaussian latent model
Builds on: SENSORIUM baseline factorized 3D convolutional core Gaussian readout variational autoencoder
Closest papers by meaning
- Time-Evolving Dynamical System for Learning Latent Representations of Mouse Visual Neural Activity (NeurIPS 2025 · similarity 0.90)
- Retrospective for the Dynamic Sensorium Competition for predicting large-scale mouse primary visual cortex activity from videos (NeurIPS 2024 · similarity 0.89)
- Multi-Modal Latent Variables for Cross-Individual Primary Visual Cortex Modeling and Analysis (AAAI 2025 · similarity 0.88)
- Multi-modal Gaussian Process Variational Autoencoders for Neural and Behavioral Data (ICLR 2024 · similarity 0.87)
- Flexible Context-Driven Sensory Processing in Dynamical Vision Models (NeurIPS 2024 · similarity 0.87)
- Neuroformer: Multimodal and Multitask Generative Pretraining for Brain Data (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.