Research map / Video / Action recognition
STSBench: A Large-Scale Dataset for Modeling Neuronal Activity in the Dorsal Stream of Primate Visual Cortex
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Releases STSBench, a dataset of ~2,244 Neuropixels single/multi-unit recordings from macaque STS (MT/MST) to natural videos, benchmarks video encoding models (3D CNNs vs. pretrained and Gabor baselines) and a neural-conditional latent diffusion model for stimulus reconstruction.
Problem
Dorsal stream (MT/MST) neuronal responses to natural videos are poorly characterized because existing datasets are tiny (about 45 neurons), which limits development of deep encoding and reconstruction models.
Approach
Large-scale Neuropixels NHP recordings in STS while macaques viewed ~4,500 egocentric Ego4D video clips, used to benchmark end-to-end trained 3D CNN encoders against pretrained and hand-tuned baselines, and to reconstruct stimuli by swapping the text conditioning in Stable Diffusion with neural activity vectors.
Key results
- End-to-end 5-layer 3D CNN achieved the best encoding performance (R^2 0.338), outperforming the 3D Gabor (0.266) and 3D ResNet-Kinetics layer 2 (0.303) baselines.
- Diffusion reconstruction from dorsal stream reached PSNR 14.161 and LPIPS 0.668, beating shuffled (9.848/0.751) and mean (11.335/0.873) nulls.
- Motion direction decoded from neural population with mean cosine similarity 0.923 versus 0.051 for shuffled control.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train a 5-layer 3D CNN end-to-end instead of using the strongest reported baseline | STSBench encoding models · STSBench test set | R²: 0.303 ± 0.004 -> 0.338 ± 0.005 | Helped |
| Increase end-to-end 3D CNN depth from 1 to 3 layers (ablation) | 3D CNN encoding model · STSBench test set | R²: 0.206 ± 0.004 -> 0.276 ± 0.004 | Helped |
| Increase end-to-end 3D CNN depth from 3 to 5 layers (ablation) | 3D CNN encoding model · STSBench test set | R²: 0.276 ± 0.004 -> 0.338 ± 0.005 | Helped |
| Increase end-to-end 3D CNN depth from 5 to 7 layers (ablation) | 3D CNN encoding model · STSBench test set | R²: 0.338 ± 0.005 -> 0.332 ± 0.005 | No clear effect |
| Use earlier rather than later 3D ResNet-Kinetics features (ablation) | 3D ResNet-Kinetics encoding model · STSBench test set | R²: 0.147 ± 0.003 -> 0.303 ± 0.004 | Helped |
| Condition image generation on neural activity rather than use the mean-image null | Latent diffusion reconstruction · STSBench (MT/MST) | PSNR: 11.335 -> 14.161 | Helped |
| Condition image generation on neural activity rather than use the mean-image null | Latent diffusion reconstruction · STSBench (MT/MST) | LPIPS: 0.873 -> 0.668 | Helped |
| Condition image generation on neural activity rather than use the mean-image null | Latent diffusion reconstruction · TVSD (V4) | PSNR: 12.192 -> 10.631 | Hurt |
| Condition image generation on neural activity rather than use the mean-image null | Latent diffusion reconstruction · TVSD (V4) | LPIPS: 0.912 -> 0.589 | Helped |
| Use a linear decoder rather than conditional diffusion | STSBench image reconstruction · STSBench (MT/MST) | PSNR: 14.161 -> 14.818 | Helped |
| Use conditional diffusion rather than a linear decoder | STSBench image reconstruction · STSBench (MT/MST) | LPIPS: 0.820 -> 0.668 | Helped |
| Decode motion direction from neural activity rather than shuffled targets | RidgeCV motion direction decoder · STSBench held-out test set | cosine similarity: 0.051 -> 0.923 | Helped |
Surprising findings
- Linear decoder achieved higher dorsal-stream PSNR (14.818) than the diffusion model (14.161), though diffusion had better LPIPS.
- For the pretrained 3D ResNet-Kinetics, earlier layers predicted MT/MST activity better than later layers.
Limitations
- Some superficial neurons may come from adjacent area 7a or white matter.
- Kilosort4 spike sorting can cause split and merge errors, and units include multi-units.
- RS/FS waveform classification likely underestimates cell-type diversity.
Experimental setup
Datasets: STSBench TVSD Ego4D
Benchmarks: STSBench TVSD
Metrics: R^2 PSNR LPIPS cosine similarity
Baselines: 3D Gabor filter bank 3D ResNet-Kinetics 3D ResNet-Self Motion (DorsalNet) 2D ResNet-ImageNet Linear decoder CNN decoder
Builds on: Stable Diffusion Kilosort4 Ego4D TVSD
Closest papers by meaning
- Sparse components distinguish visual pathways & their alignment to neural networks (ICLR 2025 · similarity 0.88)
- Vision CNNs trained to estimate spatial latents learned similar ventral-stream-aligned representations (ICLR 2025 · similarity 0.87)
- Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream (ICML 2025 · similarity 0.87)
- Retrospective for the Dynamic Sensorium Competition for predicting large-scale mouse primary visual cortex activity from videos (NeurIPS 2024 · similarity 0.86)
- Long-Range Feedback Spiking Network Captures Dynamic and Static Representations of the Visual Cortex under Movie Stimuli (NeurIPS 2024 · similarity 0.86)
- Benchmarking Out-of-Distribution Generalization Capabilities of DNN-based Encoding Models for the Ventral Visual Cortex. (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.