Research map / Architectures and efficiency / RNNs and normalization
Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains over 600 from-scratch vision models on ImageNet and EcoSet subsets and fits power-law scaling laws showing behavioral brain alignment keeps improving with scale while neural ventral-stream alignment saturates.
Problem
How model size, training data size, and compute affect alignment of task-optimized neural networks with primate ventral visual stream neural responses (V1, V2, V4, IT) and core object recognition behavior.
Approach
Systematic from-scratch training of many architectures across width, depth, and dataset subsample sizes, combined with parametric power-law fits (including a low-data offset term and a joint N-D loss with compute allocation) to model how alignment scales, separately for each brain region and behavior.
Key results
- Behavioral alignment follows S = 1 - 1.4 C^-0.06 and approaches perfect alignment at large scale, while neural alignment saturates at about 0.48 (S = 0.48 - 0.55 C^-0.16).
- Data scaling yields more brain alignment than parameter scaling; fitted optimal allocation suggests D ~ C^0.7 and N ~ C^0.3.
- Higher cortical areas (IT, behavior) benefit more from scale than V1/V2, with ordering Behavior > IT > V4 > V2 > V1 in alignment gain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase training compute | Task-optimized visual models · Behavioral benchmark | alignment score: S=1-1.4\Tilde{C}^{-0.06} | Helped |
| Increase training compute | Task-optimized visual models · Neural benchmarks | alignment score: S=0.48-0.55\Tilde{C}^{-0.16}; 0.48 | No clear effect |
| Increase dataset size rather than model size | Task-optimized visual models · Primate visual ventral stream | alignment score: 0.44; 0.46; 0.48 | Helped |
| Increase both model size and training dataset size | Task-optimized visual models · Primate visual ventral stream | alignment score: D \approx C^{0.7}; N \approx C^{0.3} | Helped |
| Use architectures with stronger inductive biases, such as convolution or recurrence | Visual models in low-data settings · Primate visual ventral stream | alignment | Helped |
| Increase training data for weaker-inductive-bias architectures | ConvNeXt and ViT · Primate visual ventral stream | neural alignment | Helped |
| Train CORnet models with limited supervision | CORnet · Primate visual ventral stream | alignment | Helped |
| Increase training data for CORnet | CORnet · Primate visual ventral stream | alignment | No clear effect |
| Train with SimCLR instead of supervised learning | ResNet models · Primate visual ventral stream | neural alignment | No clear effect |
| Fine-tune with adversarial FGSM training | ResNet models · Primate visual ventral stream | alignment | Helped |
| Use DINO or SimCLR rather than supervised learning | ViT-S · ImageNet training | alignment | Helped |
| Change the learning signal among supervised learning, SimCLR, and DINO | ResNet50 · ImageNet training | alignment | No clear effect |
Surprising findings
- Neural alignment saturates with scale across architectures, datasets, and even self-supervised training, suggesting task performance alone does not drive neural alignment.
- Models with strong inductive biases (ResNets, EfficientNets) show high neural alignment even at initialization, before any training.
Limitations
- Power-law extrapolation is constrained to the model sizes and dataset volumes tested.
- Only a subset of architectures (CNNs, ViTs, CORnet-S) was studied, not biologically plausible designs.
- Only ImageNet and EcoSet were used for main training, which may not capture visual stimuli relevant to the ventral stream.
Experimental setup
Datasets: ImageNet EcoSet ImageNet21k-P WebVision-P iNaturalist Places365 infiMNIST CIFAR-10
Benchmarks: Brain-Score V1 Brain-Score V2 Brain-Score V4 Brain-Score IT Brain-Score behavior
Metrics: Pearson correlation normalized to ceiling misalignment score power-law fit error
Baselines: ResNet EfficientNet ConvNeXt ViT CORnet-S AlexNet
Builds on: Brain-Score Chinchilla scaling laws EcoSet ConvNeXt Vision Transformer
Closest papers by meaning
- Multimodal Scaling Laws for Task & Data-Optimized Models of Visual Cortex (ICML 2026 · similarity 0.92)
- Vision CNNs trained to estimate spatial latents learned similar ventral-stream-aligned representations (ICLR 2025 · similarity 0.90)
- Sparse components distinguish visual pathways & their alignment to neural networks (ICLR 2025 · similarity 0.89)
- Model–Behavior Alignment under Flexible Evaluation: When the Best-Fitting Model Isn’t the Right One (NeurIPS 2025 · similarity 0.88)
- Bridging Critical Gaps in Convergent Learning: How Representational Alignment Evolves Across Layers, Training, and Distribution Shifts (NeurIPS 2025 · similarity 0.87)
- STSBench: A Large-Scale Dataset for Modeling Neuronal Activity in the Dorsal Stream of Primate Visual Cortex (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.