Research map / Architectures and efficiency / RNNs and normalization
Feature-Learning Networks Are Consistent Across Widths At Realistic Scales
Nikhil Vyas, Alexander Atanasov, Blake Bordelon, Depen Morwani, Sabarish Sainathan, Cengiz Pehlevan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase width in μP ResNet-18 | ResNet-18 · CIFAR-5m | training crossentropy consistency | Improved |
| Increase width in μP ResNet-18 | ResNet-18 · ImageNet | training crossentropy consistency | Improved |
| Increase width in μP transformer | Transformer · Wikitext-103 | training crossentropy consistency | Improved |
| Increase width in μP models | μP networks · ImageNet and Wikitext-103 | training crossentropy | Improved |
| Ensemble narrow networks across initializations | μP networks · Online vision and language tasks | training crossentropy | Mixed |
| Train for multiple epochs instead of online training | μP networks · CIFAR-200k | training error consistency across widths | Hurt |
| Train ensembles on noisy labels for multiple epochs | μP ResNets · Noisy CIFAR-50k | test error | Mixed |
| Use μP instead of NTK parameterization (ablation) | CNN · CIFAR-10 | width consistency of test error and kernel alignment | Improved |
| Use μP instead of standard parameterization (ablation) | MLP · Quadratic polynomial task | training-loss consistency across widths | Improved |
| Use μP instead of standard parameterization (ablation) | CNN · CIFAR-5m | large-learning-rate loss-curve consistency | Improved |
| Use μP instead of standard parameterization (ablation) | Transformer · Wikitext-103 | training-loss consistency across widths | Improved |
| Use finite-width kernels, which shift more task weight into slower eigenmodes | Ensembled eNTK and trained ResNet after-kernel · Polynomial regression and CIFAR-5m | kernel-flow loss | Hurt |
Closest papers by meaning
- On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling (NeurIPS 2025 · similarity 0.93)
- Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks (NeurIPS 2026 · similarity 0.92)
- Three Mechanisms of Feature Learning in a Linear Network (ICLR 2025 · similarity 0.90)
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.89)
- On the Diminishing Returns of Width for Continual Learning (ICML 2024 · similarity 0.89)
- Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
