Research map / Architectures and efficiency / RNNs and normalization
Revisiting Neural Scaling Laws in Language and Vision
Ibrahim Alabdulmohsin, Behnam Neyshabur, Xiaohua Zhai
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use M4 (with alpha) instead of M2 power law with offset | scaling law estimator · NMT 6 Enc 6 Dec | extrapolation RMSE: 3.9e-2 -> 1.0e-2 | Improved |
| use M4 instead of M3 | scaling law estimator · NMT 6 Enc 28 Dec | extrapolation RMSE: 1.6e-2 -> 3.0e-2 | Hurt |
| use M4 instead of M2 | scaling law estimator · NMT Transformer Enc/LSTM Dec | extrapolation RMSE: 1.3e-2 -> 1.2e-2 | No clear effect |
| use M4 instead of M2/M3 | scaling law estimator · language modeling | extrapolation RMSE: e.g. 1.68e7: 6.0e-4 -> 3.1e-4; 1.34e8: 6.6e-4 (M3) vs 1.9e-3 | Mixed |
| use M4 instead of M1 | scaling law estimator · image classification (72 tasks) | extrapolation RMSE | Improved |
| use M4 instead of M2 | scaling law estimator · BIG-Bench date_understanding | extrapolation RMSE: 3.2e-2 -> 1.5e-2 vs M3 4.7e-3 | Hurt |
| use M4 instead of M2 | scaling law estimator · BIG-Bench mult_data_wrangling 2-shot | extrapolation RMSE: 1.6e-2 -> 6.2e-3 | Improved |
| add learnable alpha parameter to M4 (ablation) | M4 · ImageNet 10-shot ViT/B/16 | extrapolation RMSE: 4.3e-2 -> 2.2e-2 | Improved |
| add learnable alpha parameter to M4 (ablation) | M4 · ImageNet 10-shot BiT/101/3 | extrapolation RMSE: 5.1e-2 -> 1.7e-2 | Improved |
| select scaling exponent by best fit vs best extrapolation | M2 on ViT/B/16 JFT-300M · ImageNet 10-shot | extrapolation RMSE: c=-0.24 best fit vs c=-0.40 best extrapolation | Mixed |
Closest papers by meaning
- Scaling Laws for Robust Comparison of Open Foundation Language-Vision Models and Datasets (NeurIPS 2025 · similarity 0.93)
- Neural Neural Scaling Laws (NeurIPS 2026 · similarity 0.92)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.91)
- On the Invariance and Generality of Neural Scaling Laws (NeurIPS 2026 · similarity 0.90)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.90)
- Scaling Law with Learning Rate Annealing (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
