Research map / Architectures and efficiency / RNNs and normalization
Neural scaling laws
Fits scaling laws for model size, data, weight decay, and merging in language and vision.
155 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in RNNs and normalization, keyword cluster “memory · scaling · cost”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
39 papers in 2023–24 and 93 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 6.0% to 10.0%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Efficient softmax approximation for GPUs (ICML 2017 · 137 citations)
- 2018 · Efficient Neural Audio Synthesis (ICML 2018 · 72 citations)
- 2019 · Contextual Memory Trees (ICML 2019 · 1 citation)
- 2020 · DynaBERT: Dynamic BERT with Adaptive Width and Depth (NeurIPS 2020 · 118 citations)
- 2021 · LRC-BERT: Latent-representation Contrastive Knowledge Distillation for Natural Language Understanding (AAAI 2021 · 55 citations)
- 2022 · Beyond neural scaling laws: beating power law scaling via data pruning (NeurIPS 2022 · 84 citations)
- 2023 · AdaTask: A Task-Aware Adaptive Learning Rate Approach to Multi-Task Learning (AAAI 2023 · 57 citations)
- 2024 · Training Compute-Optimal Protein Language Models (NeurIPS 2024 · 19 citations)
- 2025 · ScaMo: Exploring the Scaling Law in Autoregressive Motion Generation Model (CVPR 2025 · 6 citations)
- 2026 · TabICLv2: A Better, Faster, Scalable, and Open Tabular Foundation Model (ICML 2026 · 2 citations)
Earliest papers here
- Efficient softmax approximation for GPUs (ICML 2017 · 137 citations)
- Improving Neural Language Models with a Continuous Cache (ICLR 2017 · 92 citations)
- Efficient Neural Audio Synthesis (ICML 2018 · 72 citations)
What has been tried
98 papers here report 1,130 measured changes (a component added, removed or swapped, compared with a baseline): helped 59%, no clear effect 14%, hurt 22%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Repeat UR50/S across epochs instead of using diverse, approximately single-epoch UniMeta200B data | OOD generalization and training loss · generalization | Hurt | Training Compute-Optimal Protein Language Models (NeurIPS 2024 · 19 citations) | |
| Increase the synthetic-to-real data ratio from 1:3 to 1:5 | 6 common test benchmarks · average word accuracy | 96.85 -> 95.70 | Hurt | An Empirical Study of Scaling Law for Scene Text Recognition (CVPR 2024 · 11 citations) |
| Use a T5-XL text encoder with text-prefix design instead of CLIP without prefix | HumanML3D · FID | 0.226 -> 0.104 | Helped | ScaMo: Exploring the Scaling Law in Autoregressive Motion Generation Model (CVPR 2025 · 6 citations) |
| extract training features with the leave-one-fold-out strategy rather than vanilla extraction | 29 classification datasets · average rank | 5.97 -> 2.16 | Helped | A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities (NeurIPS 2025 · 6 citations) |
| Replace HSTU pointwise attention with softmax | Synthetic streaming · HR@10 | .0893 -> .0617 | Hurt | Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations (ICML 2024 · 5 citations) |
| add atom and bond features | COMPAS-1D · MAE aEA | 0.0103(4e-04) -> 0.0093(4e-05) | Helped | Exploring Molecular Pretraining Model at Scale (NeurIPS 2024 · 5 citations) |
| Reduce patch size to 6 | ImageNet-1 10-shot · performance | some absolute gains | Mixed | Navigating Scaling Laws: Compute Optimality in Adaptive Model Training (ICML 2024 · 4 citations) |
| include the 1.4B model in the downstream-error scaling fit | RedPajama 6.9B, 138B tokens; 17-task split · average top-1 error relative prediction error | 10.64% -> 0.05% | Helped | Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · 3 citations) |
| Use LVPS instead of no sampling | BERT-Base training · acceleration ratio | +9.3% | Helped | Preparing Lessons for Progressive Training on Language Models (AAAI 2024 · 3 citations) |
| Use the optimal-hyperparameter scaling law instead of fixed suboptimal hyperparameters | Validation set · mean squared error | 4.31 ⇥ 10−7 -> 2.35 ⇥ 10−7 | Helped | Towards Precise Scaling Laws for Video Diffusion Transformers (CVPR 2025 · 2 citations) |
| Use FP8 optimizer moments | 390M-token setup · final loss | 2.649 -> 2.645 | No clear effect | Towards Fully FP8 GEMM LLM Training at Scale (NeurIPS 2025 · 2 citations) |
| Add vocabulary sorting | Gemma 2 (2B), Alpaca, 8192 tokens · Loss+Gradient time | 159 ms -> 145 ms | Helped | Cut Your Losses in Large-Vocabulary Language Models (ICLR 2025 · 2 citations) |
What reviewers will expect
Datasets and baselines named in the 52 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- C4 · 10 of 52 papers
- FineWeb-Edu · 7 of 52 papers
- OpenWebText · 6 of 52 papers
- SlimPajama · 4 of 52 papers
- DCLM-Baseline · 3 of 52 papers
- ImageNet-1k · 2 of 52 papers
- LibriSpeech · 2 of 52 papers
- Paloma · 2 of 52 papers
Baselines
- Chinchilla scaling law · 5 of 52 papers
- OLMo2 · 2 of 52 papers
- cosine schedule · 2 of 52 papers
- WSD · 2 of 52 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 132 papers; from OpenAlex and the papers' affiliations).
- Google · 10 papers
- Massachusetts Institute of Technology · 7 papers
- Meta · 7 papers
- Korea Advanced Institute of Science and Technology · 5 papers
- Huawei · 4 papers
- Tsinghua University · 4 papers
- Microsoft · 4 papers
- University of Washington · 4 papers
- NVIDIA · 4 papers
- University of Oxford · 4 papers
Use it
Download BibTeX of 39 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Efficient network compression and training (229 papers)
- Dataset distillation and data-efficient training (158 papers)
- Distributed LLM inference systems (147 papers)
- Dataset condensation and flow estimation (126 papers)
- Scalable approximation and equivariant compute (120 papers)
Elsewhere in the topic
- Theory of depth and width (223 papers)
- Representation geometry and collapse (169 papers)
- Layer-wise analysis of language models (145 papers)
- Differentiable optimization layers (125 papers)
- Non-autoregressive translation (102 papers)
- RNN training dynamics and gradients (102 papers)
- Neural-symbolic reasoning (98 papers)
- Sparse dictionary representations (98 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).