Research map / Architectures and efficiency / RNNs and normalization
Weight normalization and decay interplay
Studies weight decay and weight normalization dynamics, including periodic training behavior and sparse training skew.
32 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in RNNs and normalization, keyword cluster “normalization · bn · mini”
State History Tried Checklist Preprints Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
5 papers in 2023–24 and 4 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 0.8% to 0.4%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · On Calibration of Modern Neural Networks (ICML 2017 · 2,519 citations)
- 2018 · GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks (ICML 2018 · 450 citations)
- 2019 · Riemannian batch normalization for SPD neural networks (NeurIPS 2019 · 53 citations)
- 2020 · Continual Learning With Extended Kronecker-Factored Approximate Curvature (CVPR 2020 · 58 citations)
- 2021 · Fast Certified Robust Training with Short Warmup (NeurIPS 2021 · 8 citations)
- 2022 · Set Norm and Equivariant Skip Connections: Putting the Deep in Deep Sets (ICML 2022 · 8 citations)
- 2024 · Batch Normalization Alleviates the Spectral Bias in Coordinate Networks (CVPR 2024 · 19 citations)
- 2025 · Dynamic Group Normalization: Spatio-Temporal Adaptation to Evolving Data Statistics (CVPR 2025 · 1 citation)
- 2026 · Batch Normalization for Neural Networks on Complex Domains (ICML 2026 · 0 citations)
Earliest papers here
- On Calibration of Modern Neural Networks (ICML 2017 · 2,519 citations)
- Linking Image and Text With 2-Way Nets (CVPR 2017 · 191 citations)
- Improving Training of Deep Neural Networks via Singular Value Bounding (CVPR 2017 · 80 citations)
What has been tried
28 papers here report 327 measured changes (a component added, removed or swapped, compared with a baseline): helped 62%, no clear effect 9%, hurt 24%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add batch normalization to ReLU | 2D Image, [13×49] · PSNR | 24.33 -> 29.08 | Helped | Batch Normalization Alleviates the Spectral Bias in Coordinate Networks (CVPR 2024 · 19 citations) |
| Remove momentum | GPT2-124M training · validation loss | Hurt | Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training (NeurIPS 2024 · 2 citations) | |
| add partial reassignment (p > 0) | CIFAR-100 · accuracy | 70.34 (0.63) -> 71.02 (0.64) | Helped | Dynamic Group Normalization: Spatio-Temporal Adaptation to Evolving Data Statistics (CVPR 2025 · 1 citation) |
| Use LCM-(0.5) rather than LCM-(1) | FPHA · accuracy | 77.64±1.00 -> 86.33±0.43 | Helped | A Lie Group Approach to Riemannian Batch Normalization (ICLR 2024 · 0 citations) |
| Increase the learning rate | OpenWebText, temperature experiment · GNS | Mixed | Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers (NeurIPS 2024 · 0 citations) | |
| add TA-BN | CIFAR-10 · accuracy | 0.848 -> 0.862 | Helped | Improving Neural ODE Training with Temporal Adaptive Batch Normalization (NeurIPS 2024) |
What reviewers will expect
Datasets and baselines named in the 6 papers of this approach from 2024–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-10 · 3 of 6 papers
- CIFAR-100 · 2 of 6 papers
- OpenWebText · 2 of 6 papers
Recent preprints
Loading recent arXiv preprints…
Use it
Download BibTeX of 28 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Batch normalization improvements (42 papers)
- Normalization layer theory (37 papers)
Elsewhere in the topic
- Efficient network compression and training (229 papers)
- Theory of depth and width (223 papers)
- Representation geometry and collapse (169 papers)
- Dataset distillation and data-efficient training (158 papers)
- Neural scaling laws (155 papers)
- Distributed LLM inference systems (147 papers)
- Layer-wise analysis of language models (145 papers)
- Dataset condensation and flow estimation (126 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).