Research map / Architectures and efficiency / RNNs and normalization

Normalization layer theory

Explains generalization and nonlinearity of normalization layers and removes them with careful initialization.

37 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in RNNs and normalization, keyword cluster “normalization · bn · mini”

State History Tried Checklist Preprints Who Export

Where it stands

Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.

7 papers in 2023–24 and 5 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 1.1% to 0.5%.

How this is decided

A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.

How it got here

Most cited paper of each year

2016 · Shakeout: A New Regularized Deep Neural Network Training Scheme (AAAI 2016 · 51 citations)
2017 · Self-Normalizing Neural Networks (NeurIPS 2017 · 511 citations)
2018 · Regularizing by the Variance of the Activations' Sample-Variances (NeurIPS 2018 · 8 citations)
2019 · Understanding and Improving Layer Normalization (NeurIPS 2019 · 175 citations)
2020 · On Layer Normalization in the Transformer Architecture (ICML 2020 · 105 citations)
2021 · Why Spectral Normalization Stabilizes GANs: Analysis and Improvements (NeurIPS 2021 · 29 citations)
2022 · Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction (NeurIPS 2022 · 12 citations)
2023 · On the impact of activation and normalization in obtaining isometric embeddings at initialization (NeurIPS 2023 · 3 citations)
2024 · Normalization and effective learning rates in reinforcement learning (NeurIPS 2024 · 2 citations)
2025 · Till the Layers Collapse: Compressing a Deep Neural Network Through the Lenses of Batch Normalization Layers. (AAAI 2025 · 3 citations)
2026 · Bounded Hyperbolic Tangent: A Stable and Efficient Alternative to Pre-Layer Normalization in Large Language Models (ICML 2026 · 0 citations)

Earliest papers here

Shakeout: A New Regularized Deep Neural Network Training Scheme (AAAI 2016 · 51 citations)
Self-Normalizing Neural Networks (NeurIPS 2017 · 511 citations)
The Shattered Gradients Problem: If resnets are the answer, then what is the question? (ICML 2017 · 96 citations)

What has been tried

35 papers here report 389 measured changes (a component added, removed or swapped, compared with a baseline): helped 63%, no clear effect 14%, hurt 21%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.

Ablations in recent papers

What was changed	Measured on	Result	Effect	Paper
use TLC with SiLU activation	CIFAR-10 · top-1 accuracy	92.22 -> 91.72	Hurt	Till the Layers Collapse: Compressing a Deep Neural Network Through the Lenses of Batch Normalization Layers. (AAAI 2025 · 3 citations)
add weight projection to normalization	CIFAR-10 test · top-1 accuracy	94.47 -> 94.64	Helped	Normalization and effective learning rates in reinforcement learning (NeurIPS 2024 · 2 citations)
add QK-Norm	1.5B pre-training · final training loss	2.722 -> 2.711	Helped	Peri-LN: Revisiting Normalization Layer in the Transformer Architecture (ICML 2025 · 0 citations)
scale one constant normalization factor by 0.5	OpenWebText, 0.5B, 10B tokens · validation loss	2.638 -> 2.647	Hurt	Learning in Compact Spaces with Approximately Normalized Transformer (NeurIPS 2025 · 0 citations)
use identity rather than shaped sin or tanh activations	CIFAR10 training · training accuracy		Hurt	Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion (ICLR 2024 · 0 citations)
Replace LN with BN	CIFAR-10-RL and MNIST-RL · training accuracy		Hurt	On the Nonlinearity of Layer Normalization (ICML 2024 · 0 citations)
Use scaled additive Gaussian noise rather than Gaussian dropout	STL10 · validation accuracy		Helped	On the Importance of Gaussianizing Representations (ICML 2025)

What reviewers will expect

Datasets and baselines named in the 7 papers of this approach from 2024–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.

Datasets

CIFAR-10 · 5 of 7 papers
Tiny-ImageNet · 2 of 7 papers
ImageNet · 2 of 7 papers
CIFAR100 · 2 of 7 papers
MNIST · 2 of 7 papers
C4 · 2 of 7 papers

Recent preprints

Loading recent arXiv preprints…

Who publishes on it

Institutions on the most papers of this approach since 2023 (of 12 papers; from OpenAlex and the papers' affiliations).

Ecole Normale Superieure · 2 papers

Use it

Download BibTeX of 28 key papers Open the topic in the interactive map

Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.

Same keyword cluster

Batch normalization improvements (42 papers)
Weight normalization and decay interplay (32 papers)

Elsewhere in the topic

Efficient network compression and training (229 papers)
Theory of depth and width (223 papers)
Representation geometry and collapse (169 papers)
Dataset distillation and data-efficient training (158 papers)
Neural scaling laws (155 papers)
Distributed LLM inference systems (147 papers)
Layer-wise analysis of language models (145 papers)
Dataset condensation and flow estimation (126 papers)

Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).