Research map / Architectures and efficiency / RNNs and normalization
On the Nonlinearity of Layer Normalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows theoretically that layer normalization is nonlinear via a linear-invariant SSR bound, that LN-Nets with width 3 and O(m) LN layers can fit any binary labeling (VC lower bound L+2), and that group-based LN amplifies nonlinearity, with small experiments on random-label fitting, ResNet-NA and Transformer/ViT.
Problem
Theoretical understanding of layer normalization has focused on scale invariance and optimization; its nonlinearity and representation capacity are largely unexplored.
Approach
Defines a linear-invariant lower bound (LSSR) on the sum-of-squares ratio between two classes; shows linear networks cannot break it but linear+LN can, establishing LN nonlinearity; constructs width-3 LN-Nets that shatter any labeling via geometric projection-merge algorithms; and proposes group-based LN (LN-G) that amplifies nonlinearity measured by a Hessian-based indicator.
Key results
- LN-Net reaches 55.85% training accuracy on CIFAR-10-RL versus 18.51% upper bound for a linear classifier; linear networks never exceed it.
- LN-G-Position on ResNet-NA (CIFAR-10) reaches 86.66% test accuracy versus 62.85% for LN and 39.3% for BN.
- Transformer on IWSLT14 De-En with LN-G yields BLEU 35.23 vs 35.01 for LN; Tiny-ViT on CIFAR-10 yields 89.26% vs 88.81%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace LN with LN-G | Transformer · IWSLT14 German-to-English | BLEU: 35.01 ± 0.10 -> 35.23±0.07 | Helped |
| Replace all LNs with LN-G | Tiny-VIT · CIFAR-10 test | accuracy: 88.81% -> 89.26% | Helped |
| Use LN-Net instead of a linear classifier | 256-neuron network · CIFAR-10-RL training set | accuracy: 18.51 % -> 55.85% | Helped |
| Use LN-Net instead of a linear classifier | 256-neuron network · MNIST-RL training set | accuracy: 15.38% -> 19.44% | Helped |
| Replace LN with BN (ablation) | LN-Net · CIFAR-10-RL and MNIST-RL | training accuracy | Hurt |
| Use group-based LN with group number 32 (ablation) | LN-Net · CIFAR-10-RL and MNIST-RL training sets | training accuracy | Helped |
| Increase the LN-G group number to 128 (ablation) | LN-Net · CIFAR-10-RL and MNIST-RL training sets | training accuracy | Hurt |
| Increase GN group number (ablation) | ResNet-NA · CIFAR-10 | test accuracy | Helped |
| Use LN-G-Position instead of GN (ablation) | ResNet-NA · CIFAR-10 | test accuracy | Helped |
| Increase LN-G-Position group number to 64 (ablation) | ResNet-NA · CIFAR-10 | test accuracy | Hurt |
| Replace LN with LN-G-Position | ResNet-NA · CIFAR-10 | test accuracy: 62.85% -> 86.66% | Helped |
Surprising findings
- BN-based LN-Net fails to exceed the linear classifier bound on random-label fitting, unlike LN.
- Plain GN and LN-G variants differ sharply in performance with group number, and LN-G with too-small groups (d/g<3) degrades.
Limitations
- Representation capacity results are loose, analogous to early universal approximation theory.
- LN-G effectiveness is only verified on small-scale networks and datasets.
- The Hessian-based nonlinearity measure is an assumed indicator not well verified.
Experimental setup
Datasets: CIFAR-10 MNIST IWSLT14 German-to-English
Benchmarks: CIFAR-10 MNIST CIFAR-10-RL MNIST-RL IWSLT14 De-En
Metrics: training accuracy test accuracy BLEU SSR/LSSR Hessian-based nonlinearity indicator
Baselines: LN BN GN IN linear classifier ResNet-NA without normalization
Builds on: Batch Normalization Group Normalization RMSNorm fairseq
Closest papers by meaning
- On the impact of activation and normalization in obtaining isometric embeddings at initialization (NeurIPS 2023 · similarity 0.88)
- Optimization Theory for ReLU Neural Networks Trained with Normalization Layers (ICML 2020 · similarity 0.88)
- Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction (NeurIPS 2022 · similarity 0.88)
- Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learning (NeurIPS 2021 · similarity 0.87)
- Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion (ICLR 2024 · similarity 0.86)
- Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $\mu$ Parametrization (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.