Research map / Architectures and efficiency / RNNs and normalization
Impact of Layer Norm on Memorization and Generalization in Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that removing LayerNorm's learnable parameters has opposite effects on Pre-LN (disrupts learning, worsens memorization) and Post-LN transformers (suppresses noisy-label memorization and recovers true labels), with early-layer LNs most influential, supported by gradient-norm analysis.
Problem
How LayerNorm affects learning versus label memorization in Pre-LN and Post-LN transformers remains unclear, especially which LN layers matter most.
Approach
Removes only LayerNorm's learnable affine parameters (keeping normalization) in Pre-LN and Post-LN models trained on noisy-label data, measuring learning accuracy, memorization, recovery and random-prediction scores; explains the divergent effects via ratios of learning vs memorization input-gradient norms and upper bounds on LN-input gradients.
Key results
- Removing LN parameters in Pre-LN models disrupts learning and widens the train-test gap, while in Post-LN models it suppresses memorization and recovers genuine labels without hurting test accuracy.
- Early-layer LNs are the most influential: removing them most strongly disrupts learning in Pre-LN and most strongly mitigates memorization in Post-LN.
- Gradient analysis shows learning-to-memorization gradient norm ratio is much larger for Pre-LN than Post-LN across layers, with theoretical upper bounds decreasing from early to later LN layers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove learnable LayerNorm parameters (ablation) | Pre-LN transformers · News | test accuracy | Hurt |
| Remove learnable LayerNorm parameters (ablation) | Post-LN transformers · News | test accuracy | No clear effect |
| Remove learnable LayerNorm parameters (ablation) | Post-LN transformers · Noisy-label classification tasks | memorization score | Helped |
| Remove learnable LayerNorm parameters (ablation) | Pre-LN transformers · Noisy-label classification tasks | memorization score | No clear effect |
| Remove learnable LayerNorm parameters (ablation) | Pre-LN transformers · Noisy-label classification tasks | recovery score | Hurt |
| Remove early-layer LayerNorm parameters (ablation) | Pre-LN transformers · UTK-Face | test accuracy | Hurt |
| Remove early-layer LayerNorm parameters (ablation) | Post-LN transformers · Emotions | memorization score | Helped |
| Remove middle-layer LayerNorm parameters instead of early-layer parameters (ablation) | Post-LN transformers · Noisy-label classification tasks | memorization score | Mixed |
| Remove later-layer LayerNorm parameters instead of early-layer parameters (ablation) | Post-LN transformers · Noisy-label classification tasks | memorization score | Mixed |
| Remove middle-layer LayerNorm parameters instead of early-layer parameters (ablation) | Pre-LN transformers · Noisy-label classification tasks | test accuracy | Mixed |
| Remove later-layer LayerNorm parameters instead of early-layer parameters (ablation) | Pre-LN transformers · Noisy-label classification tasks | test accuracy | Mixed |
Surprising findings
- Removing LN parameters in Post-LN models mitigates memorization without hurting learning accuracy.
- Contrary to prior work suggesting LN contributes to overfitting in Pre-LN models, LN removal in Pre-LN did not suppress memorization.
Limitations
- Memorization studied only via 1% random label noise injection, which may not capture other memorization forms.
- Post-LN results are shown mainly for language models; vision is only evaluated with Pre-LN architectures in practice.
- Theoretical bounds are upper bounds and the learning-vs-memorization gradient comparison relies on approximations (e.g., near-zero softmax residuals).
Experimental setup
Datasets: CIFAR10 NICO++ UTK-Face Emotions News TweetTopic
Benchmarks: CIFAR10 NICO++ UTK-Face Emotions News TweetTopic
Metrics: learning (test) accuracy memorization score recovery score random prediction score train-test accuracy gap gradient norm ratio
Baselines: Pre-LN with LN parameters Post-LN with LN parameters
Builds on: Layer Normalization Pre-LN transformer Post-LN transformer label memorization
Closest papers by meaning
- HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization (NeurIPS 2025 · similarity 0.88)
- The Curse of Depth in Large Language Models (NeurIPS 2025 · similarity 0.88)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.88)
- Peri-LN: Revisiting Normalization Layer in the Transformer Architecture (ICML 2025 · similarity 0.88)
- Transformer Layers as Painters (AAAI 2025 · similarity 0.88)
- SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.