Research map / Architectures and efficiency / RNNs and normalization
On Layer Normalization in the Transformer Architecture
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove warm-up from Post-LN training with Adam (ablation) | Post-LN Transformer · IWSLT14 De-En | BLEU: around 34 -> 8.45 | Hurt |
| Use a shorter warm-up of 500 iterations instead of the usual 4000 (ablation) | Post-LN Transformer with Adam · IWSLT14 De-En | BLEU: 31.16 and 2.77 | Mixed |
| Remove warm-up and use Pre-LN instead of Post-LN | Transformer · IWSLT14 De-En | BLEU: 8.45 -> around 34 | Helped |
| Use Pre-LN rather than Post-LN | Transformer · IWSLT14 De-En | training speed: 9-th checkpoint -> 15-th checkpoint | Helped |
| Replace Adam with RAdam for Post-LN without warm-up (ablation) | Post-LN Transformer · IWSLT14 De-En | training stability | Helped |
| Replace Adam with RAdam for Pre-LN (ablation) | Pre-LN Transformer · IWSLT14 De-En | performance | No clear effect |
| Remove warm-up and use Pre-LN rather than Post-LN | BERT · BERT pre-training validation | training speed: 500k -> 700k updates; 40% | Helped |
| Train Post-LN without warm-up using a large learning rate (ablation) | Post-LN BERT · BERT pre-training | optimization stability | Hurt |
| Use a fixed small learning rate of 1e^{-4} instead of warm-up for Post-LN (ablation) | Post-LN Transformer · IWSLT14 De-En | convergence speed: 1e^{-4} | Hurt |
Closest papers by meaning
- Peri-LN: Revisiting Normalization Layer in the Transformer Architecture (ICML 2025 · similarity 0.93)
- HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization (NeurIPS 2025 · similarity 0.90)
- Transformer Layers as Painters (AAAI 2025 · similarity 0.88)
- Transformers without Normalization (CVPR 2025 · similarity 0.88)
- SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization (ICML 2024 · similarity 0.88)
- Viewing Transformers Through the Lens of Long Convolutions Layers (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.