Research map / Architectures and efficiency / RNNs and normalization
PowerNorm: Rethinking Batch Normalization in Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace LN with PN | Pre-Norm Transformer · IWSLT14 De-En test | BLEU: 35.5 -> 35.9 | Helped |
| Replace LN with PN | Pre-Norm Transformer · WMT14 En-De test | BLEU: 29.5 -> 30.1 | Helped |
| Replace BN with PN | Pre-Norm Transformer · IWSLT14 De-En test | BLEU: 34.4 -> 35.9 | Helped |
| Replace BN with PN | Pre-Norm Transformer · WMT14 En-De test | BLEU: 28.1 -> 30.1 | Helped |
| Relax zero-mean normalization and use quadratic mean instead of variance (PN-V) (ablation) | Transformer with BN · IWSLT14 De-En test | BLEU: 34.4 -> 35.5 | Helped |
| Relax zero-mean normalization and use quadratic mean instead of variance (PN-V) (ablation) | Transformer with BN · WMT14 En-De test | BLEU: 28.1 -> 28.5 | Helped |
| Use running quadratic-mean statistics rather than per-batch quadratic-mean statistics (ablation) | PN-V Transformer · IWSLT14 De-En test | BLEU: 35.5 -> 35.9 | Helped |
| Use running quadratic-mean statistics rather than per-batch quadratic-mean statistics (ablation) | PN-V Transformer · WMT14 En-De test | BLEU: 28.5 -> 30.1 | Helped |
| Replace LN with PN | Tensor-Transformer 1core · PTB test | PPL: 53.2* -> 47.6 | Helped |
| Replace LN with PN | Tensor-Transformer 1core · WikiText-103 test | PPL: 20.9* -> 17.9 | Helped |
| Replace BN with PN | Tensor-Transformer 1core · PTB test | PPL: 60.7 -> 47.6 | Helped |
| Replace BN with PN | Tensor-Transformer 1core · WikiText-103 test | PPL: 27.2 -> 17.9 | Helped |
Closest papers by meaning
- Understanding the Failure of Batch Normalization for Transformers in NLP (NeurIPS 2022 · similarity 0.95)
- HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization (NeurIPS 2025 · similarity 0.89)
- Peri-LN: Revisiting Normalization Layer in the Transformer Architecture (ICML 2025 · similarity 0.88)
- SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization (ICML 2024 · similarity 0.88)
- Reducing Divergence in Batch Normalization for Domain Adaptation (AAAI 2025 · similarity 0.88)
- The Curse of Depth in Large Language Models (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.