Research map / Architectures and efficiency / RNNs and normalization
Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Peri-LN, which wraps each Transformer sub-layer with normalization on both its input and output, and shows via theory and pre-training up to 3.2B parameters that it yields more stable variance growth, gradient flow, and training than Pre-LN and Post-LN.
Problem
Choosing a layer normalization placement (Post-LN, Pre-LN) that stabilizes training and speeds convergence in large-scale Transformers; Pre-LN suffers from hidden-state variance blow-up ("massive activations") and gradient spikes, while Post-LN weakens gradient flow.
Approach
Applies normalization both before and after each sub-layer (Peri-LN, Output-LN on module output in addition to Pre-LN input norm), which bounds the magnitude of the residual-path additions and provides a damping factor on the final-layer gradient norm, unifying placements used in recent open models such as Gemma and OLMo.
Key results
- At 3.2B parameters, Peri-LN reaches 58.56 average benchmark score versus 56.69 for Pre-LN and 46.45 for Post-LN, averaged over 5 seeds.
- Peri-LN reduces the standard deviation of downstream benchmark scores across seeds by more than half relative to Pre-LN.
- Pre-LN activations at 3.2B surpass the FP16 maximum as early as 0.5B training tokens, while Peri-LN stays below it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Peri-LN instead of Pre-LN | 400M Transformer · language-model benchmarks | Avg. benchmark score: 49.69 -> 51.57 | Helped |
| use Peri-LN instead of Pre-LN | 1.5B Transformer · language-model benchmarks | Avg. benchmark score: 53.71 -> 56.55 | Helped |
| use Peri-LN instead of Pre-LN | 3.2B Transformer · language-model benchmarks | Avg. benchmark score: 56.69 -> 58.56 | Helped |
| use Peri-LN instead of Pre-LN | 400M Transformer · C4 evaluation | loss: 3.43 -> 3.34 | Helped |
| use Peri-LN instead of Pre-LN | 1.5B Transformer · C4 evaluation | loss: 3.29 -> 3.18 | Helped |
| use Peri-LN instead of Pre-LN | 3.2B Transformer · C4 evaluation | loss: 3.20 -> 3.11 | Helped |
| add QK-Norm (ablation) | Peri-LN · 1.5B pre-training | final training loss: 2.722 -> 2.711 | Helped |
| place LN at A + C rather than Peri-LN placements (ablation) | 400M Transformer · 30B-token pre-training | final training loss: 2.91 -> 3.01 | Hurt |
| place LN only at B rather than Peri-LN placements (ablation) | 400M Transformer · 30B-token pre-training | training stability | Hurt |
| increase weight decay coefficient from 0 to 0.33 (ablation) | 400M Pre-LN · pre-training | final training loss: 3.03 -> 3 | Helped |
| increase weight decay coefficient from 0 to 0.33 (ablation) | 400M Peri-LN · pre-training | final training loss: 2.94 -> 2.90 | Helped |
| increase initialization variance (ablation) | Pre-LN · pre-training | training stability | Hurt |
Surprising findings
- Pre-LN frequently diverges or shows early-stage gradient spikes across seeds, and lowering the learning rate did not substantially fix it.
- Increasing weight decay tenfold (0.033 to 0.33) prevents Pre-LN divergence but Pre-LN still does not close the performance gap with Peri-LN.
Limitations
- Experiments are limited to models up to 3.2B parameters trained on 30B tokens.
- Theoretical bounds are derived for a simplified single-MLP setting with RMSNorm and ReLU rather than the full Transformer.
- Code, data, and weights release status is not stated.
Experimental setup
Datasets: DCLM-baseline LIMA C4 Wikitext RedPajama-Data-1T OpenBookQA CommonSenseQA GLUE
Benchmarks: ARC-Easy HellaSwag PIQA SIQA Winogrande
Metrics: accuracy C4 evaluation loss training loss gradient norm activation variance angular distance
Baselines: Post-LN Pre-LN
Builds on: Pre-LN Post-LN RMSNorm Llama 3 DCLM
Closest papers by meaning
- Transformer Layers as Painters (AAAI 2025 · similarity 0.91)
- HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization (NeurIPS 2025 · similarity 0.90)
- Viewing Transformers Through the Lens of Long Convolutions Layers (ICML 2024 · similarity 0.89)
- Unraveling the Gradient Descent Dynamics of Transformers (NeurIPS 2024 · similarity 0.88)
- Impact of Layer Norm on Memorization and Generalization in Transformers (NeurIPS 2025 · similarity 0.88)
- From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamics (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.