Research map / Architectures and efficiency / Attention and transformers
HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HybridNorm, which uses QKV-Norm in attention and Post-Norm in the FFN within each transformer block, and shows more stable training and better downstream performance than Pre-Norm on dense (151M–1.2B, 7B) and MoE-1B-7B models.
Problem
Pre-Norm transformers train stably but often underperform Post-Norm, while Post-Norm gives better final performance but is hard to train at depth; the paper seeks a normalization placement that gets both.
Approach
Combine QKV normalization in the multi-head attention (normalizing Q, K, V individually) with Post-Norm in the FFN inside each block, plus a special Pre-Norm treatment for the first block (HybridNorm*), supported by a gradient-norm coupling analysis.
Key results
- On 1.2B dense models trained on 1T tokens, HybridNorm* averages 64.15 vs 62.99 for Pre-Norm across downstream tasks.
- On 7B dense models (150B tokens), HybridNorm* improves average accuracy from 60.61% to 63.06% and lowers C4 perplexity from 15.32 to 14.83.
- On MoE-1B-7B (500B tokens), HybridNorm* averages 66.12 vs 64.95 for Pre-Norm; Post-Norm diverges at 29 layers while HybridNorm trains stably.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the HybridNorm structure: QKV-Norm in attention and Post-Norm in FFN | Dense 1.2B · Average, 1T tokens | accuracy: 62.99 -> 63.25 | Helped |
| Add the specialized first-block treatment (ablation) | HybridNorm on Dense 1.2B · Average, 1T tokens | accuracy: 63.25 -> 64.15 | Helped |
| Add the specialized first-block treatment | Pre-Norm on MoE-1B-7B · Average, 500B tokens | accuracy: 64.95 -> 66.12 | Helped |
| Add the specialized first-block treatment | Pre-Norm on Dense 7B · Average, 150B tokens | accuracy: 60.61 -> 63.06 | Helped |
| Use Megatron initialization instead of Normal initialization (ablation) | HybridNorm on Dense 550M · C4, 400B tokens | perplexity: 20.44 -> 20.00 | Helped |
| Use QKV-Norm instead of QK-Norm (ablation) | Post-Norm in FFN on Dense 550M · HellaSwag, 400B tokens | accuracy: 52.26 -> 52.68 | Helped |
| Use QK-Norm instead of QKV-Norm (ablation) | Post-Norm in FFN on Dense 550M · HellaSwag, 400B tokens | accuracy: 53.35 -> 52.29 | Hurt |
| Add Pre-Norm in MHA while retaining QKV-Norm and Post-Norm in FFN (ablation) | HybridNorm on Dense 550M · HellaSwag, 400B tokens | accuracy: 53.35 -> 52.65 | Hurt |
| Use Pre-Norm instead of Post-Norm in FFN (ablation) | QKV-Norm attention on Dense 550M · HellaSwag, 400B tokens | accuracy: 53.35 -> 52.57 | Hurt |
| Use Pre-Norm in MHA and Post-Norm in FFN instead of Post-Norm in MHA and Pre-Norm in FFN (ablation) | Dense 550M · HellaSwag, 400B tokens | accuracy: 51.19 -> 52.42 | Helped |
| Replace RMSNorm with LayerNorm (ablation) | Dense 550M Pre-Norm · Training, 400B tokens | loss: 2.7631 -> 2.7639 | No clear effect |
| Add normalization after the embedding (ablation) | HybridNorm first-block variants · Training loss and downstream performance | training loss | No clear effect |
Surprising findings
- Post-Norm fails to converge at 29 layers while HybridNorm remains stable at the same depth.
- EmbedNorm (normalization after embedding) did not help, unlike the first-block special treatment.
Limitations
- Gains on several benchmarks are small (often under 1 point) and some tasks favor Pre-Norm or other variants.
- Theoretical gradient analysis is for a single-head attention layer and relies on bounds with a spectral-norm argument; the link to final performance is largely empirical.
- Most analysis is on 550M dense models; the 7B experiment is limited to 150B tokens and an extra normalization layer was added.
Experimental setup
Datasets: OLMoE Mix C4 Pile Books Common Crawl peS2o Reddit Stack Wikipedia Wikitext-103
Benchmarks: BasicArithmetic HellaSwag SciQ ARC-C ARC-E PIQA OpenbookQA COPA WinoGrande BoolQ MMLU CoQA
Metrics: accuracy training loss validation perplexity gradient norm
Baselines: Pre-Norm Post-Norm Mix-LN QK-Norm Post-Pre Pre-Post DeepNorm
Builds on: QKV-Norm QK-Norm Mix-LN OLMoE Megatron initialization
Closest papers by meaning
- Peri-LN: Revisiting Normalization Layer in the Transformer Architecture (ICML 2025 · similarity 0.90)
- SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization (ICML 2024 · similarity 0.89)
- The Curse of Depth in Large Language Models (NeurIPS 2025 · similarity 0.89)
- Impact of Layer Norm on Memorization and Generalization in Transformers (NeurIPS 2025 · similarity 0.88)
- A Multi-Level Framework for Accelerating Training Transformer Models (ICLR 2024 · similarity 0.87)
- Building on Efficient Foundations: Effective Training of LLMs with Structured Feedforward Layers (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.