Research map / Large language models / Instruction and fine-tuning
Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Mix-LN, a hybrid normalization that applies Post-LN to early layers and Pre-LN to deeper layers, arguing this yields healthier gradients and better use of deep layers in LLM pre-training, SFT, and RLHF.
Problem
Deeper layers in Pre-LN LLMs contribute little (they can be pruned with little loss), which the authors attribute to Pre-LN's small gradients in deep layers, while Post-LN causes gradient vanishing in early layers and training instability at scale.
Approach
Mix-LN uses Post-LN for the first a*L layers and Pre-LN for the remaining layers within one model, combining Post-LN's large deep-layer gradients with Pre-LN's early-layer gradient flow; a=0.25 is chosen for most sizes and 6.25% for 7B.
Key results
- Mix-LN achieves lower C4 perplexity than Pre-LN across LLaMA 71M, 130M, 250M, and 1B (e.g., 33.12 vs 34.77 at 71M; 18.18 vs 18.65 at 1B), while Post-LN diverges at 1B.
- On Commonsense170K SFT, Mix-LN averages 41.34 vs 36.93 for Pre-LN at LLaMA-250M (+4.26 points on average) and 44.66 vs 43.01 at LLaMA-1B.
- RLHF final reward at LLaMA-1B is 1.32 for Mix-LN vs 0.75 for Pre-LN; ViT-Small ImageNet accuracy is 76.40 vs 75.99 for Pre-LN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Pre-LN with Mix-LN | LLaMA pre-training · LLaMA-71M, LLaMA-130M, LLaMA-250M, and LLaMA-1B | perplexity: 34.77 -> 33.12; 26.78 -> 26.07; 21.92 -> 21.39; 18.65 -> 18.18 | Helped |
| Replace Pre-LN with Mix-LN | LLaMA-250M SFT · Commonsense170K downstream tasks | Avg.: 36.93 -> 41.34 | Helped |
| Replace Pre-LN with Mix-LN | LLaMA-1B SFT · Commonsense170K downstream tasks | Avg.: 43.01 -> 44.66 | Helped |
| Replace Pre-LN with Mix-LN | LLaMA-1B RLHF · ultrafeedback | final reward: 0.75 -> 1.32 | Helped |
| Replace Pre-LN with Mix-LN | ViT · ImageNet-1K, ViT-Tiny and ViT-Small | accuracy: 67.30 -> 67.34; 75.99 -> 76.40 | Helped |
| Increase Post-LN ratio from 25.0% to 33.0% (ablation) | Mix-LN on LLaMA-1B · C4 pre-training | perplexity: 18.18 -> 18.41 | Hurt |
| Increase Post-LN ratio from 25.0% to 50.0% (ablation) | Mix-LN on LLaMA-1B · C4 pre-training | perplexity: 18.18 -> 18.86 | Hurt |
| Add Scaled Initialization (ablation) | Mix-LN on LLaMA-130M · pre-training | perplexity: 29.95 -> 29.77 | Helped |
| Add Scaled Embed to Scaled Initialization (ablation) | Mix-LN on LLaMA-130M · pre-training | perplexity: 29.77 -> 31.19 | Hurt |
| Use Post-LN alone instead of Mix-LN | LLaMA pre-training · LLaMA-250M and LLaMA-1B | perplexity: 21.39 vs. 1409.09; 18.18 vs. 1411.54 | Hurt |
| Use Mix-LN instead of Pre-LN | LLaMA-130M representations · layer representations | angular distance | Helped |
Surprising findings
- Post-LN diverges at 250M and 1B scale in pre-training, while Mix-LN remains stable.
- Mix-LN needed a much smaller Post-LN ratio (6.25%) at 7B and became more sensitive to alpha at that scale.
Limitations
- Mix-LN becomes unstable and more sensitive to the Post-LN ratio at 7B scale, and 7B training was only run for 13K steps due to compute constraints.
- The optimal Post-LN ratio alpha is chosen by sweep and differs across scales; not tuned per model.
- Final layers still receive smaller gradients under the Pre-LN portion of the network.
Experimental setup
Datasets: C4 Commonsense170K UltraFeedback ImageNet-1K SQuAD v1.1 MMLU ARC-e
Benchmarks: C4 perplexity MMLU BoolQ ARC-e PIQA Hellaswag OBQA Winogrande SQuAD v1.1 ImageNet-1K
Metrics: perplexity accuracy average accuracy reward angular distance performance drop gradient norm
Baselines: Post-LN Pre-LN DeepNorm Admin Sandwich-LN Group-LN
Builds on: Pre-LN Post-LN DeepNorm Admin Sandwich-LN
Closest papers by meaning
- The Curse of Depth in Large Language Models (NeurIPS 2025 · similarity 0.90)
- The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.89)
- GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling (NeurIPS 2025 · similarity 0.89)
- Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation (NeurIPS 2025 · similarity 0.88)
- Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.88)
- Streamlining Redundant Layers to Compress Large Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.