Research map / Large language models / Instruction and fine-tuning

The Curse of Depth in Large Language Models

NeurIPS 2025 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies the "Curse of Depth" in modern LLMs (deep Pre-LN layers are near-identity) and proposes LayerNorm Scaling (LNS), which scales LN output by 1/sqrt(depth), improving pre-training and SFT across 130M–7B models.

Problem

Deeper Transformer layers in modern Pre-LN LLMs contribute much less than expected (removable or near-identity), wasting training resources; the root cause and a fix were unclear.

Approach

Pre-LN output variance grows exponentially with depth, making deep block Jacobians near identity; LayerNorm Scaling multiplies the LN output by 1/sqrt(layer index), which bounds variance growth to polynomial and makes deep layers contribute, with no extra parameters or hyperparameters.

Key results

• LayerNorm Scaling lowers LLaMA-130M perplexity from 26.73 (Pre-LN) to 25.76 and LLaMA-1B from 17.02 to 15.71.
• On OLMo-7B trained for 20B tokens, LNS reduces final training loss from 2.69 to 2.50.
• On LLaMA-250M SFT, LNS averages 43.14 vs 36.93 for Pre-LN and 41.34 for Mix-LN across eight tasks.

Measured effects reported in the paper

Surprising findings

• Combining LNS with Scaled Initialization yields worse perplexity than LNS alone (25.80 vs 25.76 on LLaMA-130M).
• Learnable-scalar methods such as LayerScale and SkipInit consistently degrade LLM pre-training performance.

Limitations

• Theoretical analysis relies on simplifying assumptions (Gaussian weights, approximately uniform softmax attention).
• Mix-LN and Post-LN instability results depend on the chosen training setup; scaled-up runs for 1B and 7B were terminated early before the 100B-token schedule finished.
• Optimal LNS insertion position differs for vision transformers (after Attn/MLP), so the recipe does not transfer directly across modalities.

Experimental setup

Datasets: C4 Commonsense170K ImageNet-1K

Benchmarks: MMLU SQuAD v1.1 ARC-e Commonsense170K downstream tasks (MMLU, BoolQ, ARC-e, PIQA, Hellaswag, OBQA, Winogrande) ImageNet-1K

Metrics: perplexity performance drop angular distance top-1 accuracy training loss

Baselines: Post-LN DeepNorm Mix-LN Pre-LN LayerScale SkipInit Depth-Scaled Initialization Scaled Initialization Admin Sandwich-LN Group-LN

Builds on: Pre-LN Post-LN Scaled Initialization Mix-LN OLMo

Closest papers by meaning

• Do Language Models Use Their Depth Efficiently? (NeurIPS 2025 · similarity 0.92)
• Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.90)
• Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN (ICLR 2025 · similarity 0.90)
• GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling (NeurIPS 2025 · similarity 0.90)
• Building on Efficient Foundations: Effective Training of LLMs with Structured Feedforward Layers (NeurIPS 2024 · similarity 0.90)
• The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.90)