Research map / Architectures and efficiency / RNNs and normalization
Don't be lazy: CompleteP enables compute-efficient deep transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CompleteP (alpha=1 residual scaling plus LN, bias, weight decay and AdamW epsilon corrections) that gives depth-wise hyperparameter transfer and complete feature learning, yielding 11.8-34.4% FLOP savings over muP at 1.5B non-embedding parameters.
Problem
Hyperparameter transfer across depth fails under standard parameterization and muP, forcing costly re-tuning or suboptimal training for deep transformers; existing depth-scaling parameterizations either lack transfer or remain in a lazy regime.
Approach
Scale residual branches by L^{-1} (alpha=1) and extend the parameterization with principled learning-rate, weight decay, and AdamW epsilon rescalings for LayerNorm, biases, and embeddings, justified by a new "complete feature learning" desideratum that rules out lazy layers for alpha<1.
Key results
- CompleteP (alpha=1) saves 11.8% FLOPs over muP at optimal N:L and 34.4% at the deepest (179-layer) 1.5B non-embedding models.
- Only alpha=1 achieves depth-wise learning rate and initialization std transfer from 2 to 128 layers.
- At 1.5B non-embedding parameters, CompleteP keeps N:L=11.8 within 1% of compute-optimal, versus N:L=38.7 for muP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CompleteP (α=1) rather than μP | pre-LN transformer language models · SlimPJ validation loss, 200 TPP, 50M and 300M models | validation loss: 2.867 -> 2.859; 2.973 -> 2.950; 2.451 -> 2.443; 2.537 -> 2.497 | Helped |
| Use CompleteP (α=1) rather than μP | 1.5B non-embedding parameter models · 20 TPP, optimal and deepest N:L | FLOP savings: 11.8%; 34.4% | Helped |
| Use CompleteP (α=1) rather than μP | 1.5B non-embedding parameter models · 20 TPP, downstream average, optimal and deepest N:L | accuracy: 54.3 ± 0.3 -> 55.2 ± 0.3; 52.0 ± 0.3 -> 54.3 ± 0.3 | Helped |
| Use α=0.5 instead of α=1 (ablation) | pre-LN transformer language models · SlimPJ validation loss, 200 TPP, 50M and 300M models | validation loss: 2.859 -> 2.874; 2.950 -> 2.998; 2.443 -> 2.455; 2.497 -> 2.534 | Hurt |
| Use α=0.5 instead of α=1 (ablation) | 1.5B non-embedding parameter models · 20 TPP, downstream average, optimal and deepest N:L | accuracy: 54.3 ± 0.3 -> 54.3 ± 0.3; 54.3 ± 0.3 -> 50.1 ± 0.3 | Mixed |
| Use CompleteP (α=1) rather than SP, μP, or α=0.5 without retuning learning rate | transformer models as depth varies · Depth-wise HP transfer | loss | Helped |
| Scale residual branch outputs and depth-dependent hyperparameters with α=0.5 rather than α=1 (ablation) | transformer models as depth increases · Depth-wise HP transfer and pretraining | loss | Hurt |
| Add bias and LayerNorm learning-rate adjustments to α=0.5 (ablation) | pre-LN transformers · Training stability across depths | stability | Helped |
| Train at larger TPP | parameterization learning-rate transfer tests · Depth-wise HP transfer | HP sensitivity | Helped |
| Use CompleteP (α=1) rather than μP | transformer models at increasing depth · Compute-efficient pretraining | FLOPs | Helped |
Surprising findings
- Compute-optimal N:L ratio is much smaller than the commonly used ~100 once depth is properly parameterized, with N:L around 10 remaining near-optimal.
- Standard alpha=0.5 parameterization fails training stability for pre-LN transformers unless LayerNorm and bias learning rates are corrected.
Limitations
- Theoretical analysis covers fixed token count; compute-optimal (fixed TPP) predictions are empirical extrapolation.
- Scaling-law fits use only three points for the 1.5B-scale comparison due to budget constraints.
- Experiments limited to models up to 1.5B non-embedding parameters.
Experimental setup
Datasets: SlimPajama
Benchmarks: SlimPajama validation cross-entropy HellaSwag ARC-Easy LAMBADA RACE PIQA BoolQ
Metrics: validation cross-entropy loss FLOP savings zero-shot accuracy downstream average accuracy
Baselines: Standard parameterization (SP) muP alpha=0.5 parameterization
Builds on: muP Tensor Programs Depthwise hyperparameter transfer (alpha=0.5 mean-field ResNet parameterization) Chinchilla scaling laws Pre-LN transformer
Closest papers by meaning
- Power Lines: Scaling laws for weight decay and batch size in LLM pre-training (NeurIPS 2025 · similarity 0.85)
- Tensor Programs VI: Feature Learning in Infinite Depth Neural Networks (ICLR 2024 · similarity 0.84)
- Learning Rate Transfer for Hybrid Transformer-SSM Architectures (NeurIPS 2026 · similarity 0.84)
- On the Provable Separation of Scales in Maximal Update Parameterization (ICML 2025 · similarity 0.84)
- Sparse maximal update parameterization: A holistic approach to sparse training dynamics (NeurIPS 2024 · similarity 0.84)
- The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networks (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.