Research map / Architectures and efficiency / Quantization
u-$\mu$P: The Unit-Scaled Maximal Update Parametrization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes u-µP, which combines Maximal Update Parametrization with Unit Scaling to get unit-scale initialization, a smaller and more separable HP set, an independent sweep strategy, and out-of-the-box FP8 training of Llama-style LLMs.
Problem
Standard µP has practical problems for LLM training: LR transfer breaks in Llama-style setups, the HP set to sweep is chosen arbitrarily, HPs are strongly interdependent, base-shape implementation is awkward, and µP models struggle in low precision (e.g. FP16 underflow).
Approach
Combine µP with Unit Scaling so that weights, activations and gradients start at unit scale; this yields a reduced set of interpretable, weakly coupled multiplier HPs (defaults near optimal), a cheap independent search that sweeps mainly the LR, and a simple FP8 mixed-precision scheme via casting.
Key results
- Independent search with only an LR sweep (9 runs) reaches near-optimal loss for u-µP, while µP needs the full random search over 339 runs.
- Average transfer error is 0.03 for µP versus 0.005 for u-µP in the HP interdependence analysis.
- A 7B u-µP model trained in FP8 on 300B tokens is competitive with SP and BF16 u-µP on 0-shot benchmarks (e.g. FP8 MMLU 31.2 vs SP BF16 29.6).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use u-µP instead of SP, both in BF16 | 7B Llama · HellaSwag 0-shot | score: 52.4 -> 53.4 | Helped |
| Use u-µP instead of SP, both in BF16 | 7B Llama · OpenBookQA 0-shot | score: 27.8 -> 31.6 | Helped |
| Use u-µP instead of SP, both in BF16 | 7B Llama · PIQA 0-shot | score: 76.5 -> 77.1 | Helped |
| Use u-µP instead of SP, both in BF16 | 7B Llama · TriviaQA 0-shot | score: 22.2 -> 23.4 | Helped |
| Use u-µP instead of SP, both in BF16 | 7B Llama · WinoGrande 0-shot | score: 63.3 -> 63.7 | Helped |
| Use u-µP instead of SP, both in BF16 | 7B Llama · MMLU 0-shot | score: 29.6 -> 29.0 | Hurt |
| Use FP8 mixed precision instead of BF16 | 7B u-µP Llama · MMLU 0-shot | score: 29.0 -> 31.2 | Helped |
| Use FP8 mixed precision instead of BF16 | 7B u-µP Llama · HellaSwag 0-shot | score: 53.4 -> 53.4 | No clear effect |
| Use FP8 mixed precision instead of BF16 | 7B u-µP Llama · OpenBookQA 0-shot | score: 31.6 -> 29.6 | Hurt |
| Use FP8 mixed precision instead of BF16 | 7B u-µP Llama · PIQA 0-shot | score: 77.1 -> 77.6 | Helped |
| Use FP8 mixed precision instead of BF16 | 7B u-µP Llama · TriviaQA 0-shot | score: 23.4 -> 21.3 | Hurt |
| Use FP8 mixed precision instead of BF16 | 7B u-µP Llama · WinoGrande 0-shot | score: 63.7 -> 65.7 | Helped |
Surprising findings
- Sweeping only the learning rate with all other u-µP multipliers fixed at 1 is already near-optimal, suggesting unit scale is close to ideal for learning dynamics.
- Standard µP's embedding LR scaling rule transfers poorly across width, and the paper's fix lets a 2048-width u-µP model match a 4096-width µP model.
Limitations
- The embedding LR scaling rule and several HP choices are justified only empirically, without theoretical explanation.
- Neither µP nor Unit Scaling guarantees well-behaved quantities during training; critical tensors show scale growth that can push them out of FP8 range.
- Most experiments use a small Llama model on WikiText-103 with a single architecture family.
Experimental setup
Datasets: WikiText-103 SlimPajama WikiText-2
Benchmarks: MMLU HellaSwag OpenBookQA PIQA TriviaQA WinoGrande
Metrics: validation loss transfer error 0-shot accuracy RMS of tensors
Baselines: µP (random search) standard parametrization (SP) BF16 u-µP Pythia-style SP with Llama 3 LR scaling
Builds on: maximal update parametrization Unit Scaling Tensor Programs V depth-µP
Closest papers by meaning
- On the Provable Separation of Scales in Maximal Update Parameterization (ICML 2025 · similarity 0.89)
- Sparse maximal update parameterization: A holistic approach to sparse training dynamics (NeurIPS 2024 · similarity 0.87)
- Scaling Exponents Across Parameterizations and Optimizers (ICML 2024 · similarity 0.87)
- $\boldsymbol{\mu}\mathbf{P^2}$: Effective Sharpness Aware Minimization Requires Layerwise Perturbation Scaling (NeurIPS 2024 · similarity 0.86)
- Power Lines: Scaling laws for weight decay and batch size in LLM pre-training (NeurIPS 2025 · similarity 0.86)
- µnit Scaling: Simple and Scalable FP8 LLM Training (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.