Research map / Architectures and efficiency / RNNs and normalization
Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration
Bruno Kacper Mlodozeniec, Pierre Ablin, Louis Béthune, Dan Busbridge, Michal Klein, Jason Ramapuram, marco cuturi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use alpha=1/2 residual scaling in CompleteP instead of only alpha=1 | CompleteP · GPT-style LM on RedPajama | HP transfer across depth | No clear effect |
| remove QK norms (ablation) | CompleteP · GPT-style LM on RedPajama | HP transfer across depth | No clear effect |
| rescale weight decay by 1/sqrt(kappa) with batch size (ablation) | AdamW · GPT-style LM on RedPajama | weight decay transfer across batch size | Improved |
| scale LR by 1/sqrt(kappa) with token horizon | AdamW · GPT-style LM on RedPajama | LR transfer across token horizon | Improved |
| scale token horizon by increasing batch size only (SDE iso-horizon rule) | AdamW · GPT-style LM on RedPajama | LR transfer across token horizon | Improved |
| add per-depth multipliers to per-module learning rates (ablation) | per-module LR search · GPT-style LM on RedPajama | validation loss | Improved |
| uncouple per-layer learning rates beyond Kronecker factorisation (ablation) | per-module LR search · GPT-style LM on RedPajama | validation loss | No clear effect |
| use optimal per-module HPs instead of optimal global HPs when transferring to 1.3B / 26B tokens | CompleteP · GPT-style LM on RedPajama | speed-up to reach equal loss: 27% speed-up at 420x compute | Improved |
| use per-module search from 50M instead of global random search | 50M transformer · GPT-style LM on RedPajama | validation NLL: global best 3.34 nats is substantially higher | Improved |
Closest papers by meaning
- On the Provable Separation of Scales in Maximal Update Parameterization (ICML 2025 · similarity 0.91)
- Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit (ICLR 2024 · similarity 0.90)
- Scaling Exponents Across Parameterizations and Optimizers (ICML 2024 · similarity 0.89)
- Scaling Optimal LR Across Token Horizons (ICLR 2025 · similarity 0.88)
- Principled Architecture-aware Scaling of Hyperparameters (ICLR 2024 · similarity 0.87)
- Power Lines: Scaling laws for weight decay and batch size in LLM pre-training (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
