Research map / Architectures and efficiency / RNNs and normalization
Scaling and Transferability of Annealing Strategies in Large Language Model Training
Siqi Wang, Zhengyu Chen, Teng Xiao, Zheqi Lv, Jinluan Yang, Xunliang Cai, Jingang Wang, Xiaomeng Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Adam-style momentum term instead of multiplicative CMMT (lambda=0.999) (ablation) | Forward-Momentum scaling law · Fit across batch sizes, Dense 1B | MAPE (%): 2.151 -> 0.674 | Improved |
| use Adam-style momentum term instead of multiplicative CMMT (lambda=0.99) (ablation) | Forward-Momentum scaling law · Fit across batch sizes, Dense 1B | MAPE (%): 0.681 -> 0.674 | No clear effect |
| use Adam-style momentum instead of CMMT (lambda=0.999) (ablation) | Forward-Momentum scaling law · Fit across model sizes, Dense LRmax 5e-5 batch 512 | MAPE (%): 0.943 -> 0.579 | Improved |
| use Adam-style momentum instead of CMMT (lambda=0.99) (ablation) | Forward-Momentum scaling law · Fit across model sizes, MoE LRmax 2e-4 | MAPE (%): 0.750 -> 0.957 | Hurt |
| use Adam-style momentum instead of CMMT (lambda=0.999) (ablation) | Forward-Momentum scaling law · Fit across model sizes, MoE LRmax 2e-3 | MAPE (%): 0.487 -> 0.574 | Hurt |
| use Adam-style momentum instead of CMMT (lambda=0.999) (ablation) | Forward-Momentum scaling law · Cosine prediction from WSD fit, Dense 100M | MAPE (%): 2.251 -> 0.232 | Improved |
| track loss by training steps instead of tokens when batch size exceeds optimal batch size (ablation) | loss curve modelling · Dense 500M and MoE 700M | loss curve similarity | Improved |
| increase total training steps from 30k to 300k | WSD scheduler · 1B MoE | optimal annealing ratio | Mixed |
| increase model size at fixed LRmax | WSD scheduler · Dense and MoE | optimal annealing ratio | No clear effect |
| train on one dataset and evaluate annealing ratio on validation sets | WSD scheduler · 1B MoE | optimal annealing ratio | No clear effect |
Closest papers by meaning
- Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Training (NeurIPS 2025 · similarity 0.91)
- Scaling Law with Learning Rate Annealing (NeurIPS 2025 · similarity 0.90)
- Model Merging in Pre-training of Large Language Models (NeurIPS 2025 · similarity 0.90)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.90)
- Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers (AAAI 2025 · similarity 0.88)
- Boosting Multi-Domain Fine-Tuning of Large Language Models through Evolving Interactions between Samples (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
