Research map / Large language models / Language, safety and interpretability
How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
Kairong Luo, Zhenbo Sun, Haodong Wen, Xinyu Shi, Jiarui Cui, Chenyi Dang, Kaifeng Lyu, Wenguang Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add ascending data curriculum under standard WSD decay (ablation) | WSD + Uniform · DCLM-Baseline 30B tokens, DCLM-score curriculum | average score: 50.56 -> 50.34 | No clear effect |
| add ascending data curriculum under cosine decay (ablation) | Cosine + Uniform · DCLM-Baseline 30B tokens, DCLM-score curriculum | average score: 49.13 -> 49.73 | Improved |
| replace LR decay with EMA averaging and add ascending curriculum (ablation) | WSD + Uniform · DCLM-Baseline 30B tokens, DCLM-score curriculum | average score: 50.56 -> 50.95 | Improved |
| use model averaging with uniform data order and constant LR (ablation) | WSD + Uniform · DCLM-Baseline 30B tokens, DCLM-score curriculum | average score: 50.56 -> 49.94 | Hurt |
| add ascending curriculum to EMA constant-LR training (ablation) | EMA + Uniform + Const · DCLM-Baseline 30B tokens, DCLM-score curriculum | average score: 49.94 -> 50.95 | Improved |
| use EMA or SMA instead of WMA with curriculum (ablation) | CMA · DCLM-Baseline 30B tokens, DCLM-score curriculum | Core score: 46.49 -> 47.02 (SMA), 46.95 (EMA) | Improved |
| use moderate LR decay (ending LR about 1/3 of peak) with curriculum (ablation) | WSD + Ascend · DCLM-Baseline 30B tokens, DCLM-score curriculum | validation loss / accuracy | Improved |
| make LR decay more aggressive (longer decay, lower end LR) with curriculum (ablation) | Ascend + WSD · DCLM-Baseline 30B tokens, DCLM-score curriculum | curriculum advantage | Hurt |
| use folding curriculum instead of simple ascending (ablation) | Ascending curriculum · DCLM-Baseline 30B tokens, DCLM-score curriculum | validation loss | Mixed |
| CMA with A-T ordering in mid-training vs WSD uniform (ablation) | CMA · Mid-training (29B WebOrganizer + 5B DCLM) | average score: 47.49 -> 48.69 | Improved |
| apply curriculum only to final high-quality phase (U,A) (ablation) | EMA + Const · Mid-training (29B WebOrganizer + 5B DCLM) | average score: 47.71 -> 47.45 | No clear effect |
| apply curriculum to both phases (A,A) instead of last phase only (ablation) | EMA + Const · Mid-training (29B WebOrganizer + 5B DCLM) | average score: 47.45 -> 48.69 | Improved |
Closest papers by meaning
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.91)
- Improving Pretraining Data Using Perplexity Correlations (ICLR 2025 · similarity 0.89)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.89)
- An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence (NeurIPS 2026 · similarity 0.89)
- Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum (NeurIPS 2024 · similarity 0.89)
- Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
