AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining

Kairong Luo, Zhenbo Sun, Haodong Wen, Xinyu Shi, Jiarui Cui, Chenyi Dang, Kaifeng Lyu, Wenguang Chen

ICLR 2026 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 9 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add ascending data curriculum under standard WSD decay (ablation)WSD + Uniform · DCLM-Baseline 30B tokens, DCLM-score curriculumaverage score: 50.56 -> 50.34No clear effect
add ascending data curriculum under cosine decay (ablation)Cosine + Uniform · DCLM-Baseline 30B tokens, DCLM-score curriculumaverage score: 49.13 -> 49.73Improved
replace LR decay with EMA averaging and add ascending curriculum (ablation)WSD + Uniform · DCLM-Baseline 30B tokens, DCLM-score curriculumaverage score: 50.56 -> 50.95Improved
use model averaging with uniform data order and constant LR (ablation)WSD + Uniform · DCLM-Baseline 30B tokens, DCLM-score curriculumaverage score: 50.56 -> 49.94Hurt
add ascending curriculum to EMA constant-LR training (ablation)EMA + Uniform + Const · DCLM-Baseline 30B tokens, DCLM-score curriculumaverage score: 49.94 -> 50.95Improved
use EMA or SMA instead of WMA with curriculum (ablation)CMA · DCLM-Baseline 30B tokens, DCLM-score curriculumCore score: 46.49 -> 47.02 (SMA), 46.95 (EMA)Improved
use moderate LR decay (ending LR about 1/3 of peak) with curriculum (ablation)WSD + Ascend · DCLM-Baseline 30B tokens, DCLM-score curriculumvalidation loss / accuracyImproved
make LR decay more aggressive (longer decay, lower end LR) with curriculum (ablation)Ascend + WSD · DCLM-Baseline 30B tokens, DCLM-score curriculumcurriculum advantageHurt
use folding curriculum instead of simple ascending (ablation)Ascending curriculum · DCLM-Baseline 30B tokens, DCLM-score curriculumvalidation lossMixed
CMA with A-T ordering in mid-training vs WSD uniform (ablation)CMA · Mid-training (29B WebOrganizer + 5B DCLM)average score: 47.49 -> 48.69Improved
apply curriculum only to final high-quality phase (U,A) (ablation)EMA + Const · Mid-training (29B WebOrganizer + 5B DCLM)average score: 47.71 -> 47.45No clear effect
apply curriculum to both phases (A,A) instead of last phase only (ablation)EMA + Const · Mid-training (29B WebOrganizer + 5B DCLM)average score: 47.45 -> 48.69Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy