Research map / Large language models / Language, safety and interpretability
Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMs
Shane Bergsma, Nolan Simran Dey, Joel Hestness
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| place high-quality (code) data at the TREC minimum instead of other 10% segments (ablation) | 610M GPT2-style LLM · code-blend validation loss, step/10x/D2Z LR schedules | validation loss | Improved |
| place HQ data at TREC minimum instead of at the very end of training (ablation) | 610M LLM, step-decay and decay-to-zero schedules · code-blend validation loss | validation loss | Improved |
| placement curriculum versus uniform training on aggregate blend (ablation) | 610M LLM · aggregate blend / CB validation | validation loss | Improved |
| place math HQ data halfway through CPT (TREC minimum) instead of at the end (ablation) | 3.9B LLM CPT · OpenWebMath validation | validation loss | Improved |
| use high CPT peak learning rate 0.015 (ablation) | 3.9B LLM CPT · OpenWebMath validation | validation loss | Hurt |
| insert HQ data at predicted TREC minimum (~97%) during pre-training instead of CPT (ablation) | 3.9B LLM · OpenWebMath validation | validation loss | No clear effect |
| increase number of MoE experts from 1 to 32 (ablation) | 111M MoE · TREC drop | TREC depth | Improved |
| anneal LR over final ~3 optimizer steps on GSM8k/MATH train set | Llama 3 405B · GSM8k validation | accuracy | No clear effect |
| match AdamW timescale tau across LR, weight decay, batch size (ablation) | 610M LLM · TREC shape | TREC | Improved |
Closest papers by meaning
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.86)
- Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning (ICML 2024 · similarity 0.86)
- The Best Instruction-Tuning Data are Those That Fit (NeurIPS 2025 · similarity 0.86)
- Better Instruction-Following Through Minimum Bayes Risk (ICLR 2025 · similarity 0.86)
- NICE Data Selection for Instruction Tuning in LLMs with Non-differentiable Evaluation Metric (ICML 2025 · similarity 0.86)
- Improving Data Efficiency via Curating LLM-Driven Rating Systems (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
