Research map / Large language models / Language, safety and interpretability
Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Training
Jakub Krajewski, Amitis Shidani, Dan Busbridge, Sam Wiseman, Jason Ramapuram
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| predict accuracy directly from FLOPs with a power law instead of a two-stage linear NLL proxy | accuracy prediction · mean over benchmarks | MRE (%): 6.667 -> 1.963 | Improved |
| predict directly with a power law instead of two-stage logistic | accuracy prediction · mean over benchmarks | MRE (%): 6.351 -> 1.963 | Improved |
| use power law instead of BNSL | accuracy prediction · mean over benchmarks | MRE (%): 2.713 -> 1.963 | Improved |
| use power law instead of BNSL | accuracy prediction · average over 12 benchmarks | MRE (%): 4.96 -> 4.72 | Improved |
| use BNSL instead of power law | accuracy prediction · WebQS | MRE (%): 7.28 -> 1.46 | Improved |
| use BNSL instead of power law | accuracy prediction · LBPP | MRE (%): 13.43 -> 6.70 | Improved |
| use BNSL instead of power law | accuracy prediction · GSM8K | MRE (%): 9.67 -> 13.90 | Hurt |
| choose a different proxy metric in two-stage (log-prob, loss, Brier) (ablation) | two-stage scaling law · benchmarks | R2 | No clear effect |
| two-stage models fit training data better yet extrapolate worse | two-stage logistic · training fit | R2: 0.974 vs 0.986 | Mixed |
| fit only on runs with accuracy at least 5 points above random (ablation) | power law fit · benchmarks | fit quality | Improved |
| change pretraining data from the main mixture to C4 | power law scaling · code and math tasks | accuracy | Hurt |
| change pretraining data to C4 | power law scaling · general QA tasks | fit quality | No clear effect |
Closest papers by meaning
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.94)
- LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws (ICML 2025 · similarity 0.92)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.92)
- Fast and Accurate Probing of In-Training LLMs' Downstream Performances (NeurIPS 2026 · similarity 0.91)
- When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.90)
- Sloth: scaling laws for LLM skills to predict multi-benchmark performance across families (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
