Research map / Architectures and efficiency / RNNs and normalization
Pretraining Scaling Laws for Generative Evaluations of Language Models
Rylan Schaeffer, Noam Itzhak Levi, Brando Miranda, Sanmi Koyejo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase attempts per problem k | compute scaling law · GSM8K | compute exponent alpha: 1.21e-1 -> 3.75e-1 | Improved |
| increase attempts per problem k to about 1e2 | compute scaling law · GSM8K | irreducible error E0: approx 2 -> 0 | Improved |
| increase k to about 3e2 | parameters+tokens scaling law · GSM8K | irreducible error: 3.87 -> 0 | Improved |
| decompose compute into parameters and tokens | scaling law fit · GSM8K | in-range fit tightness | Improved |
| decompose compute into parameters and tokens | scaling law prediction · GSM8K | prediction of Pythia-12B | No clear effect |
| use gold reference likelihood as covariate instead of compute | scaling law · GSM8K | parameter stability: 2 -> 5 orders of magnitude | Improved |
| use gold reference likelihood law at large k | scaling law prediction · GSM8K | relative error | Hurt |
| move from GSM8K to harder MATH | compute scaling law · MATH vs GSM8K | irreducible error at k=1e4: 0 -> 0.45 | Hurt |
| move from GSM8K to harder MATH | compute scaling law · MATH vs GSM8K | compute exponent at k=1e4: 0.38 -> 0.58 | Improved |
| use k=1e4 instead of k=1 or 1e2 | compute scaling law · GSM8K | orders of magnitude needed for extrapolation: 2 -> 1.5 | Improved |
Closest papers by meaning
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.93)
- Resolving Discrepancies in Compute-Optimal Scaling of Language Models (NeurIPS 2024 · similarity 0.92)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.92)
- Scaling Laws for Precision (ICLR 2025 · similarity 0.91)
- The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.91)
- A Simple Model of Inference Scaling Laws (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
