Research map / Architectures and efficiency / RNNs and normalization
Language models scale reliably with over-training and on downstream tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a 104-model testbed of decoder-only LMs (0.011B-6.9B params, token multipliers 5-640) on C4, RedPajama and RefinedWeb, and fits scaling laws that extrapolate validation loss over-training and map loss to average downstream top-1 error.
Problem
Scaling laws are usually fit in the compute-optimal (Chinchilla) regime and predict next-token loss rather than downstream task accuracy, while practitioners over-train models and compare them on benchmarks.
Approach
Reparameterize the Chinchilla loss law in terms of compute C and token multiplier M=D/N so that the exponent stays constant across over-training levels, and model average downstream top-1 error as an exponential decay in validation loss, chaining the two to predict downstream error.
Key results
- Scaling law fit on models up to 0.411B (plus 1.4B at M=20 for downstream) predicts the C4 loss of a 1.4B, 900B-token (M=640) run within 0.7% relative error using ~300x less compute.
- The downstream error law predicts the average top-1 error of a 6.9B, 138B-token RedPajama model within 0.05% relative error using ~20x less compute.
- Removing the 1.4B point from the downstream fit raises relative error on the 6.9B RedPajama prediction from 0.05% to 10.64%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fit the over-training scaling law using smaller-scale runs | C4 eval loss prediction for RedPajama-trained models · RedPajama 1.4B, 900B tokens | relative prediction error: 0.7% | Helped |
| chain the loss scaling law with the loss-to-error law | RedPajama 6.9B, 138B tokens · 17-task split | average top-1 error relative prediction error: 0.05% | Helped |
| include the 1.4B model in the downstream-error scaling fit (ablation) | Chained loss-to-error scaling law · RedPajama 6.9B, 138B tokens; 17-task split | average top-1 error relative prediction error: 10.64% -> 0.05% | Helped |
| include the 1.4B model in the downstream-error scaling fit (ablation) | Chained loss-to-error scaling law · RefinedWeb 6.9B, 138B tokens; 17-task split | average top-1 error relative prediction error: 15.79% -> 2.94% | Helped |
| include the 1.4B model in the downstream-error scaling fit (ablation) | Chained loss-to-error scaling law · C4 6.9B, 138B tokens; 17-task split | average top-1 error relative prediction error: 0.42% -> 0.14% | Mixed |
| use token multiplier M=5 (ablation) | Over-training scaling law · C4 eval loss | scaling reliability | Hurt |
| evaluate English-filtered C4-trained models on programming-language data (ablation) | Over-training scaling law · Programming-language next-token prediction | relative prediction error | Hurt |
| evaluate English-filtered C4-trained models on Penn Tree Bank (ablation) | Over-training scaling law · Penn Tree Bank next-token prediction | relative prediction error | Hurt |
| evaluate English-filtered C4-trained models on German text (ablation) | Over-training scaling law · German C4 next-token prediction | maximum relative prediction error: 7.6% | No clear effect |
| average across all 46 downstream evaluations instead of selecting the 17 high-signal tasks (ablation) | Chained loss-to-error scaling law · 6.9B, 138B-token predictions | average top-1 error relative prediction error: less than 3% | No clear effect |
| increase the downstream-task exclusion threshold to t≥40 (ablation) | Chained loss-to-error scaling law · Downstream evaluation-set sensitivity | relative prediction error | Hurt |
| use C4, RedPajama, or RefinedWeb as the validation-loss distribution (ablation) | Loss-to-downstream-error relationship · Average downstream top-1 error | exponential-decay relationship | No clear effect |
Surprising findings
- Scaling breaks down for small token multipliers (M=5) and out-of-distribution evaluation on programming languages and Penn Tree Bank, while German C4 loss remains predictable.
- Individual downstream task predictions are much noisier than predictions of the average top-1 error.
Limitations
- Hyperparameters were tuned only at M=20 and require extensive sweeps; no hyperparameter-free scaling law.
- Scaling is not validated at very large scale and MMLU-level performance is near chance for these models.
- Post-training effects are not considered; only base pretrained models are studied.
Experimental setup
Datasets: C4 RedPajama RefinedWeb OpenLM eval Paloma LLM-Foundry 17-task downstream split
Benchmarks: C4 eval loss ARC-Easy LAMBADA OpenBookQA HellaSwag 17-task LLM-Foundry average 46-task LLM-Foundry average Paloma
Metrics: validation loss average top-1 error relative prediction error
Baselines: Chinchilla scaling law parameterization Chinchilla 70B LLaMA-2 7B
Builds on: Chinchilla scaling laws OpenLM LLM-Foundry evaluation suite Hoffmann et al. compute-optimal scaling Sardana and Frankle inference-aware scaling
Closest papers by meaning
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.95)
- Resolving Discrepancies in Compute-Optimal Scaling of Language Models (NeurIPS 2024 · similarity 0.93)
- Scaling Inference-Efficient Language Models (ICML 2025 · similarity 0.92)
- Neural Neural Scaling Laws (NeurIPS 2026 · similarity 0.92)
- Scaling Laws for Precision (ICLR 2025 · similarity 0.92)
- Observational Scaling Laws and the Predictability of Langauge Model Performance (NeurIPS 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.