Research map / Architectures and efficiency / RNNs and normalization

Bridging Compute- and Data-Optimal Pretraining

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Neural scaling laws · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Compute-Data (CD) scaling laws with a token effectiveness function η that maps derived tokens (multi-epoch repetition, paraphrasing) to fresh-data equivalents, fit on Dolma-3 models from 14M to 600M parameters.

Problem

Classical Chinchilla-style scaling laws assume unbounded fresh data, but pretraining increasingly faces a fixed corpus where extra compute must go to derived tokens; how much can such compute reduce loss?

Approach

Introduces a token effectiveness function η with a saturating exp-sat form, where the effective derived data η·r saturates to a ceiling R*(D,N) (power law in tokens-per-parameter and N), unifying compute-optimal and data-optimal scaling into one law and partitioning training into compute-bound, data-bound and model-bound regimes.

Key results

• Fitted CD law with exp-sat R* achieves held-out RMSE 0.079 on log loss when fit only on 14M/30M models, vs in-sample 0.048.
• Paraphrasing beats repetition for small models (N up to about 50M) at 20x Chinchilla-like TTP; repetition wins for N above about 190M.
• Extrapolation suggests 4-epoch repetition is recommended only for around 3B-parameter models near 1x Chinchilla, and paraphrasing becomes ineffective for N >= 7B or budgets >= 4x Chinchilla.

Surprising findings

• Paraphrasing becomes ineffective at large model sizes and large data budgets, so the commonly recommended multi-epoch or paraphrase strategies do not transfer uniformly.
• Compute-optimal allocation is suboptimal across most of the practically relevant regime.

Limitations

• eta absorbs residual misspecification of the 1-epoch fit and can exceed 1, making fits sensitive to 1-epoch run quality.
• Only learning rate and weight decay were swept; batch size, sequence length and schedule were fixed.
• Only small-to-medium scale (up to 600M) models were trained; large-scale predictions are extrapolated, and only two strategies were evaluated.

Experimental setup

Datasets: Dolma-3 150B

Benchmarks: GSM8K TriviaQA HumanEval LAMBADA HellaSwag OpenBookQA RACE SQuAD

Metrics: validation loss (BPB) RMSE on log loss bits-per-byte accuracy leave-one-out RMSE

Baselines: Chinchilla scaling law constant-R* data-repetition law (Muennighoff et al.)

Builds on: Chinchilla scaling laws Muennighoff data-constrained scaling laws OLMo 3

Closest papers by meaning

• Resolving Discrepancies in Compute-Optimal Scaling of Language Models (NeurIPS 2024 · similarity 0.91)
• Data Engineering for Scaling Language Models to 128K Context (ICML 2024 · similarity 0.90)
• Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws (NeurIPS 2026 · similarity 0.90)
• Scaling Laws for Optimal Data Mixtures (NeurIPS 2025 · similarity 0.89)
• Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.89)
• Prescriptive Scaling Reveals the Evolution of Language Model Capabilities (ICML 2026 · similarity 0.89)