Research map / Architectures and efficiency / RNNs and normalization

Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Neural scaling laws · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that fixed tokens-per-parameter (collinear) scaling-law designs make Gauss-Newton fits ill-conditioned (condition number ~ inverse square of exponent gap), and that non-collinear (N,D) grids beat collinear ones on held-out splits (97.3% win rate).

Problem

Scaling-law fits from runs at a fixed tokens-per-parameter ratio D=kN are collinear, which, given near-equal N and D exponents, makes the parameters practically unidentifiable and extrapolation fragile off the training ray.

Approach

Analyzes the Jacobian geometry of single-ray (collinear) scaling-law fits, proves condition number grows as the inverse square of the exponent gap for four formalisms, derives a TPP-diversity threshold for well-conditioned estimation, and validates that two-dimensional (non-collinear) designs generalize better.

Key results

• Non-collinear designs win 97.3% (1,460/1,500) of seed-paired holdout comparisons against collinear designs across four laws and five corpora.
• Unified holdout R^2 rises from 0.837 (collinear) to 0.932 (non-collinear), with RMSE dropping from 0.237 to 0.156.
• Kaplan collinear fit on RedPajama reaches holdout R^2 of only 0.51 versus 0.91 for non-collinear.

Surprising findings

• Adding more TPP ratios (12 collinear rays) does not rescue collinear designs; the condition number depends on the exponent gap rather than the number of rays.
• Budget-matched win rates drop to near 50% on the Cosmopedia and peS2o corpora.

Limitations

• Experiments use small models (5.04M-76.5M parameters), a single LLaMA-style architecture, and English-only text.
• Corollary CI inflation assumes i.i.d. Gaussian residuals; one run per (N,D) pair.
• Budget-matched advantage weakens on specialized corpora (Cosmopedia, peS2o).

Experimental setup

Datasets: C4 RedPajama Wikipedia Cosmopedia peS2o

Metrics: holdout RMSE holdout R^2 win rate confidence interval width condition number

Baselines: Chinchilla scaling law (collinear fit) Kaplan scaling law (collinear fit) Droppo-Elibol scaling law (collinear fit) Repeated-data scaling law (collinear fit)

Builds on: Chinchilla scaling law Kaplan scaling laws Droppo-Elibol scaling law repeated-data scaling law

Closest papers by meaning

• Scaling Law with Learning Rate Annealing (NeurIPS 2025 · similarity 0.91)
• A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.90)
• Universal One-third Time Scaling in Learning Peaked Distributions (ICML 2026 · similarity 0.90)
• Resolving Discrepancies in Compute-Optimal Scaling of Language Models (NeurIPS 2024 · similarity 0.90)
• Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.90)
• On the Invariance and Generality of Neural Scaling Laws (NeurIPS 2026 · similarity 0.89)