Research map / Large language models / Language, safety and interpretability
Compute-Optimal LLMs Provably Generalize Better with Scale
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a fully empirical Freedman-type martingale concentration bound on the token-wise generalization gap of compute-optimal LLMs (Chinchilla scaling), showing the bound tightens with model scale on Pythia checkpoints.
Problem
Explaining why larger language models generalize better, i.e., why the token-wise generalization gap shrinks along the compute-optimal (Chinchilla) frontier, with non-vacuous, empirically evaluable bounds.
Approach
A new fully empirical Freedman-type martingale concentration inequality whose variance term is computed from the observed loss differences (replacing the population variance), combined with a prior-weighted union bound over quantized models, prediction smoothing, and a prequential-coding information bound that shows model information grows sublinearly with data.
Key results
- Loss variation Sigma decreases with model size approximately as 0.27 + 8337 N^-0.54 on Chinchilla-optimal Pythia checkpoints.
- The generalization bound improves with model scale in line with empirical loss, with a term-wise breakdown showing C log V and smoothing terms dominate and do not decrease with scale.
- A typical-scenario bound gives R_sq - R_h of about 1.8 nats per token (V=50000, b=3, G^2=1/20), so bounds are non-vacuous relative to log V of about 11 nats.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale up model size along the compute-optimal frontier | Pythia models · Pile | loss variation Σ: 0.27 + 8337 N^{-0.54} | Helped |
| Scale up model size along the compute-optimal frontier | Pythia models · Pile | quantization gap | Helped |
| Use the empirical Freedman inequality instead of the Azuma-based bound | Tokenwise generalization bound · Compute-optimal language models | complexity scaling: O(√C) to O(C) | Helped |
| Use prequential coding complexity instead of quantization-based complexity | Generalization bound · Pythia model scale | bound tightness | Hurt |
| Scale up model size using prequential coding complexity | Generalization bound · Compute-optimal scaling | generalization gap bound: O(D^{-β/2}) | Helped |
Surprising findings
- The dominant bound terms (C log V and smoothing cost) do not decrease with scale, so the improvement comes mainly from loss variance and quantization gap.
- Prequential-coding bounds are worse than quantization-based bounds at Pythia scale, though they improve with scale.
Limitations
- The smoothing term sqrt(2C) is pessimistic and the bound is loose in absolute terms (about 1.8 nats per token gap).
- Scaling-law extrapolations of the prequential bound rely on fitted power laws and assumptions about the Chinchilla regime.
- Only token-wise (not full-sequence) generalization is bounded; many pieces (Hessian PSD, quantization error) are argued theoretically rather than verified.
Experimental setup
Datasets: The Pile
Benchmarks: The Pile (Pythia checkpoints, token-wise risk)
Metrics: generalization bound value (nats per token) empirical loss loss variation Sigma quantization gap
Baselines: Azuma-based token-wise bound (Lotfi et al.) GPTQ quantization
Builds on: Azuma's inequality Freedman's inequality prediction smoothing GPTQ QuIP Chinchilla scaling laws
Closest papers by meaning
- Unlocking Tokens as Data Points for Generalization Bounds on Larger Language Models (NeurIPS 2024 · similarity 0.92)
- Non-Vacuous Generalization Bounds for Large Language Models (ICML 2024 · similarity 0.91)
- LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws (ICML 2025 · similarity 0.90)
- Resolving Discrepancies in Compute-Optimal Scaling of Language Models (NeurIPS 2024 · similarity 0.90)
- Scaling Laws for Precision (ICLR 2025 · similarity 0.89)
- What Makes Quantization for Large Language Model Hard? An Empirical Study from the Lens of Perturbation (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.