Research map / Architectures and efficiency / RNNs and normalization
Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves statistical estimation and approximation bounds for transformers regressing Hölder functions on low-dimensional manifolds, deriving data and model scaling exponents that depend on intrinsic dimension, and checks them on small GPT-style LLMs and public scaling suites.
Problem
Why transformer neural scaling laws exist: a rigorous explanation of how generalization error scales with model size and training data size for transformers, given that real data lies on low-dimensional manifolds.
Approach
Derives transformer approximation and generalization bounds for Hölder functions on a d-dimensional manifold, with a shallow (O(log d) depth) construction using an Interaction Lemma, and converts the bounds into data and model scaling exponents 2β/(2β+d) and 2β/d expressed via intrinsic dimension.
Key results
- Empirical data-scaling exponents on the three datasets lie between 0.1 and 0.15, with predicted alpha_D matching observed values within ±0.02 absolute error.
- Model-scaling predictions alpha_N = 2/d under-estimate empirical exponents on GPT-2 and Pythia suites; the paper reports this as less accurate than the data-scaling predictions.
- Estimated intrinsic dimension of The Pile rises from 15.56 (410M model) to 20.02 (12.8B model), and is stable across embedding dimension, depth and context length.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| predict the data scaling exponent from the model scaling exponent | ID-free scaling-law prediction · GPT-2 | data scaling exponent: 0.070 vs 0.095 | Mixed |
| predict the model scaling exponent from the data scaling exponent | ID-free scaling-law prediction · GPT-2 | model scaling exponent: 0.106 vs 0.076 | Mixed |
| predict the data scaling exponent from the model scaling exponent | ID-free scaling-law prediction · Chinchilla | data scaling exponent: 0.25 vs 0.28 | Mixed |
| predict the model scaling exponent from the data scaling exponent | ID-free scaling-law prediction · Chinchilla | model scaling exponent: 0.33 vs 0.34 | Mixed |
| increase model parameter count from 410 million to 12.8 billion (ablation) | embedding model used to estimate intrinsic dimension · The Pile | estimated intrinsic dimension: 15.56 -> 20.02 | Hurt |
| increase embedding dimension from 768 to 1536 (ablation) | embedding model used to estimate intrinsic dimension · OpenWebText | estimated intrinsic dimension: 15.56 -> 18.68 | Hurt |
| increase sequence length from very short sequences to around 1024 tokens (ablation) | embedding model used to estimate intrinsic dimension · Text sequences | estimated intrinsic dimension: 15.86 -> 12.9 | Helped |
| predict the data scaling exponent using estimated intrinsic dimension | theory-based scaling prediction · OpenWebText, The Stack-SQL, and Tiny Stories | data scaling exponent: ± 0.02 | Mixed |
| predict the model scaling exponent using estimated intrinsic dimension | theory-based scaling prediction · GPT2 and Pythia scaling suites | model scaling exponent | Hurt |
Surprising findings
- The predicted model-scaling exponent under-estimates the empirically observed exponent, which the authors attribute to possible under-training of the largest models and intrinsic data entropy.
- The estimated intrinsic dimension of text decreases with sequence length (about 15.86 for very short sequences to 12.9 near 1024 tokens).
Limitations
- Empirical experiments assume Lipschitz target regularity (beta = 1), which the authors note may be inaccurate for language modeling.
- The computational scaling exponent alpha_C is not addressed.
- The cross-entropy loss bound is left as future work; classification results only partially justify the squared-error connection.
Experimental setup
Datasets: OpenWebText The Stack (SQL) TinyStories The Pile
Benchmarks: OpenWebText The Stack (SQL) TinyStories The Pile
Metrics: test loss scaling exponent intrinsic dimension (MLE estimator)
Baselines: GPT-2 scaling suite Pythia scaling suite Chinchilla scaling results
Builds on: Sharma and Kaplan intrinsic dimension scaling laws Chinchilla scaling laws Schmidt-Hieber ReLU network estimation theory
Closest papers by meaning
- On the Invariance and Generality of Neural Scaling Laws (NeurIPS 2026 · similarity 0.89)
- Scaling Laws for Sparsely-Connected Foundation Models (ICLR 2024 · similarity 0.89)
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.89)
- Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra (ICLR 2025 · similarity 0.89)
- Understanding Transformers via N-Gram Statistics (NeurIPS 2024 · similarity 0.88)
- Scaling Laws in Linear Regression: Compute, Parameters, and Data (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.