Research map / Large language models / Language, safety and interpretability
Towards a theory of how the structure of language is acquired by deep neural networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analytically derives token-token correlation decay in a Random Hierarchy Model (PCFG) and shows a finite training set yields an effective context window, predicting stepwise loss and scaling laws that are checked on synthetic data, tiny-Shakespeare and WikiText-103.
Problem
How much data is needed for next-token prediction to learn the hierarchical structure of a language, and why do test loss scaling laws with training set size and context length arise?
Approach
Shows that token correlations in a Random Hierarchy Model decay stepwise (power law in distance) and that finite training set size P limits resolvable correlations to an effective context window t*(P) ~ P^{1/z}; deeper hidden variables are learned as P grows, producing loss steps and a context-dependent scaling law L(P,t)=t^{-alpha z} f(P/t^z).
Key results
- Correlations of the masked token decay stepwise as a power law with exponent beta = log m / log s; for WikiText-103 the rescaling collapse uses z = 2*beta ≈ 3.1.
- On tiny-Shakespeare the fitted exponent is beta ≈ 1.4, giving z ≈ 2.8 and a context-dependent scaling collapse of test loss.
- Deep CNNs and transformers trained on RHM data show loss steps at sample complexities P_l = v m^{2l-1}, matching the predicted sample complexities.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase the context window | Transformer trained on tiny-Shakespeare · tiny-Shakespeare | test loss | Helped |
| increase the context window | Transformer trained on WikiText-103 · WikiText-103 | test loss | Helped |
| restrict the context window to size t (ablation) | Transformer trained on RHM data · RHM | test loss | Hurt |
| remove residual connections, layer normalization, and MLPs between attention layers (ablation) | Multi-layer self-attention network on RHM · RHM | performance | No clear effect |
| increase the number of attention heads (ablation) | Multi-layer self-attention network on RHM · RHM | performance | No clear effect |
| increase the embedding dimension up to 4 n_h×v (ablation) | Multi-layer self-attention network on RHM · RHM | performance | No clear effect |
| increase the number of CNN channels (ablation) | Deep CNN on RHM · RHM | performance | No clear effect |
| increase the number of layers or attention heads (ablation) | Encoder-only Transformer on real text · tiny-Shakespeare and WikiText-103 | performance | No clear effect |
Limitations
- Relies on context-free grammar with uniform production probabilities and unambiguity, not satisfied by natural language.
- No proof connecting the reconstruction strategy to sample complexity of gradient-descent-trained networks.
- Empirical tests limited to small transformers on character-level Shakespeare and WikiText-103, not state-of-the-art LLMs.
Experimental setup
Datasets: Random Hierarchy Model synthetic data tiny-Shakespeare WikiText-103
Benchmarks: tiny-Shakespeare WikiText-103
Metrics: test cross-entropy loss token-token correlation function relative sensitivity of representations
Baselines: n-gram language model
Builds on: Random Hierarchy Model BERT maximal update parametrization
Closest papers by meaning
- Language Models over Canonical Byte-Pair Encodings (ICML 2025 · similarity 0.86)
- Correlation Dimension of Autoregressive Large Language Models (NeurIPS 2025 · similarity 0.86)
- How Compositional Generalization and Creativity Improve as Diffusion Models are Trained (ICML 2025 · similarity 0.86)
- Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers (NeurIPS 2025 · similarity 0.86)
- Short-Context Dominance: How Much Local Context Natural Language Actually Needs? (NeurIPS 2026 · similarity 0.85)
- Large language models suffer from a curse of ambiguity (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.