Research map / Architectures and efficiency / RNNs and normalization
Learning curves theory for hierarchically compositional data with power-law distributed features
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies learning curves for hierarchical PCFG-style Random Hierarchy Model data whose production rules are Zipf-distributed, showing power-law classification curves with exponent a/(1+a) and unchanged asymptotic scaling for next-token prediction.
Problem
Neural scaling laws are explained either by power-law feature distributions or by hierarchical compositional structure; the paper asks how the two combine to set learning-curve exponents for classification and next-token prediction.
Approach
Extends the Random Hierarchy Model by making production rules at one layer Zipf-distributed, then derives via a correlation-based learnability assumption that classification error decays as P^{-a/(1+a)} while next-token prediction loss exponent depends only on the hierarchy (log(m/v^{s-1})/(2 log m)).</1>
Key results
- Classification with Zipf exponent a shows asymptotic error decay P^{-a/(1+a)} after a pre-asymptotic phase set by v m^{L-1}; empirical CNN curves match theory predictions.
- Next-token prediction with Zipf production rules keeps the uniform-RHM scaling exponent log(m/v^{s-1})/(2 log m); transformer learning curves agree with this asymptotic decay.
- Zipf distribution of production rules changes only the local details and the asymptotic test loss L_infinity(a) of next-token prediction, not the decay exponent.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Zipf-distributed production rules | CNN on RHM root classification · RHM root classification | asymptotic test-error decay exponent: P^{-a/(1+a)} | Helped |
| Move Zipf-distributed rules from the last RHM layer to the first (ablation) | CNN on RHM root classification · RHM root classification | test error | No clear effect |
| Vary the Zipf exponent of production-rule probabilities (ablation) | CNN on RHM root classification · RHM root classification | asymptotic test-error decay exponent: P^{-a/(1+a)} | Mixed |
| Vary the number of production rules per symbol (ablation) | CNN on RHM root classification · RHM root classification | test error: v m^{L-1} | No clear effect |
| Use Zipf-distributed rather than uniform production rules | Transformer on RHM next-token prediction · RHM next-token prediction | asymptotic cross-entropy decay exponent: \log{(m/v^{s-1})}/(2\log{m}) | No clear effect |
| Use Zipf-distributed rather than uniform production rules | Transformer on RHM next-token prediction · RHM next-token prediction | residual cross-entropy: \mathcal{L}_\infty(a) | Mixed |
| Vary the Zipf exponent of production-rule probabilities (ablation) | Transformer on RHM next-token prediction · RHM next-token prediction | approach of cross-entropy to its residual value | No clear effect |
| Concentrate all production-rule probability mass on one rule instead of using a uniform distribution (ablation) | Transformer on RHM next-token prediction · RHM next-token prediction | asymptotic learning-curve decay: \log{(m/v^{s-1})}/(2\log{m}) | No clear effect |
| Remove residual connections, layer normalisation, and MLP layers (ablation) | Multi-layer self-attention transformer on RHM · RHM next-token prediction | performance | No clear effect |
| Vary the number of attention heads from 16 up to 64 (ablation) | Multi-layer self-attention transformer on RHM · RHM next-token prediction | performance: n_h\,{=}\,16 | No clear effect |
| Scale embedding dimension up to 4 n_h times the vocabulary size (ablation) | Multi-layer self-attention transformer on RHM · RHM next-token prediction | performance: 4 n_h\times v | No clear effect |
Surprising findings
- For next-token prediction the Zipf exponent a does not change the large-scale decay exponent, only the residual loss plateau.
Limitations
- Tree-like hierarchical model is simpler than natural data; topology of latent tree may vary.
- Context-dependent effects that cannot be described by a tree model are not captured.
- Zipf production rules are placed in a single layer; the all-layers case is only discussed informally.
Experimental setup
Datasets: synthetic Random Hierarchy Model data
Benchmarks: synthetic RHM classification synthetic RHM next-token prediction
Metrics: test error test cross-entropy loss learning-curve exponent
Baselines: uniform RHM Hutter's Zipf memorisation model
Builds on: Random Hierarchy Model Hutter's theory of learning sigmoidal learning curves of hierarchical data (Cagnetta et al. 2024)
Closest papers by meaning
- How Compositional Generalization and Creativity Improve as Diffusion Models are Trained (ICML 2025 · similarity 0.88)
- On the Invariance and Generality of Neural Scaling Laws (NeurIPS 2026 · similarity 0.88)
- Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks (NeurIPS 2026 · similarity 0.87)
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.87)
- How Do Large Language Monkeys Get Their Power (Laws)? (ICML 2025 · similarity 0.86)
- Zero-Shot Performance Prediction for Probabilistic Scaling Laws (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.