Research map / Architectures and efficiency / RNNs and normalization
AlphaZero Neural Scaling and Zipf's Law: a Tale of Board Games and Power Laws
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that AlphaZero game-state frequencies follow Zipf's law arising from game-tree structure, that agents minimize loss in descending frequency order, and that inverse scaling in Oware and Checkers coincides with late-game states dominating the frequency distribution.
Problem
Why neural scaling laws emerge in reinforcement learning (AlphaZero), and whether the Zipf-law quantization model used to explain LLM scaling applies to RL game-state distributions, including the cause of inverse scaling.
Approach
Connects AlphaZero scaling laws to Zipf's law over game states, arguing Zipf curves arise from game-tree branching structure; modulates policy temperature to show correlation between Zipf exponent and size-scaling exponent, and explains inverse scaling via late-game states dominating frequency ranks and selective forgetting of early-game states.
Key results
- Game-state visit frequencies follow Zipf's law with exponents roughly in the range 0.8-1.0 across Connect Four and Pentago agents; environment exponents lie in [0.75,1.05].
- Inverse scaling in Oware and Checkers: Elo stops increasing and reverses beyond a certain network size, coinciding with late-game states making up >80% of states at rank ~164 in Oware.
- Value loss of most Oware agents jumps by up to an order of magnitude at the transition point, while the largest models have better loss on late-game states but worse on early-game states.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase neural-network size | AlphaZero agents · Connect Four ground-truth value loss | value loss | Helped |
| Increase neural-network size | AlphaZero agents · Oware | Elo | Hurt |
| Increase neural-network size | AlphaZero agents · Oware early-game states | value loss | Hurt |
| Increase neural-network size | AlphaZero agents · Oware late-game states | value loss | Helped |
| Increase inference policy temperature within the examined range | Connect Four AlphaZero · Connect Four | size-scaling exponent | Mixed |
| Increase inference policy temperature below the policy-quality degradation threshold | Connect Four AlphaZero · Connect Four | probability of playing an optimal move: T \approx 0.5 | No clear effect |
| Use a non-uniform policy instead of a uniform policy | Random toy-model games · Toy-model game | state-frequency distribution: \alpha=1 | Helped |
| Make the policy strongly biased, close to deterministic | Random toy-model games · Toy-model game | state-frequency distribution | Hurt |
Surprising findings
- Agents minimize value loss in descending order of state frequency even though absolute loss increases with state rank, so better performance is achieved on harder tasks.
- Large Oware models fit late-game states better while forgetting early-game states, producing inverse scaling.
Limitations
- Temperature modulation is an imperfect causal probe, since the influence of low-rank plateaus on performance is unclear, especially at very low temperature.
- Game states are used as proxies for independent task quanta, which the paper acknowledges may not be correct.
- Inverse scaling mechanism is a hypothesis; the paper does not fully connect state distribution to Elo scaling laws.
Experimental setup
Datasets: self-play game states from Connect Four Pentago Oware Checkers
Benchmarks: Connect Four Pentago Oware Checkers
Metrics: Elo score Zipf exponent value loss probability of optimal move
Baselines: Michaud et al. (2023) quantization model alpha-beta pruning solver
Builds on: AlphaZero quantization model of LLM scaling OpenSpiel Neumann & Gros (2022)
Closest papers by meaning
- How Do Large Language Monkeys Get Their Power (Laws)? (ICML 2025 · similarity 0.89)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.88)
- Observational Scaling Laws and the Predictability of Langauge Model Performance (NeurIPS 2024 · similarity 0.88)
- Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf’s Law (NeurIPS 2025 · similarity 0.87)
- Scaling Law with Learning Rate Annealing (NeurIPS 2025 · similarity 0.87)
- Universal One-third Time Scaling in Learning Peaked Distributions (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.