Research map / Large language models / Language, safety and interpretability
The Power of Power Law: Asymmetry Enables Compositional Reasoning
Zixuan Wang, Xingyu Dang, Jason D. Lee, Kaifeng Lyu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| sample skills from a power law (alpha=1.0) instead of uniform in training | 0.6B Qwen3-style model trained from scratch · multi-step arithmetic (k=4, no CoT), uniform test | accuracy / learning speed | Improved |
| sample group elements from a power law (alpha=1.5) instead of uniform | transformer · S5 state tracking, no CoT or curriculum | accuracy | Improved |
| use power-law instead of uniform skill sampling | small GPT-like model · multi-hop QA (|E|=50, |R|=20, k=3) | test accuracy vs training tokens | Improved |
| use power-law instead of uniform number sampling | language model on synthetic grade-school math (iGSM, ops 2-8, mod 211) · synthetic GSM | test accuracy vs training tokens | Improved |
| add power-law sampling on top of an explicit hop curriculum (k=1,2,3,4) (ablation) | S5 state tracking with curriculum · S5 state tracking | loss plateaus / training speed | Improved |
| vary power-law exponent alpha (ablation) | S5 state tracking · S5 state tracking | training speed | Mixed |
| use a more fine-grained (more bins) power-law distribution (ablation) | S5 state tracking · S5 state tracking, m in {5,10,20,40,60} bins | learning speed | Improved |
| order skills in power law lexicographically instead of random or reversed order (ablation) | S5 state tracking · S5 state tracking | learning speed | Improved |
| power-law distribution vs uniform in the final convergence stage | S5 state tracking · S5 tail permutations (Bins 4, 5) | test loss on tail skills | Hurt |
Closest papers by meaning
- Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillation (NeurIPS 2026 · similarity 0.88)
- Compositional Reasoning in Language Models under Reinforcement Learning Post-Training (NeurIPS 2026 · similarity 0.88)
- Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoning (ICLR 2026 · similarity 0.87)
- How Do Large Language Monkeys Get Their Power (Laws)? (ICML 2025 · similarity 0.87)
- Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening (ICML 2026 · similarity 0.87)
- Are Transformers Able to Reason by Connecting Separated Knowledge in Training Data? (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
