Research map / Large language models / Reasoning and chain of thought
Prototype Entropy Alignment: Reinforcing Structured Uncertainty in LLM Reasoning
Zhengyuan Pan, Yanhao Chen, Zhongquan Jian, Wanru Zhao, Haonan Ma, Meihong Wang, Qingqiang Wu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Prototype Entropy Alignment (PEA), an RL framework that clusters expert trajectories' token-entropy patterns into learnable "entropy signature" prototypes and aligns model reasoning to them.
Problem
Reasoning quality in LLMs is driven by a minority of high-entropy tokens, but existing RL methods do not model the structured uncertainty patterns of effective reasoning as a whole.
Approach
Instead of treating effective reasoning as a single path, model it as a collection of entropy signatures: cluster expert trajectories' uncertainty patterns into prototypes and use them as RL alignment targets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add dynamic prototype updates (periodic re-clustering) instead of static prototypes (ablation) | PEA on Qwen2.5-7B · AIME 2024 | accuracy: 9.1 -> 11.9 | Improved |
| add dynamic prototype updates instead of static prototypes (ablation) | PEA on Qwen2.5-7B · NaturalReasoning | accuracy: 21.3 -> 65.8 | Improved |
| add dynamic prototype updates (reward variance) (ablation) | PEA on Qwen2.5-7B · AIME 2024 | reward variance: 0.312 -> 0.452 | Improved |
| restrict entropy signature to top 20% high-entropy tokens instead of all tokens (ablation) | PEA on Qwen2.5-7B · training reward and gain | performance gain and average reward | Improved |
| use 20% instead of 10% or 30% high-entropy tokens (ablation) | PEA on Qwen2.5-7B · training reward and gain | performance gain | Improved |
| add PEA reward to C/I reward | Qwen2.5-7B-Instruct · AMC23 | Pass@1: 45.8 -> 48.7 | Improved |
| add PEA reward alone on a standard instruct model | Qwen2.5-7B-Instruct · AMC23 | Pass@1: 42.3 -> 40.3 | Hurt |
| add PEA reward alone on a standard instruct model | Qwen2.5-1.5B-Instruct · AIME | Pass@1: 2.6 -> 4.7 | Improved |
| add PEA reward alone on a reasoning-specialized model | DeepSeek-R1-Distill-Qwen-7B · AIME | Pass@1: 47.1 -> 55.3 | Improved |
| add outcome C/I reward alone on a reasoning-specialized model | DeepSeek-R1-Distill-Qwen-7B · MATH500 | Pass@1: 93.6 -> 90.6 | Hurt |
| add PEA + C/I on a reasoning-specialized model | DeepSeek-R1-Distill-Qwen-7B · AIME | Pass@1: 47.1 -> 56.7 | Improved |
| add C/I reward alone on a standard instruct model | Qwen2.5-1.5B-Instruct · AIME | Pass@1: 2.6 -> 16.8 | Improved |
Experimental setup
Datasets: expert reasoning trajectories
Builds on: high-entropy token RL
Closest papers by meaning
- Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.90)
- Reasoning with Exploration: An Entropy Perspective (AAAI 2026 · similarity 0.90)
- The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning (NeurIPS 2025 · similarity 0.90)
- DiffAdapt: Difficulty-Adaptive Reasoning for Token-Efficient LLM Inference (ICLR 2026 · similarity 0.89)
- PEAR: Phase Entropy Aware Reward for Efficient Reasoning (ICLR 2026 · similarity 0.89)
- ARES: Multimodal Adaptive Reasoning via Difficulty-Aware Token-Level Entropy Shaping (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
