Research map / Large language models / Reasoning and chain of thought
Quantile Advantage Estimation for Entropy-Safe Reasoning
Junkang Wu, Kexin Huang, Jiancan Wu, An Zhang, Xiang Wang, Xiangnan He
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Quantile Advantage Estimation (QAE), replacing the group-mean baseline in value-free RLVR (GRPO/DAPO) with a group-wise K-quantile to keep entropy stable during LLM reasoning training.
Problem
RLVR training for LLM reasoning oscillates between entropy collapse and entropy explosion; the authors trace this to the mean baseline in value-free RL methods like GRPO and DAPO, which mispenalizes negative-advantage samples under reward outliers.
Approach
Replace the group mean baseline used for advantage computation with a group-wise K-quantile baseline, so that reward outliers do not distort advantages and negative-advantage samples are not improperly penalized, yielding entropy-safe training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace mean baseline with K-quantile baseline (QAE, K=0.4) | DAPO Clip-Higher on Qwen3-8B-Base · AIME24 pass@1 | accuracy: 39.69 -> 48.23 | Improved |
| add QAE | Clip-Cov on Qwen3-8B-Base · AIME25 pass@1 | accuracy: 33.02 -> 37.40 | Improved |
| add QAE | KL-Cov on Qwen3-8B-Base · AIME24 pass@1 | accuracy: 44.90 -> 44.69 | No clear effect |
| add QAE | GSPO on Qwen3-30B-A3B-Base · AIME24 pass@1 | accuracy: 43.75 -> 47.50 | Improved |
| add QAE | GSPO on Qwen3-30B-A3B-Base · AMC23 pass@1 | accuracy: 90.00 -> 89.38 | No clear effect |
| add QAE | DAPO on Qwen3-14B-Base · AIME24 pass@1 | accuracy: 56.56 -> 58.96 | Improved |
| add QAE | DAPO Clip-Higher · pass@16 on AIME/AMC | accuracy: AIME24 71.23 -> 71.63 | No clear effect |
| tune token-level clip threshold epsilon_high from 0.20 to 0.28 (ablation) | DAPO · AIME24 pass@1 | accuracy: 32.29 -> 40.63 (0.26) -> 39.69 (0.28) | Mixed |
| quantile baseline versus mean baseline for entropy | DAPO · entropy dynamics | policy entropy | Improved |
| use large K (0.8) instead of 0.4 (ablation) | QAE · Qwen3-8B-Base AIME24 | entropy and accuracy | Hurt |
| use small K (0.2) instead of 0.4 (ablation) | QAE · Qwen3-8B-Base AIME24 | accuracy | Hurt |
| mask only negative-advantage updates versus only positive (ablation) | QAE with weak clipping (epsilon_high 0.28) · AIME24/25 pass@1 | accuracy | Mixed |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- DAPO — DAPO: An Open-Source LLM Reinforcement Learning System at Scale (NeurIPS 2025) · better on 2 of 2 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Surprising findings
- Training instability in RLVR is attributed to the mean baseline mispenalizing negative-advantage samples under reward outliers, rather than only to the policy update itself.
Experimental setup
Metrics: entropy stability
Baselines: GRPO DAPO
Builds on: GRPO DAPO
Closest papers by meaning
- STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability (NeurIPS 2026 · similarity 0.91)
- Conditional Advantage Estimation for Reinforcement Learning in Large Reasoning Models (ICLR 2026 · similarity 0.90)
- Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.90)
- Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward (ICLR 2026 · similarity 0.88)
- Reasoning with Exploration: An Entropy Perspective (AAAI 2026 · similarity 0.88)
- The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
