AI Paper Atlas

Research map / Large language models / Reasoning and chain of thought

Quantile Advantage Estimation for Entropy-Safe Reasoning

Junkang Wu, Kexin Huang, Jiancan Wu, An Zhang, Xiang Wang, Xiangnan He

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Quantile Advantage Estimation (QAE), replacing the group-mean baseline in value-free RLVR (GRPO/DAPO) with a group-wise K-quantile to keep entropy stable during LLM reasoning training.

Problem

RLVR training for LLM reasoning oscillates between entropy collapse and entropy explosion; the authors trace this to the mean baseline in value-free RL methods like GRPO and DAPO, which mispenalizes negative-advantage samples under reward outliers.

Approach

Replace the group mean baseline used for advantage computation with a group-wise K-quantile baseline, so that reward outliers do not distort advantages and negative-advantage samples are not improperly penalized, yielding entropy-safe training.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace mean baseline with K-quantile baseline (QAE, K=0.4)DAPO Clip-Higher on Qwen3-8B-Base · AIME24 pass@1accuracy: 39.69 -> 48.23Improved
add QAEClip-Cov on Qwen3-8B-Base · AIME25 pass@1accuracy: 33.02 -> 37.40Improved
add QAEKL-Cov on Qwen3-8B-Base · AIME24 pass@1accuracy: 44.90 -> 44.69No clear effect
add QAEGSPO on Qwen3-30B-A3B-Base · AIME24 pass@1accuracy: 43.75 -> 47.50Improved
add QAEGSPO on Qwen3-30B-A3B-Base · AMC23 pass@1accuracy: 90.00 -> 89.38No clear effect
add QAEDAPO on Qwen3-14B-Base · AIME24 pass@1accuracy: 56.56 -> 58.96Improved
add QAEDAPO Clip-Higher · pass@16 on AIME/AMCaccuracy: AIME24 71.23 -> 71.63No clear effect
tune token-level clip threshold epsilon_high from 0.20 to 0.28 (ablation)DAPO · AIME24 pass@1accuracy: 32.29 -> 40.63 (0.26) -> 39.69 (0.28)Mixed
quantile baseline versus mean baseline for entropyDAPO · entropy dynamicspolicy entropyImproved
use large K (0.8) instead of 0.4 (ablation)QAE · Qwen3-8B-Base AIME24entropy and accuracyHurt
use small K (0.2) instead of 0.4 (ablation)QAE · Qwen3-8B-Base AIME24accuracyHurt
mask only negative-advantage updates versus only positive (ablation)QAE with weak clipping (epsilon_high 0.28) · AIME24/25 pass@1accuracyMixed

Earlier methods this paper compares with

Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Surprising findings

Experimental setup

Metrics: entropy stability

Baselines: GRPO DAPO

Builds on: GRPO DAPO

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy