Research map / Large language models / Reasoning and chain of thought

Conditional Advantage Estimation for Reinforcement Learning in Large Reasoning Models

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CANON, a conditional advantage estimation that splits sampled responses by a metric (entropy or length) and combines inter-group and intra-group advantages, outperforming DR.GRPO on math and logic reasoning with Qwen2.5-Math-7B and related models.

Problem

Directional handcrafted priors (higher- or lower-is-better) on training metrics such as entropy or response length, used in reward or advantage shaping for RLVR, can be overly biased and fail without careful tuning.

Approach

Regroup sampled responses into two halves by a training metric value, compute an inter-group advantage (compare a response to the other group) that identifies which metric trend yields higher reward, and an intra-group advantage (compare within the same group), then mix them with weight mu; DR.GRPO is the mu=0.5 special case.

Key results

• CANON-Inter based on entropy reaches 57.6 average accuracy across six math benchmarks versus 55.7 for DR.GRPO on Qwen2.5-Math-7B (+1.9).
• CANON-Intra based on entropy improves the hardest ZebraLogic XLarge subset by 5.2 points over DR.GRPO (20.3 vs 15.1).
• CANON-Eff (alpha=0.88) reduces token consumption by 45.5% at the same performance level as Length Reward (*) and achieves 2.63x the DR.GRPO performance in low-budget settings.

Surprising findings

• Random regrouping matches baseline performance while producing longer responses, indicating the metric choice matters.
• Length Reward (+) performance dropped from 54.8 to 22.5 when its penalty coefficient was changed from 0.004 to 0.005, showing baseline instability.

Limitations

• Only numerically orderable conditions (entropy and length) are explored; other metrics unexplored.
• Only one metric considered at a time; multi-metric combinations not attempted.
• Experiments use one main 7B model family plus Llama3.1-8B and Qwen2.5-Math-1.5B; compute scale and seed variance not reported.

Experimental setup

Datasets: OpenR1-Math-220k GSM8k MATH DeepMath-103k ZebraLogic

Benchmarks: AIME 24 AIME 25 OlympiadBench AMC MATH-500 GSM8K ZebraLogic

Metrics: Pass@1 Avg@10 accuracy token count budget-performance curve cost-performance Pareto frontier

Baselines: Qwen2.5-Math-7B-Instruct ReMax REINFORCE++ RLOO GRPO DR.GRPO Clip Length Length Reward

Builds on: GRPO DR.GRPO DAPO PPO VeRL

Closest papers by meaning

• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.92)
• The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning (NeurIPS 2025 · similarity 0.91)
• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.91)
• On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation (ICLR 2026 · similarity 0.91)
• Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.91)
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.91)