Research map / Large language models / Reasoning and chain of thought

Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptation

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies an implicit advantage symmetry in GRPO's group relative advantages and proposes A-GRAE, which attenuates positive advantages and shifts sample-difficulty focus from easy to hard during training, improving GRPO, DAPO and Dr.GRPO on math and multimodal reasoning.

Problem

GRPO's exploration efficiency and difficulty adaptation are limited: it tends to shrink reasoning boundaries (Pass@k drops) and implicitly focuses on medium-difficulty samples regardless of training stage.

Approach

Shows that GRAE's zero-sum advantages make unsampled trajectories receive no direct reward signal and that sum of absolute advantages peaks at p=0.5; proposes A-GRAE that scales positive advantages by min(1, omega/alpha) and mixes hard- and easy-focused 1/sqrt(p) and 1/sqrt(1-p) rescalings weighted by the batch mean reward, so the focus moves from easy to hard over training.

Key results

• On AIME 2025 with Qwen2.5-Math-7B, DAPO + A-GRAE reaches 32.9 average Pass@k versus 30.5 for DAPO.
• On MATH, Dr.GRPO + A-GRAE achieves 91.3 average Pass@k versus 89.3 for Dr.GRPO.
• On medical imaging with Qwen2.5-VL-3B, GRPO + A-GRAE improves Xray300 from 63.2 to 71.3 Pass@1.

Surprising findings

• Naively promoting entropy (entropy-matched control) degrades performance and induces repetitive outputs, so gains are attributed to trajectory-level asymmetry rather than entropy inflation.
• Plain GRPO's Pass@256 falls below the base model on AMC23 and MATH.

Limitations

• Negative-dominant down-weighting of correct trajectories can cause training instability and collapse in later stages, motivating the attenuation schedule.
• Evaluation of key models is limited to a few benchmarks and model sizes (up to 7B).

Experimental setup

Datasets: MATH HuatuoGPT-Vision Geo3K

Benchmarks: MATH AMC23 AIME 2025 Geo3K MathVision MathVerse HuatuoGPT-Vision

Metrics: Pass@k Pass@1 entropy

Baselines: GRPO DAPO Dr.GRPO W-REINFORCE P@kT GRPO-LEAD SEC

Builds on: GRPO DAPO Dr.GRPO REINFORCE++ verl

Closest papers by meaning

• Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPO (NeurIPS 2026 · similarity 0.88)
• Stratified GRPO: Handling Structural Heterogeneity in Reinforcement Learning of LLM Search Agents (ICML 2026 · similarity 0.88)
• $f$-GRPO & Beyond: Divergence-Based Reinforcement Learning Algorithms for General LLM Alignment (NeurIPS 2026 · similarity 0.87)
• On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization (NeurIPS 2025 · similarity 0.87)
• PAIR: Prefix-Aware Internal Reward Model for Multi-Turn Agent Optimization (NeurIPS 2026 · similarity 0.87)
• STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability (NeurIPS 2026 · similarity 0.87)