Research map / Reinforcement learning / RL from feedback

Leveraging Error Diversity in Group Rollouts for Reinforcement Learning

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: Scaling RL with verifiable rewards · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes EDAS, a post-hoc advantage shaping method for RLVR that reweights penalties on incorrect group rollouts by the entropy-based diversity of their final wrong answers (or exception types for code), improving GRPO/DAPO on math and code.

Problem

Group-based RLVR methods use only binary correctness per rollout and discard the distribution of errors among incorrect rollouts within a group, even though error diversity appears to predict training gains.

Approach

Partition incorrect rollouts into error classes (canonical final answer or exception type), then redistribute their negative advantages by normalized relative self-information: frequent errors get amplified penalties, rare errors attenuated, with uniform extra penalty when all wrong answers are identical, plus a kappa-clipping to avoid sign inversion. Algorithm-agnostic and applied on top of GRPO

Key results

• EDAS w/ DAPO improves average math accuracy over DAPO by 6.29 points on Qwen3-8B (54.11 vs 47.82) across seven benchmarks.
• On code generation with Qwen3-4B, DAPO w/ EDAS averages 55.64 vs 54.19 for DAPO (+1.45), with +3.55 on Codeforces.
• On zero-APR hard problems for Qwen3-8B, EDAS breaks through 30 of 64 vs 22 for DAPO (46.9% vs 34.4%).

Surprising findings

• Embedding-based grouping of rollouts merges trajectories with different numerical answers in 76.4% of cases (cosine similarity >0.95), so embeddings are unsuitable for error partitioning.
• The kappa-clipping step can break the exact zero-sum property of the redistribution, which the authors acknowledge as a deviation.

Limitations

• Error partitioning relies on observable outputs (final answer or exception type), missing differences in reasoning paths that yield the same answer.
• Equivalence function must be manually designed per domain.
• Hyperparameters alpha, beta, kappa are fixed across training and group size fixed at 10, so entropy estimates may be noisy.

Experimental setup

Datasets: DAPO-Math-17K Code-R1

Benchmarks: AMC23 AMC24 AIME24 AIME25 AIME26 HMMT25 OlympiadBench LiveCodeBench Codeforces HumanEval+ MBPP+

Metrics: average accuracy (repeat-32) Pass@k breakthrough rate on zero-APR problems unique wrong answers / diverse group count

Baselines: GRPO DAPO PKPO DAPO w/ Entropy Advantage

Builds on: GRPO DAPO PKPO

Closest papers by meaning

• SALT: When More Rollouts Don’t Help in Group-Based Policy Optimization and How to Make Them Matter (NeurIPS 2026 · similarity 0.88)
• What are Key Factors for Updates in RL for LLM Reasoning? (NeurIPS 2026 · similarity 0.87)
• Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex (NeurIPS 2026 · similarity 0.87)
• Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversity (ICLR 2026 · similarity 0.87)
• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.86)
• Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration (ICML 2026 · similarity 0.86)