Research map / Large language models / Reasoning and chain of thought
Group Distributionally Robust Optimization-Driven RL for LLM Reasoning
Kishan Panaganti, Zhenwen Liang, Wenhao Yu, Haitao Mi, Dong Yu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Casts GRPO post-training as two independent group distributionally robust optimization (GDRO) games over dynamically defined difficulty groups, targeting heavy-tailed hard reasoning problems.
Problem
GRPO reasoning post-training uses static uniform prompt sampling and a fixed number of rollouts per prompt, which wastes compute on solved patterns and under-trains the long tail of hard problems in heterogeneous, heavy-tailed reasoning data.
Approach
Replaces static uniformity with two independent GDRO games over dynamic difficulty groups defined by an online training signal, adapting both prompt sampling and rollout allocation toward hard groups.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Prompt-GDRO adversarial prompt reweighting | GRPO · Qwen3-4B-Base, DAPO-14.1k | pass@8: 56.31 -> 63.70 (+13.13%) | Improved |
| add Prompt-GDRO | GRPO · Qwen3-1.7B-Base | pass@8: 50.74 -> 55.68 (+9.74%) | Improved |
| add Prompt-GDRO | GRPO · Qwen3-8B-Base | pass@8: 62.04 -> 67.60 (+8.96%) | Improved |
| add Rollout-GDRO adaptive rollout allocation | GRPO · Qwen3-1.7B-Base | pass@8: 50.74 -> 56.14 (+10.64%) | Improved |
| add Rollout-GDRO | GRPO · Qwen3-8B-Base | pass@8: 62.04 -> 67.75 (+9.20%) | Improved |
| rerun Prompt-GDRO with a second seed (ablation) | Prompt-GDRO · Qwen3-4B-Base | pass@8: 56.36 -> 63.47 | Improved |
| vary bins, history, adversary LR, weight cap (ablation) | Prompt-GDRO · Qwen3-4B-Base | pass@8: within -1.31 to +0.41 pp of 63.70 | No clear effect |
| use online Rollout-GDRO instead of fixed hard-focused static rollout schedule (ablation) | Rollout-GDRO · Qwen3-4B-Base | pass@8: 56.70 -> 62.27 | Improved |
| focus fixed rollouts on hard instead of easy prompts (ablation) | StaticRollout · Qwen3-4B-Base | pass@8: 54.34 -> 56.70 | Improved |
| vary bins, arms, dual LR (ablation) | Rollout-GDRO · Qwen3-4B-Base | pass@8: 0.81 pp spread | No clear effect |
| add Rollout-GDRO (second seed) (ablation) | GRPO · Qwen3-4B-Base MINERVA | mean@8: 30.48 -> 27.28 | Hurt |
| add Prompt-GDRO | GRPO · Qwen3-1.7B-Base MINERVA | mean@8: 14.56 -> 14.61 | No clear effect |
Experimental setup
Baselines: GRPO
Builds on: GRPO
Closest papers by meaning
- Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts (NeurIPS 2025 · similarity 0.90)
- Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning (ICLR 2026 · similarity 0.89)
- Multi-Task GRPO: Reliable LLM Reasoning Across Tasks (ICML 2026 · similarity 0.89)
- WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning (ICML 2026 · similarity 0.89)
- DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization (NeurIPS 2025 · similarity 0.88)
- Nudging the Boundaries of LLM Reasoning (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
