Research map / Reinforcement learning / RL from feedback
C-GRPO: Conformal Group Relative Policy Optimization
Arya Fayyazi, Seyedarmin Azizi, Mehdi Kamal, Massoud Pedram
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes C-GRPO, a GRPO variant that adapts the number of on-policy rollouts per prompt using a conformal stopping rule with a finite-sample calibration guarantee.
Problem
Standard GRPO uses a fixed group size, wasting compute on easy prompts and under-exploring hard ones; existing adaptive-budget methods use heuristic stopping rules without finite-sample calibration guarantees.
Approach
Replaces heuristic adaptive rollout stopping with a conformal calibration procedure that provides a finite-sample guarantee on the stopping decision within GRPO training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace fixed G=32 GRPO rollouts with conformal adaptive budgets | GRPO on Qwen2.5-7B-Instruct · six benchmarks | training rollouts: 49-70% fewer rollouts at matched accuracy | Improved |
| use C-GRPO instead of static GRPO | Qwen2.5-7B-Instruct · GSM8K | pass@1 (%): 54.8 -> 57.2 | No clear effect |
| apply C-GRPO to a stronger base model | Llama-3.1-8B-Instruct · GSM8K | rollout savings: 68% -> 82% (mean rollouts 10.4 -> 5.7) | Improved |
| train fixed-G GRPO at matched compute G=10 (ablation) | GRPO · GSM8K | pass@1 (%): 53.1 vs 57.2 for C-GRPO | Hurt |
| use fixed miscoverage 0.05 instead of auto-delta (ablation) | C-GRPO · GSM8K | mean training rollouts: 10.4 -> 29.5; pass@1 57.2 -> 54.5 | Hurt |
| add automatic miscoverage selection (ablation) | C-GRPO · GSM8K | pass@1 (%): 54.5 -> 57.2 vs fixed 0.05 | Improved |
| recalibrate every 25 steps instead of 50 (ablation) | C-GRPO · GSM8K | pass@1 (%): 57.2 -> 57.3; +12 overhead | No clear effect |
| never recalibrate (static thresholds from step 0) (ablation) | C-GRPO · GSM8K | mean training rollouts: 10.4 -> 18.2 | Hurt |
| add periodic recalibration at 100-step interval versus 50 (ablation) | C-GRPO · GSM8K | pass@1 (%): 57.2 -> 55.8 | Hurt |
| use coarse budget grid {2,8,32} instead of {2,4,8,16,32} (ablation) | C-GRPO · MATH-500 | pass@16 (%): 71.2 -> 69.5 | Hurt |
| use finer budget grid including k=1 (ablation) | C-GRPO · MATH-500 | pass@16 (%): 71.2 -> 70.8 | No clear effect |
| train C-GRPO for 1,000 GSM8K steps | C-GRPO · GSM8K | pass@1 (%): 52.1 -> 64.5; mean k 18.2 -> 4.2 | Improved |
Experimental setup
Baselines: GRPO adaptive-budget GRPO variants
Builds on: GRPO
Closest papers by meaning
- Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning (ICLR 2026 · similarity 0.92)
- WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning (ICML 2026 · similarity 0.92)
- Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning (ICLR 2026 · similarity 0.90)
- BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning (NeurIPS 2025 · similarity 0.89)
- GVPO: Group Variance Policy Optimization for Large Language Model Post-Training (NeurIPS 2025 · similarity 0.89)
- CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
