Research map / Reinforcement learning / RL from feedback
CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CPPO, which speeds up GRPO training by pruning completions with low absolute advantage and reallocating freed GPU capacity to new questions, reporting up to 7.98x speedup on GSM8K and 3.48x on MATH.
Problem
GRPO requires sampling many completions per question, making reinforcement learning training of reasoning models computationally expensive and training time scaling multiplicatively with completion count.
Approach
Use the prior-forward advantage term to decide which completions to keep before the policy/reference/old-policy forward passes: retain only the top-k completions by absolute advantage per question (pruning rate P), and dynamically refill freed device capacity with completions from new questions to keep GPUs fully utilized.
Key results
- On GSM8K with Qwen2.5-1.5B-Instruct, CPPO at 93.75% pruning reaches a 7.98x speedup, and 87.50% pruning gives 80.01% accuracy vs GRPO 77.38%.
- On MATH with Qwen2.5-7B-Instruct, CPPO at 87.50% pruning achieves 75.95% accuracy vs GRPO 75.26% with a 3.48x speedup.
- On Qwen2.5-14B-Instruct (MATH), CPPO reaches up to 2.83x speedup while accuracy is 78-79.4% vs GRPO 77.6%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| prune completions by absolute advantage at 87.50% (retain k=2) | GRPO training with Qwen2.5-1.5B-Instruct · GSM8K test subset | accuracy: 77.38 -> 80.01 | Helped |
| prune completions by absolute advantage at 93.75% (retain k=1) | GRPO training with Qwen2.5-1.5B-Instruct · GSM8K test subset | acceleration ratio: 1.00 -> $7.98 \pm 0.23$ | Helped |
| prune completions by absolute advantage at 87.50% (retain k=2) | GRPO training with Qwen2.5-7B-Instruct · MATH test subset | accuracy: 75.26 -> 75.95 | Helped |
| prune completions by absolute advantage at 93.75% (retain k=1) | GRPO training with Qwen2.5-7B-Instruct · MATH test subset | accuracy: 75.26 -> 74.65 | Hurt |
| prune completions by absolute advantage at 50.00% (retain k=8) | GRPO training with Qwen2.5-7B-Instruct · AMC2023 | Pass@1 accuracy: 46.88 -> 53.12 | Helped |
| prune completions by absolute advantage at 50.00% (retain k=8) | GRPO training with Qwen2.5-7B-Instruct · AIME2024 | Pass@1 accuracy: 5.83 -> 10.00 | Helped |
| prune completions by absolute advantage (ablation) | GRPO training with Qwen2.5-1.5B-Instruct · GSM8K test subset | accuracy: 55.72% -> 77.67% | Helped |
| prune completions with largest absolute advantages instead of smallest (ablation) | CPPO · GSM8K | accuracy: 77.67% -> 74.23% | Hurt |
| prune completions randomly instead of by absolute advantage (ablation) | CPPO · GSM8K | accuracy: 77.67% -> 76.98% | Hurt |
| prune by raw advantage instead of absolute advantage (ablation) | CPPO · GSM8K | accuracy: 77.67% vs. 76.83%; 73.32% | Hurt |
| add completion pruning (ablation) | GRPO with Qwen2.5-7B-Instruct · MATH | accuracy: 75.20% -> 75.80% | Helped |
| add completion pruning (ablation) | GRPO with Qwen2.5-7B-Instruct · MATH | acceleration ratio: 1.00× -> $1.23\times$ | Helped |
Surprising findings
- CPPO often matches or exceeds GRPO accuracy despite training on far fewer completions.
- Pruning by smallest absolute advantage outperforms random pruning, while pruning by largest absolute advantage performs worst.
Limitations
- Completion generation time is not reduced, so speedup shrinks when generation dominates training.
- Evaluated only on models under 14B and on math datasets (MATH, GSM8K).
Experimental setup
Datasets: GSM8K MATH
Benchmarks: GSM8K MATH AMC 2023 AIME 2024
Metrics: accuracy Pass@1 training time accelerate ratio
Baselines: GRPO DAPO Dr. GRPO REINFORCE++ PPO w/o KL
Builds on: GRPO PPO DAPO Dr. GRPO REINFORCE++
Closest papers by meaning
- Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning (ICLR 2026 · similarity 0.89)
- DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization (NeurIPS 2025 · similarity 0.88)
- WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning (ICML 2026 · similarity 0.88)
- S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models (NeurIPS 2025 · similarity 0.88)
- Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models (NeurIPS 2025 · similarity 0.87)
- Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPO (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.