Research map / Large language models / Reasoning and chain of thought
S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes S-GRPO, an RL method that samples one full reasoning path, inserts early-exit interventions at random positions to form a serial group, and assigns exponentially decaying rewards to correct early exits, reducing reasoning length by 35-61% while improving accuracy on math and GPQA benchmarks.
Problem
Reasoning models overthink, generating redundant chain-of-thought, because outcome-only 0/1 RL rewards (e.g., GRPO) ignore whether intermediate reasoning steps are already sufficient.
Approach
Replace GRPO's parallel group of independent samples with a serial group: one reasoning path is truncated at several random positions and forced to answer early; correct answers receive rewards that decay with exit position (1/2^(N_right-1)), so earlier correct exits are favored, and advantages are computed relative to this serial group.
Key results
- On DeepSeek-R1-Distill-Qwen-7B, S-GRPO reaches 76.10 overall accuracy (+6.08) with 61.1% fewer tokens (3,556 vs 9,147).
- On DeepSeek-R1-Distill-Qwen-14B, S-GRPO improves overall accuracy by +0.72 to 81.08 while cutting tokens by 35.4%.
- On Qwen3-8B, S-GRPO improves overall accuracy by +2.36 to 84.26 with 40.6% fewer tokens, outperforming DEER and GRPO on accuracy and length.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with S-GRPO instead of the strongest accuracy baseline, GRPO | DeepSeek-R1-Distill-Qwen-7B · Overall | accuracy: 76.00 -> 76.10 | Helped |
| Train with S-GRPO instead of RL + Length Penalty | DeepSeek-R1-Distill-Qwen-7B · Overall | token count: 3,700 -> 3,556 | Helped |
| Train with S-GRPO instead of DEER | DeepSeek-R1-Distill-Qwen-14B · Overall | accuracy: 80.36 -> 81.08 | Helped |
| Train with S-GRPO instead of RL + Length Penalty | DeepSeek-R1-Distill-Qwen-14B · Overall | token count: 3,699 -> 3,253 | Helped |
| Train with S-GRPO instead of DEER | Qwen3-8B · Overall | accuracy: 82.62 -> 84.26 | Helped |
| Train with S-GRPO instead of RL + Length Penalty | Qwen3-8B · Overall | token count: 4,914 -> 4,922 | Hurt |
| Train with S-GRPO instead of GRPO | Qwen3-14B · Overall | accuracy: 85.46 -> 85.80 | Helped |
| Train with S-GRPO instead of RL + Length Penalty | Qwen3-14B · Overall | token count: 4,604 -> 4,322 | Helped |
| Add decaying rewards rather than rewarding only the shortest correct response (ablation) | S-GRPO on Qwen3-8B · Overall | accuracy: 81.70 -> 84.26 | Helped |
| Add decaying rewards rather than assigning 1 to every correct response (ablation) | S-GRPO on Qwen3-8B · Overall | token count: 8,179 -> 4,922 | Helped |
| Add Serial-Group Generation to the all-1 reward configuration (ablation) | S-GRPO on Qwen3-8B · Overall | accuracy: 82.30 -> 84.26 | Helped |
Surprising findings
- Plain GRPO on DeepSeek-R1-Distill-Qwen-14B lowered overall accuracy (-1.18) while increasing tokens by 18.9%.
- Removing decaying rewards (all correct answers rewarded 1) causes token count to rise by 66.2% over the S-GRPO model on Qwen3-8B, showing the decay is what drives length reduction.
Limitations
- Gains on the 14B DeepSeek distill are small (+0.72 accuracy overall), and some baselines reported as higher on single benchmarks.
- Training uses only math problems from DeepMath-103K, so generalization to other domains relies on GPQA alone.
- Several baseline cells are missing ('--'), so comparisons are incomplete across models.
Experimental setup
Datasets: DeepMath-103K
Benchmarks: GSM8K AIME 2024 AMC 2023 MATH-500 GPQA Diamond
Metrics: accuracy (pass@1) token count
Baselines: DEER ConCISE GRPO RL + Length Penalty ShorterBetter
Builds on: GRPO DAPO DeepSeek-R1
Closest papers by meaning
- Fast-Slow Thinking GRPO for Large Vision-Language Model Reasoning (NeurIPS 2025 · similarity 0.90)
- Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning (ICML 2026 · similarity 0.89)
- Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO (NeurIPS 2025 · similarity 0.88)
- All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Models (CVPR 2026 · similarity 0.88)
- Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.