Research map / Reinforcement learning / RL from feedback
Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Segment Policy Optimization (SPO), a critic-free RL framework that estimates advantages at intermediate segment granularity via Monte Carlo, with cutpoint-based chain estimation (SPO-chain) for short CoT and tree-based estimation (SPO-tree) for long CoT.
Problem
Credit assignment in RL for LLM reasoning: token-level critics (PPO) give inaccurate value estimates, while trajectory-level advantages (GRPO) give coarse, imprecise credit over long sequences.
Approach
Partition responses into contiguous segments and estimate segment advantages as differences of Monte Carlo value estimates at boundaries, without a critic. Uses adaptive cutpoint-based partitions (low-probability tokens) for short CoT, a tree-structured sampler that reuses MC rollouts for long CoT, and a probability-mask loss that applies segment advantages only to low-probability tokens.
Key results
- SPO-chain (int5) reaches about 56.7% on GSM8K test with RhoMath 1.1B, outperforming PPO and GRPO by 6-12 percentage points.
- SPO-tree trained on MATH at 2K context reaches 0.736 on MATH500 vs 0.62 for GRPO (about 11 points higher); at 4K it reaches 0.828 vs 0.752.
- Applying the probability-mask technique raises GRPO on GSM8K from 45.7% to 53.6%, while removing it lowers SPO-chain from 56.7% to 55.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SPO-chain (int5) instead of PPO or GRPO | RhoMath 1.1B · GSM8K | accuracy: +6-12 percentage points over PPO and GRPO | Helped |
| Use SPO-tree instead of GRPO | DeepSeek-R1-Distill-Qwen-1.5B · MATH500, 2K eval context | accuracy: 0.62 -> 0.736 | Helped |
| Use SPO-tree instead of GRPO | DeepSeek-R1-Distill-Qwen-1.5B · MATH500, 4K eval context | accuracy: 0.752 -> 0.828 | Helped |
| Use SPO-tree instead of GRPO | DeepSeek-R1-Distill-Qwen-1.5B · MATH500, 32K eval context | accuracy: 0.84 -> 0.848 | Helped |
| Use SPO-tree instead of GRPO | DeepSeek-R1-Distill-Qwen-1.5B · AIME24, 2K eval context | accuracy: 0.033 -> 0.1 | Helped |
| Use SPO-tree instead of GRPO | DeepSeek-R1-Distill-Qwen-1.5B · AIME24, 4K eval context | accuracy: 0.2 -> 0.2 | No clear effect |
| Use SPO-tree instead of GRPO | DeepSeek-R1-Distill-Qwen-1.5B · AIME24, 32K eval context | accuracy: 0.333 -> 0.333 | No clear effect |
| Add probability-mask optimization to SPO-chain (int5) (ablation) | SPO-chain (int5) · GSM8K test | accuracy: 55.0% -> 56.7% | Helped |
| Add probability-mask optimization to GRPO (ablation) | GRPO · GSM8K test | accuracy: 45.7% -> 53.6% | Helped |
| Use cutpoint-based partition instead of fixed-token or line-break partition (ablation) | SPO-chain · GSM8K test | accuracy | Helped |
| Use a moderate cutpoint interval (5) rather than interval 100 (ablation) | SPO-chain · GSM8K validation and test | accuracy | Helped |
| Use interval 2 rather than interval 5 (ablation) | SPO-chain · GSM8K final accuracy | accuracy | Helped |
Surprising findings
- Adding the probability-mask technique to GRPO gives a large gain (45.7% to 53.6% on GSM8K), larger than its effect on SPO-chain.
- DeepScaleR, trained with longer context and more data, performs worst at 2K and 4K evaluation despite being best at 32K, suggesting GRPO-based training may hurt token efficiency.
Limitations
- Long-CoT experiments limited to a 4K maximum context due to compute.
- Experiments focus mainly on mathematical tasks; broader tasks such as code and RLHF are left untested.
- Tree-based sampling is constrained by on-policy training, limiting how far trajectories can be reused across iterations.
Experimental setup
Datasets: GSM8K MATH Knights-and-Knaves
Benchmarks: GSM8K MATH500 AIME24 Knights-and-Knaves
Metrics: Pass@1 accuracy
Baselines: RestEM DPO PPO GRPO RLOO VinePPO DeepScaleR STILL-3
Builds on: PPO GRPO VinePPO GAE RLOO
Closest papers by meaning
- Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.90)
- VinePPO: Refining Credit Assignment in RL Training of LLMs (ICML 2025 · similarity 0.90)
- Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models (ICLR 2024 · similarity 0.89)
- Not Only Where, But When: Temporal Scheduling for RLVR (NeurIPS 2026 · similarity 0.88)
- Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs (ICLR 2026 · similarity 0.88)
- Conditional Advantage Estimation for Reinforcement Learning in Large Reasoning Models (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.