Research map / Reinforcement learning / RL from feedback
Single-stream Policy Optimization
Zhongwen Xu, Zihan Ding
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace GRPO with SPO | Qwen3-8B tool-integrated RL · Average of five benchmarks | maj@32: 60.4 -> 63.8 | Improved |
| replace GRPO with SPO | Qwen3-8B tool-integrated RL · BRUMO 25 | maj@32: 56.7 -> 64.0 | Improved |
| replace GRPO with SPO | Qwen3-8B tool-integrated RL · AIME 25 | maj@32: 72.1 -> 76.5 | Improved |
| replace GRPO with SPO | Qwen3-8B tool-integrated RL · HMMT 25 | maj@32: 44.2 -> 47.5 | Improved |
| replace GRPO with SPO | Qwen3-8B tool-integrated RL · AIME 24 | avg@32: 77.6 -> 74.9 | Hurt |
| replace GRPO with SPO | Qwen3-8B tool-integrated RL · Average of five benchmarks | avg@32: 55.7 -> 56.0 | No clear effect |
| use group-free collection of the first 24 finished samples from 48 instead of group-based collection | agentic rollout simulation · Agentic batch-collection simulation | training throughput speedup: 486s -> 112s (4.35x) | Improved |
| add the adaptive KL-based value tracker baseline to SPO (vs no baseline) (ablation) | SPO · AIME 25 (avg@16, ablation setting) | avg@16 | Improved |
| use adaptive tracker baseline instead of static A*-PO baseline (ablation) | SPO · AIME 25 (avg@16, ablation setting) | avg@16 | Improved |
| add offline initialization of the value tracker (ablation) | SPO · AIME 25 (avg@16, ablation setting) | avg@16 | Improved |
| use SPO history-informed baseline instead of raw rewards (ablation) | SPO · Advantage variance analysis | advantage variance: nearly 50% variance reduction | Improved |
| use SPO instead of GRPO regarding ineffective samples | GRPO · Training dynamics | share of zero-advantage samples: roughly 60% to over 80% (GRPO degenerate groups) | Improved |
Closest papers by meaning
- Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.91)
- One-Way Policy Optimization for Self-Evolving LLMs (ICML 2026 · similarity 0.91)
- Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.90)
- Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex (NeurIPS 2026 · similarity 0.90)
- Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning (ICLR 2026 · similarity 0.90)
- SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning (ICML 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
