Research map / Large language models / Reasoning and chain of thought

SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SRPO, which turns an LLM's own reflections on failed trajectories into reflection-conditioned teacher scores used for on-policy distillation with dense token-level reverse-KL rewards, reporting gains on math and agentic benchmarks with fewer FLOPs than GRPO.

Problem

Sparse terminal rewards in long-horizon LLM post-training give little credit-assignment signal, and existing reflection methods either act only at inference time, double compute, or do local repair without training the base policy.

Approach

The same model is used as a teacher by prepending a self-generated reflection patch to the prompt; the student (without the patch) samples on-policy rollouts, and per-token log-ratio between the reflection-conditioned teacher and the student serves as a dense advantage in a PPO-style update, with a reset-with-memory design instead of iterative appending.

Key results

• SRPO on Qwen3-8B reaches 73.3% on AIME'24 vs 68.0 for GRPO and 70.0 for OPD with Qwen3-32B teacher, per the main table.
• SRPO reports 64.7% WebShop, 76.8% ALFWorld and 31.2% SWE-Bench-Lite success/pass rates.
• Reported total training cost is 5.4e18 FLOPs vs 20.8e18 for GRPO (~3.8x fewer), though the main text also quotes 0.08x and 0.26x FLOPs ratios that are inconsistent with this.

Surprising findings

• Self-distillation with Qwen3-8B's own reflections (73.3) reportedly outperforms distillation from a Qwen3-72B teacher (72.5) on AIME'24.
• Random reflection patches of equal token length reportedly match no-reflection performance (66.5 vs 65.8), suggesting semantic content drives gains.

Limitations

• Reflection quality limited by model knowledge; failures include generic advice (42%), incorrect diagnosis (35%), and beyond-capability cases (23%).
• Inconsistent FLOPs and efficiency claims across abstract, main text, and appendix (0.08x, 0.26x, 3.8x, ~10x).
• Small AIME'24 evaluation set (30 problems) with high per-seed variance.

Experimental setup

Datasets: OpenThoughts-3 AIME'24 WebShop ALFWorld SWE-Bench-Lite

Benchmarks: AIME'24 MATH-500 GSM8K DeepScaleR WebShop ALFWorld SWE-Bench-Lite LiveCodeBench

Metrics: accuracy success rate pass@1 training FLOPs average episode steps

Baselines: SFT GRPO PPO On-Policy Distillation (Qwen3-32B teacher) On-Policy Distillation (Qwen3-72B teacher) Reflexion Self-Refine SCoRe RISE R3L

Builds on: GRPO PPO on-policy distillation Reflexion SCoRe

Closest papers by meaning

• SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• Incorporating Self-Rewriting into Large Language Model Reasoning Reinforcement (AAAI 2026 · similarity 0.90)
• MM-HELIX: Boosting Multimodal Long-Chain Reflective Reasoning with Holistic Platform and Adaptive Hybrid Policy Optimization (ICLR 2026 · similarity 0.89)
• Stop Unnecessary Reflection: Training LRMs for Efficient Reasoning with Adaptive Reflection and Length Coordinated Penalty (ICLR 2026 · similarity 0.89)
• Reinforce LLM Reasoning through Multi-Agent Reflection (ICML 2025 · similarity 0.89)
• One-Way Policy Optimization for Self-Evolving LLMs (ICML 2026 · similarity 0.89)