Research map / Large language models / Reasoning and chain of thought

Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Supervised Reinforcement Learning (SRL), which decomposes expert solutions into step-wise actions and rewards the model's own inner-monologue-plus-action with a difflib sequence similarity reward under GRPO, improving math reasoning on hard s1K data and SWE-bench agent tasks.

Problem

Outcome-based RLVR gets sparse rewards on problems the model almost never solves, while SFT overfits to long expert demonstrations via rigid token imitation, leaving small LLMs unable to learn difficult multi-step reasoning.

Approach

Treat each step of an expert solution as an action in a sequential decision process; the policy generates a private think block then a step, and receives a dense reward equal to difflib sequence similarity between its action and the expert's step, optimized with GRPO and step-level dynamic sampling.

Key results

• On math, SRL averages 27.6 vs 24.6 for the base Qwen2.5-7B-Instruct and 24.5 for RLVR; SRL → RLVR reaches 28.3 average.
• On SWE-Bench-Verified, SRL achieves 14.8% oracle-file-edit resolve rate vs 8.4% for SWE-Gym-7B, and 8.6% end-to-end vs 4.2%.
• Plain SFT on s1K degrades performance (S1K-7B average 11.7; SFT R1 reasoning 16.6) relative to the base model's 24.6.

Surprising findings

• Standard SFT on the s1K data substantially hurts the 7B base model, while RL-based methods preserve performance.
• Multi-step SRL outperforms the one-step holistic similarity reward, indicating step granularity matters, and SRL gains do not come from longer reasoning outputs.

Limitations

• Relies on high-quality, cleanly decomposable expert trajectories with well-defined step granularity.
• Requires the student to have baseline instruction-following ability so rollouts yield non-trivial rewards.
• Gains on the smallest 3B model are limited on extremely hard benchmarks such as AIME24 and AIME25.

Experimental setup

Datasets: s1K-1.1 SWE expert trajectories (5,000, claude-3-7-sonnet generated) SWE-Gym

Benchmarks: AMC23 AIME24 AIME25 Minerva Math SWE-Bench-Verified

Metrics: Avg@32 accuracy greedy accuracy resolve rate

Baselines: SFT S1K-7B RLVR (GRPO) SFT → RLVR R^3 SWE-Gym-7B Qwen2.5-Coder-Instruct

Builds on: GRPO DAPO DeepSeek-R1 R^3

Closest papers by meaning

• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.91)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.90)
• Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning (ICML 2024 · similarity 0.89)
• Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Models (ICLR 2026 · similarity 0.89)
• Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.89)
• BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning (NeurIPS 2025 · similarity 0.89)