Research map / Reinforcement learning / RL from feedback
BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BREAD, a GRPO variant that branches rollouts from short expert-trace prefixes (found by binary-search episode anchor search) when self-generated success is zero, unifying SFT and RL for small reasoning models.
Problem
Small language models struggle to learn complex reasoning: SFT on expert traces can be too complex to imitate, and RL (GRPO) suffers sparse rewards when the base model rarely produces correct traces.
Approach
When all self-generated rollouts fail, binary-search a short expert hint prefix such that the success rate is non-zero, then run GRPO on hint-conditioned branched rollouts, densifying rewards and inducing a curriculum along the reasoning trace, with theory in a Markov random-walk model showing SFT+RL can fail while branching succeeds.
Key results
- BREAD improves final accuracy by more than 15% over vanilla GRPO on NuminaMath-CoT with Qwen2.5-3B-Instruct.
- BREAD reaches the best baseline's accuracy in about 25% of training steps, roughly 75% reduction in estimated training FLOPs.
- On the MATH dataset with Qwen-1.5B-Instruct, BREAD reaches 0.788 versus 0.774 for SFT(full)+GRPO and 0.590 for GRPO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BREAD instead of GRPO | Qwen-1.5B-Instruct · MATH | accuracy: 0.590 -> 0.788 | Helped |
| Use BREAD instead of GRPO | Qwen-3B-Instruct · NuminaMath-CoT | accuracy: 0.475 -> 0.647 | Helped |
| Use BREAD instead of SFT(full)+GRPO | Qwen-3B-Instruct · NuminaMath-CoT | accuracy: 0.537 -> 0.647 | Helped |
| Add SFT on S1K traces (ablation) | Qwen2.5-1.5B-Instruct · GPQA | accuracy: 0.258 -> 0.177 | Hurt |
| Add SFT on S1K-1.1 traces (ablation) | Qwen2.5-1.5B-Instruct · GPQA | accuracy: 0.258 -> 0.121 | Hurt |
| Add SFT on S1K traces (ablation) | Qwen2.5-3B-Instruct · MATH | accuracy: 0.624 -> 0.616 | Hurt |
| Add SFT on S1K-1.1 traces (ablation) | Qwen2.5-3B-Instruct · MATH | accuracy: 0.624 -> 0.630 | Helped |
| Use BREAD instead of SFT(full)+GRPO | Qwen-1.5B-Instruct · NuminaMath-CoT | accuracy: 0.242 -> 0.361 | Helped |
| Use partial expert hints during RL rather than full-trace SFT | Qwen2.5-3B-Instruct · NuminaMath-CoT hard questions | accuracy | Helped |
| Reduce BREAD rollout budget from 8 to 5 (ablation) | Qwen2.5-3B-Instruct · MATH | accuracy | No clear effect |
| Reduce GRPO rollout budget from 8 to 5 (ablation) | Qwen2.5-3B-Instruct · MATH | accuracy | Hurt |
| Train BREAD rather than SFT+GRPO | Qwen-2.5-3B-Instruct · NuminaMath-CoT | training steps to reach best-baseline accuracy: 25% | Helped |
Surprising findings
- SFT on expert traces from stronger models decreased accuracy of Qwen2.5 1.5B and 3B on GPQA and MATH (e.g., GPQA 0.258 to 0.177 for 1.5B with S1K).
Limitations
- Assumes access to a strong expert model that provides correct traces.
- Student may fail to gain reward even from partial expert traces if it cannot follow the argument.
- Theory is based on a simplified random-walk/Markov model rather than real LLM reasoning.
Experimental setup
Datasets: MATH NuminaMath-CoT OpenR1-Math-220K S1K S1K-1.1 GPQA
Benchmarks: MATH NuminaMath-CoT GPQA
Metrics: accuracy training steps to accuracy FLOPs fraction of expert traces used
Baselines: GRPO SFT+GRPO SFT(full)+GRPO SFT(random)+GRPO SFT(selected)+GRPO GRPO w/ Expert Trace DeepSeek-R1-Distill
Builds on: GRPO DAgger Go-Explore Kickstarting
Closest papers by meaning
- RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.92)
- Advancing LLM Reasoning with Natural Language and Numerical Feedback (ICML 2026 · similarity 0.91)
- RAST: Reasoning Activation in LLMs via Small-model Transfer (NeurIPS 2025 · similarity 0.90)
- Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.90)
- Tina: Tiny Reasoning Models via LoRA (ICLR 2026 · similarity 0.90)
- Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.