Research map / Reinforcement learning / RL from feedback
Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling
Zeyu Huang, Tianhao Cheng, Zihan Qiu, Zili Wang, Xu Yinghui, Edoardo Ponti, Ivan Titov
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Sample a demonstration prefix and continue on-policy (Prefix-RFT) instead of pure RFT | Qwen2.5-Math-7B · Math reasoning average | average accuracy: 45.5 -> 51.8 | Improved |
| Add SFT loss directly during RFT (RL w/ SFT Loss) | Qwen2.5-Math-7B · Math reasoning average | average accuracy: 45.5 -> 40.1 | Hurt |
| Run SFT then RFT instead of RFT alone | Qwen2.5-Math-7B · Math reasoning average | average accuracy: 45.5 -> 48.2 | Improved |
| Use Prefix-RFT instead of LUFFY | Qwen2.5-Math-7B · Math reasoning average | average accuracy: 50.1 -> 51.8 | Improved |
| Use Prefix-RFT on a smaller model | Qwen2.5-Math-1.5B · Math reasoning average | average accuracy: 30.0 -> 41.1 (vs RFT) | Improved |
| Use Prefix-RFT for code generation | Qwen3-1.7B-Base · HumanEval+ | accuracy: 0.649 -> 0.689 | Improved |
| Reduce demonstration data from 45k to 0.45k (ablation) | Prefix-RFT, Qwen2.5-Math-1.5B · Math reasoning average | average accuracy: 40.8 -> 37.6 (full 41.1 -> 37.6) | Hurt |
| Use weaker demonstration generator (1.5B instead of 32B) (ablation) | Prefix-RFT, Qwen2.5-Math-1.5B · Math reasoning average | average accuracy: 40.6 -> 39.8 | No clear effect |
| Clip prefix tokens to top 20% highest entropy instead of top 50% or 80% (ablation) | Prefix-RFT · Benchmark performance | benchmark performance | Improved |
| Select high-entropy prefix tokens instead of random or bottom 20% (ablation) | Prefix-RFT · Benchmark performance | training reward and benchmark | Improved |
| Use cosine decay prefix-length scheduler instead of uniform scheduler (ablation) | Prefix-RFT · Math benchmarks | performance: uniform drop of -0.2 to -5.3 points | Improved |
| Use prefix only as guidance without learning from prefix loss (DR-GRPO style) (ablation) | Prefix-RFT · Math reasoning | performance | Hurt |
Closest papers by meaning
- UFT: Unifying Supervised and Reinforcement Fine-Tuning (NeurIPS 2025 · similarity 0.89)
- Vanishing Gradients in Reinforcement Finetuning of Language Models (ICLR 2024 · similarity 0.89)
- First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Training (NeurIPS 2025 · similarity 0.88)
- AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy (ICLR 2026 · similarity 0.88)
- Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections (NeurIPS 2025 · similarity 0.88)
- Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
