AI Paper Atlas

Research map / Reinforcement learning / RL from feedback

Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling

Zeyu Huang, Tianhao Cheng, Zihan Qiu, Zili Wang, Xu Yinghui, Edoardo Ponti, Ivan Titov

ICML 2026 · Reinforcement learning › RL from feedback · approach: RL fine-tuning of generative models · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Sample a demonstration prefix and continue on-policy (Prefix-RFT) instead of pure RFTQwen2.5-Math-7B · Math reasoning averageaverage accuracy: 45.5 -> 51.8Improved
Add SFT loss directly during RFT (RL w/ SFT Loss)Qwen2.5-Math-7B · Math reasoning averageaverage accuracy: 45.5 -> 40.1Hurt
Run SFT then RFT instead of RFT aloneQwen2.5-Math-7B · Math reasoning averageaverage accuracy: 45.5 -> 48.2Improved
Use Prefix-RFT instead of LUFFYQwen2.5-Math-7B · Math reasoning averageaverage accuracy: 50.1 -> 51.8Improved
Use Prefix-RFT on a smaller modelQwen2.5-Math-1.5B · Math reasoning averageaverage accuracy: 30.0 -> 41.1 (vs RFT)Improved
Use Prefix-RFT for code generationQwen3-1.7B-Base · HumanEval+accuracy: 0.649 -> 0.689Improved
Reduce demonstration data from 45k to 0.45k (ablation)Prefix-RFT, Qwen2.5-Math-1.5B · Math reasoning averageaverage accuracy: 40.8 -> 37.6 (full 41.1 -> 37.6)Hurt
Use weaker demonstration generator (1.5B instead of 32B) (ablation)Prefix-RFT, Qwen2.5-Math-1.5B · Math reasoning averageaverage accuracy: 40.6 -> 39.8No clear effect
Clip prefix tokens to top 20% highest entropy instead of top 50% or 80% (ablation)Prefix-RFT · Benchmark performancebenchmark performanceImproved
Select high-entropy prefix tokens instead of random or bottom 20% (ablation)Prefix-RFT · Benchmark performancetraining reward and benchmarkImproved
Use cosine decay prefix-length scheduler instead of uniform scheduler (ablation)Prefix-RFT · Math benchmarksperformance: uniform drop of -0.2 to -5.3 pointsImproved
Use prefix only as guidance without learning from prefix loss (DR-GRPO style) (ablation)Prefix-RFT · Math reasoningperformanceHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy