Research map / Large language models / Instruction and fine-tuning
Clipping Low-Probability Tokens in SFT Yields a Generalizable Initialization for RL
Tian-Shuo Liu, Chengxing Jia, Haoyu Liu, Pengyuan Wang, Shiyuan Zhang, Jie Fu, Yang Yu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| clip low-probability tokens in SFT (OPC-SFT) instead of plain SFT | SFT cold-start · ScienceWorld+ALFWorld, after GRPO | average success rate (Qwen2.5-7B): 69.97 -> 77.70 | Improved |
| clip low-probability tokens in SFT | SFT cold-start · ScienceWorld+ALFWorld, after GRPO | average success rate (Llama3.2-3B): 79.02 -> 81.49 | Improved |
| clip low-probability tokens in SFT | SFT cold-start · OOD benchmarks after ALFWorld SFT | average performance drop: 11.54% smaller drop | Improved |
| clip low-probability tokens in SFT | SFT cold-start · ALFWorld/ScienceWorld in-distribution before RL | average success rate: 64.17 -> 68.61 | Improved |
| add NEFTune noise to SFT | SFT cold-start · ScienceWorld+ALFWorld, after GRPO | average success rate (Qwen2.5-7B): 69.97 -> 63.85 | Hurt |
| use DFT instead of SFT | SFT cold-start · ScienceWorld+ALFWorld, after GRPO | average success rate: Qwen 7B 69.97 -> 72.84; Llama 79.02 -> 72.57 | Mixed |
| apply GRPO to base model without SFT cold-start | Qwen2.5-7B-Instruct · ScienceWorld+ALFWorld | average success rate: 77.70 -> 52.86 | Hurt |
| clip low-probability tokens in SFT | SFT on math (DeepScaleR) data · math reasoning | accuracy | No clear effect |
| remove clipping but keep ratio-based reweighting (ablation) | OPC-SFT · ALFWorld OOD | generalization | Hurt |
| increase old-policy update interval to 1024+ samples (ablation) | OPC-SFT · Llama3.2-3B ALFWorld-unseen | success rate: 77.6 -> 70.9 | Hurt |
| update old policy too frequently (interval 64) (ablation) | OPC-SFT · Llama3.2-3B ALFWorld-unseen | success rate: 77.6 -> 75.4 | Hurt |
| vary clipping ratio eps in moderate range (ablation) | OPC-SFT · agentic and OOD benchmarks | success rate | No clear effect |
Closest papers by meaning
- On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification (ICLR 2026 · similarity 0.91)
- Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs (ICLR 2026 · similarity 0.90)
- Reinforcement Learning Finetunes Small Subnetworks in Large Language Models (NeurIPS 2025 · similarity 0.90)
- On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026 · similarity 0.90)
- Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning (ICLR 2025 · similarity 0.89)
- SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
