Research map / Reinforcement learning / RL from feedback
Getting Your LLMs Ready for Reinforcement Learning with Lightweight SFT
Xinran Li, Guangda Huzhang, Siqi Shen, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, Jun Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AESL cold-start instead of CE cold-start | GRPO post-training · Qwen2.5-7B-Instruct, Math avg after RL | average: 40.19 -> 42.26 | Improved |
| use AESL cold-start instead of CE cold-start | GRPO post-training · Qwen2.5-Math-7B, Math avg after RL | average: 48.6 -> 50.04 | Improved |
| use AESL cold-start instead of CE cold-start | GRPO post-training · Llama-3.1-8B-Instruct, Math avg after RL | average: 22.46 -> 25.5 | Improved |
| early-stop CE cold-start at the diversity turning point | CE cold-start · Qwen2.5-7B-Instruct, Math avg after RL | average: 40.19 -> 41.54 | Improved |
| add cold-start SFT before RL (AESL) | GRPO · Qwen2.5-7B-Instruct, Math avg after RL | average: 38.09 -> 42.26 | Improved |
| use prefix-confidence denominator instead of constant denominator (ablation) | AESL · Qwen2.5-7B-Instruct, Math avg after RL | average: 40.35 -> 42.26 | Improved |
| use soft weighting instead of masking top-1 tokens (ablation) | AESL · Qwen2.5-7B-Instruct, Math avg after RL | average: 40.6 -> 42.26 | Improved |
| use AESL weighting instead of linear inverse weighting (ablation) | AESL · Qwen2.5-7B-Instruct, Math avg after RL | average: 40.96 -> 42.26 | Improved |
| increase t_scaling from 1 to 5 (ablation) | AESL · Qwen2.5-7B-Instruct, Math avg after RL | average: 41.09 -> 42.26 | Improved |
| increase t_scaling from 5 to 7 (ablation) | AESL · Qwen2.5-7B-Instruct, Math avg after RL | average: 42.26 -> 41.1 | Hurt |
| increase cold-start data from 1k to 6k (ablation) | AESL · Qwen2.5-7B-Instruct, Math avg after RL | average: 41.14 -> 42.79 | Improved |
| use AESL instead of CE-ES on simple questions | cold-start · Qwen2.5-7B-Instruct, simple 3k Math avg after RL | average: 39.35 -> 42.23 | Improved |
Closest papers by meaning
- Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead (ICLR 2026 · similarity 0.91)
- On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026 · similarity 0.90)
- Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.90)
- AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy (ICLR 2026 · similarity 0.90)
- ShiQ: Bringing back Bellman to LLMs (NeurIPS 2025 · similarity 0.90)
- RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
