Research map / Reinforcement learning / RL from feedback
Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes reward-learning SFT (RFT and IRFT) that treats demonstration-only alignment as bilevel inverse RL, showing gains over SFT and SPIN on Anthropic-HH and Open LLM Leaderboard with pythia and zephyr models.
Problem
Standard SFT imitates demonstration data via supervised likelihood and learns no reward, unlike RLHF/DPO which learn reward from preference pairs; the paper asks whether learning a reward from demonstration data alone improves LLM alignment.
Approach
Formulates SFT as maximum-likelihood inverse RL (bilevel), reduces it to a minimax problem, and derives a self-generation gradient so the reward is implicit via the policy (IRFT), which reduces to SPIN as a special case; an explicit reward variant (RFT) trains a reward and policy jointly.
Key results
- On zephyr-7b-sft-full, IRFT with T=10 raises Open LLM Leaderboard average from 59.48 (base) to 61.03, versus 60.32 for SPIN.
- On pythia-1.4b, IRFT with T=10 reaches average 37.69 versus 37.27 for SFT and 37.35 for SPIN.
- Example 2 shows IRL-trained models assign higher log-probability gap to preferred over non-preferred continuations, while SFT does not, despite training only on chosen data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IRFT T=10 instead of the strongest listed baseline | pythia-1.4b · Open LLM Leaderboard Average | average score: 37.35 -> 37.69 | Helped |
| Use IRFT T=10 instead of the strongest listed baseline | zephyr-7b-sft-full · Open LLM Leaderboard Average | average score: 61.02 -> 61.03 | No clear effect |
| Use IRFT T=10 rather than IRFT (SPIN iter 2), T=2 (ablation) | zephyr-7b-sft-full · GSM8K | exact_match: 35.56 -> 34.34 | Hurt |
| Use IRFT T=10 rather than IRFT (SPIN iter 2), T=2 (ablation) | zephyr-7b-sft-full · TruthfulQA | acc: 36.56 -> 39.99 | Helped |
| Use IRFT T=10 rather than IRFT (SPIN iter 2), T=2 (ablation) | zephyr-7b-sft-full · AI2_Arc | acc_norm: 76.13 -> 76.78 | Helped |
| Use IRFT T=10 rather than IRFT (SPIN iter 2), T=2 (ablation) | zephyr-7b-sft-full · HellaSwag | acc_norm: 83.39 -> 83.05 | Hurt |
| Use IRFT with more frequent generation (T>1) rather than SPIN-equivalent T=1 (ablation) | pythia-1.4b · Open LLM Leaderboard | average score: 37.35; 37.69 | Mixed |
| Use IRFT T=5 rather than T=10 (ablation) | zephyr-7b-sft-full · TruthfulQA | acc: 36.84 -> 39.99 | Helped |
| Use IRFT T=10 rather than T=5 (ablation) | zephyr-7b-sft-full · Open LLM Leaderboard Average | average score: 60.71 -> 61.03 | Helped |
| Use IRFT T=5 rather than T=10 (ablation) | pythia-1.4b · HellaSwag | acc_norm: 54.79; 54.77 | No clear effect |
| Replace reward-free SFT with explicit reward-learning RFT | pythia-1.4b · Anthropic-HH | Beaver reward and win rate | Helped |
| Train further with SFT rather than the pretrained model alone | pythia-1.4b · Open LLM Leaderboard Average | average score: 37.27 -> 37.27 | No clear effect |
Surprising findings
- Pythia-1.4b SFT gains essentially nothing on the Open LLM Leaderboard average (37.27 both before and after), and differences between methods are small (under 0.5 points for pythia).
Limitations
- Convergence theory only shows convergence to a stationary point and does not characterize what the policy converges to.
- Extra compute for training two models or generating synthetic data is not negligible.
- Experiments limited to 1B-7B models and specific datasets.
Experimental setup
Datasets: Anthropic-HH UltraChat200k
Benchmarks: Anthropic-HH HuggingFace Open LLM Leaderboard
Metrics: average reward by beaver-7b-v3.0-reward win rate accuracy (AI2 Arc acc_norm) TruthfulQA acc Winogrande acc GSM8K exact match HellaSwag acc_norm MMLU acc average score
Baselines: SFT SPIN pythia-1.4b base model zephyr-7b-sft-full
Builds on: SPIN ML-IRL DPO RLHF
Closest papers by meaning
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.89)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.89)
- Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model Alignment (AAAI 2025 · similarity 0.89)
- Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment (ICML 2025 · similarity 0.89)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.89)
- Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.