Research map / Reinforcement learning / RL from feedback
Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematic study of preference fine-tuning methods (PPO, REINFORCE, RWR, best-of-N, DPO/IPO, Pref-FT) showing on-policy sampling and negative gradients help, especially when reward peaks lie in low-likelihood regions of the reference policy, and that on-policy contrastive training is complementary.
Problem
Practitioners are unsure whether on-policy RL is required versus contrastive or supervised fine-tuning for preference fine-tuning of LLMs, and whether preference data should be collected on-policy.
Approach
Characterize preference fine-tuning methods along on-policy sampling, sample reuse, and negative gradient axes, and evaluate them in didactic bandits, synthetic LLM tasks, and full-scale AlpacaFarm/UltraFeedback; explain findings via mode-seeking (reverse KL) versus mode-covering (forward KL) objectives.
Key results
- On-policy sampling improves gold reward when the reward peak lies in low-likelihood regions of the reference policy (e.g., Min Length scenario, smaller batch sizes yield lower completion length).
- DPO with negative gradient significantly outperforms Pref-FT on full-scale UltraFeedback with Mistral-7B, which shows much larger improvement over the reference policy.
- On-policy DPO/IPO converges faster in wall-clock time than offline DPO and PPO/RWR in the synthetic Skew Length setting (0.4 hours vs 0.12 hours flatlining at sub-optimal length for offline DPO).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use more frequent, smaller-batch on-policy sampling | Best-of-N · Bandit R1 and R2 | Reward / convergence | Helped |
| Use more off-policy updates instead of frequent on-policy sampling | PPO · Mode Length | Gold reward | No clear effect |
| Use more on-policy sampling (smaller B) | PPO and on-policy RWR / REINFORCE · Min Length and Skew Length | Gold reward | Helped |
| Increase sample reuse from T=1 to T=8 (ablation) | PPO · Synthetic LLM problems | Gold reward: T=8; T=1 | No clear effect |
| Increase sample reuse from T=1 to T=2 (ablation) | On-policy Best-of-N · Synthetic LLM problems | Gold reward: T=1 to 2 | Helped |
| Add a negative-gradient loss on dispreferred responses (ablation) | Best-of-N · Bandit R1 and R2 | Reward | Helped |
| Use DPO with a negative gradient instead of Pref-FT | Offline preference fine-tuning · Min Length and Skew Length | Reward | Helped |
| Use DPO instead of Pref-FT | Offline preference fine-tuning · Mode Length | Reward | No clear effect |
| Use on-policy DPO / IPO instead of offline DPO / IPO | DPO / IPO · Bandit R1 and R2; synthetic LLM problems | Reward / convergence | Helped |
| Use on-policy DPO / IPO instead of offline DPO / IPO | DPO / IPO · Bandit (R1) | Reward: 0.82 (0.04) -> 0.92 (0.01) | Helped |
| Use on-policy DPO / IPO instead of offline DPO / IPO | DPO / IPO · Min Length | Time: 1.3 hours -> 0.4 hours | Helped |
| Use on-policy DPO / IPO instead of offline DPO / IPO | DPO / IPO · Skew Length | Completion Length: 11.8 (14.0) -> 0.0 (0.0) | Helped |
Surprising findings
- Offline DPO can decrease the likelihood of preferred responses during training, with recovered probability mass going to other out-of-distribution responses.
- Pref-FT increases likelihood of dispreferred responses despite training only on preferred responses.
Limitations
- No rigorous statistical guarantees for the negative gradient effect.
- Does not study the effect of reward model quality.
- Does not account for pre-training distribution coverage.
Experimental setup
Datasets: AlpacaFarm UltraFeedback UltraChat
Benchmarks: AlpacaFarm UltraFeedback
Metrics: gold reward proxy reward completion length wall-clock time
Baselines: PPO REINFORCE RWR Best-of-N Pref-FT DPO IPO Binary FeedME
Builds on: DPO IPO PPO REINFORCE ReST
Closest papers by meaning
- The Importance of Online Data: Understanding Preference Fine-tuning via Coverage (NeurIPS 2024 · similarity 0.92)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.90)
- Explicit Preference Optimization: No Need for an Implicit Reward Model (ICML 2025 · similarity 0.89)
- MallowsPO: Fine-Tune Your LLM with Preference Dispersions (ICLR 2025 · similarity 0.89)
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.89)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.