Research map / Reinforcement learning / RL from feedback
The Importance of Online Data: Understanding Preference Fine-tuning via Coverage
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically separates offline contrastive preference methods (DPO/IPO, needing global coverage) from online RLHF (needing only partial KL-ball coverage), and proposes HyPO, which adds online unlabeled samples for a reverse-KL regularizer to DPO.
Problem
Understanding the statistical separation between online RLHF (PPO with a learned reward model) and offline contrastive preference optimization (DPO, IPO) for LLM fine-tuning, and why online methods often outperform offline ones when preference data lacks diversity.
Approach
Uses dataset coverage to show global coverage is necessary and sufficient for DPO/IPO, while a weaker local KL-ball coverage suffices for online RLHF because reverse KL is controlled via online samples; this motivates HyPO, which applies the DPO loss to offline data and a reverse-KL penalty estimated on unlabeled online samples.
Key results
- On TL;DR with Pythia-1.4B, HyPO reaches 46.44% GPT-4 win rate versus 42.17% for DPO, and reverse KL drops from 44.90 to 27.07.
- On AlpacaEval 2.0 with Llama-3-8B-Instruct, HyPO gets 30.7 LC win rate versus 28.4 for DPO and 26.0 for the base model.
- HyPO keeps validation reverse KL lower than DPO when both are trained for 5 epochs on TL;DR, mitigating overfitting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add online reverse-KL regularization to DPO | DPO on TL;DR with Pythia 1.4B · TL;DR | GPT4 winrate: 42.17% (2.5%) -> 46.44% (2.39%) | Helped |
| Add online reverse-KL regularization to DPO | DPO on TL;DR with Pythia 1.4B · TL;DR | RM score: 0.16 (0.05) -> 0.37 (0.05) | Helped |
| Add online reverse-KL regularization to DPO | DPO on TL;DR with Pythia 1.4B · TL;DR | reverse KL: 44.90 (1.29) -> 27.07 (2.34) | Helped |
| Add online reverse-KL regularization to DPO | DPO on TL;DR with Pythia 2.8B · TL;DR | GPT4 winrate: 44.39% (0.4%) -> 50.50% (1.89%) | Helped |
| Add online reverse-KL regularization to DPO | DPO on TL;DR with Pythia 2.8B · TL;DR | RM score: 2.43 (0.10) -> 2.51 (0.13) | Helped |
| Add online reverse-KL regularization to DPO | DPO on TL;DR with Pythia 2.8B · TL;DR | reverse KL: 68.95 (3.08) -> 48.98 (4.23) | Helped |
| Add online reverse-KL regularization to DPO | DPO-Llama-3 · MT-Bench | Average: 7.75 -> 8.09 | Helped |
| Add online reverse-KL regularization to DPO | DPO-Llama-3 · AlpacaEval 2.0 | LC Win Rate: 28.4 -> 30.7 | Helped |
| Add online reverse-KL regularization to DPO | DPO-Llama-3 · AlpacaEval 2.0 | Win Rate: 30.9 -> 32.2 | Helped |
| Add online reverse-KL regularization to DPO | DPO-Llama-3 · Open LLM leaderboard | Average: 65.34 -> 65.51 | Helped |
| Add online reverse-KL regularization to DPO | DPO-Llama-3 · MMLU (5-shot) | score: 65.82 -> 65.74 | Hurt |
| Add online reverse-KL regularization to DPO | DPO-Llama-3 · Arc (25-shot) | score: 63.14 -> 62.71 | Hurt |
Surprising findings
- DPO can assign increasing probability to unseen suboptimal out-of-distribution responses when function approximation is absent (multi-armed bandit setting).
- HyPO still underperforms PPO-based online RLHF in win rate on TL;DR, suggesting the theory does not fully explain the benefit of online RLHF.
Limitations
- Experiments cover only a few models and benchmarks; HyPO remains below online RLHF in win rate.
- Conjectured rather than proven that KL-ball coverage is necessary for online RL and that single-policy coverage is insufficient.
Experimental setup
Datasets: TL;DR summarization preference dataset UltraFeedback (HuggingFaceH4/ultrafeedback_binarized)
Benchmarks: TL;DR summarization AlpacaEval 2.0 MT-Bench MMLU GSM8K ARC TruthfulQA HellaSwag
Metrics: GPT-4 win rate reward model score reverse KL to reference length-controlled win rate MT-Bench score accuracy
Baselines: DPO IPO Meta-Llama-3-8B-Instruct
Builds on: DPO PPO IPO RLOO
Closest papers by meaning
- Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data (ICML 2024 · similarity 0.92)
- Design Considerations in Offline Preference-based RL (ICML 2025 · similarity 0.91)
- Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF (ICLR 2025 · similarity 0.91)
- Explicit Preference Optimization: No Need for an Implicit Reward Model (ICML 2025 · similarity 0.91)
- Online Preference Alignment for Language Models via Count-based Exploration (ICLR 2025 · similarity 0.90)
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.