Research map / Reinforcement learning / RL from feedback
Design Considerations in Offline Preference-based RL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives a theoretical framework for offline preference-based RL (DPO, IPO, SLiC and variants) showing how loss curvature, base policy choice and coverage bound the KL to a benchmark policy, and validates on TL;DR summarization with a T5-large policy.
Problem
Offline RLHF methods such as DPO, IPO and SLiC are used widely, but lack a unified theoretical account of how their design choices (loss, base policy, constraints) affect the quality of the learned policy and why they degenerate.
Approach
Treat offline RLHF as loss minimization over log-likelihood ratios with a general base policy mu, loss ell and policy class Pi; derive a KL suboptimality bound against a benchmark policy pi* using loss curvature and a coverage condition, without relying on reparameterization arguments.
Key results
- Squared loss variants (IPO-style) reach peak preference and maintain it stably, while logistic loss (DPO) variants peak around 2k steps and then collapse.
- Using the SFT base policy with squared loss is consistently better than the uniform base policy; for logistic loss the base choice has little effect against the SFT policy.
- Empirical curvature constant c_mu is about 0.00066 for DPO versus 2.0 for squared loss, with reward range R about 73 for DPO versus 4.7 for squared loss.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use squared loss instead of logistic loss | Offline preference learning on T5 large · TL;DR | summary preference quality | Helped |
| Use logistic loss instead of squared loss | Offline preference learning on T5 large · TL;DR | training stability | Hurt |
| Use SFT reference policy instead of uniform base policy | Squared-loss preference learning · TL;DR | summary preference quality | Helped |
| Use SFT reference policy instead of uniform base policy | Logistic-loss preference learning · TL;DR | summary preference quality | No clear effect |
| Normalize log-likelihoods with the reference policy | Offline preference learning · TL;DR | learned policy quality | Helped |
| Vary β among 0.1, 0.5, and 1.0 (ablation) | Squared-loss preference learning · TL;DR | summary preference quality | No clear effect |
Surprising findings
- The DPO-style logistic loss log-likelihoods collapse toward zero, making the theoretical bound drastically worse, offering a theoretical explanation for DPO degeneracy.
- Uniform base policy (mu=1) and SFT base policy give comparable logistic-loss results against the SFT reference, despite theory favoring the SFT base.
Limitations
- Small empirical study on a single summarization benchmark with a 770M T5 model.
- Theoretical bound depends on realizability and coverage assumptions that may fail in practice; length normalization of SimPO and ODPO margins are not covered.
- Evaluation relies on a single prompted Gemini judge.
Experimental setup
Datasets: TL;DR summarization preference dataset
Benchmarks: TL;DR summarization
Metrics: preference win rate versus SFT policy judged by Gemini 1.0 U log-probabilities of preferred and dispreferred responses approximate reward range and curvature constant
Baselines: DPO IPO SLiC Gemini 1.0 Ultra evaluator comparison to SFT policy
Builds on: DPO IPO SLiC GPO CPO
Closest papers by meaning
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.92)
- The Importance of Online Data: Understanding Preference Fine-tuning via Coverage (NeurIPS 2024 · similarity 0.91)
- Preference Elicitation for Offline Reinforcement Learning (ICLR 2025 · similarity 0.91)
- Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF (ICLR 2025 · similarity 0.91)
- Policy-labeled Preference Learning: Is Preference Enough for RLHF? (ICML 2025 · similarity 0.91)
- Greedy Sampling Is Provably Efficient For RLHF (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.