Research map / Reinforcement learning / RL from feedback
Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical comparison of RLHF (reward learning plus KL-regularized policy optimization) and DPO under loglinear policies and linear rewards, deriving minimax suboptimality bounds, convergence rates, non-realizable-reward analysis, and an MDP extension of DPO.
Problem
Statistical comparison of RLHF and DPO for learning from human preferences: when one paradigm yields a smaller suboptimality gap than the other, as a function of sample size, reward and policy dimensions, regularization temperature, and reward realizability.
Approach
Analyzes RLHF and DPO under a shared setting (finite spaces, Bradley-Terry preferences, linear rewards, loglinear policies), showing DPO's loss is logistic regression in d_P+1 dimensions so it admits smoothness-based minimax bounds; compares RLHF's sqrt(d_R/n) rate against DPO's d_P/(beta n) rate and shows DPO can tune beta to absorb reward non-realizability while RLHF incurs a constant error.
Key results
- RLHF suboptimality scales as sqrt(d_R/n) while DPO scales as d_P/(beta n) under exact optimization with realizable rewards; DPO matches RLHF order when beta = Theta(sqrt(d_P/n)).
- Under approximate optimization, both paradigms admit exponentially decaying convergence terms, with RLHF's reward-learning gradient descent converging at (1-1/n)^t and policy phase at exp(-t)/beta, and DPO's gradient descent at (1/beta)(1-beta/n)^t.
- With non-realizable rewards, RLHF incurs a constant 2*epsilon_app error term while DPO's mismatch term can be controlled by tuning beta, giving asymptotically decaying gap when beta = O(1/sqrt(n)).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RLHF rather than DPO when the sample size is small relative to the dimensions, assuming comparable regularization-gap terms | RLHF vs. DPO · Contextual bandit, realizable rewards | suboptimality gap: RLHF: \widetilde{\Theta}(\sqrt{d_R/n}); DPO: \widetilde{\Theta}(d_P/(\beta n)) | Helped |
| Use DPO rather than RLHF at large sample sizes when d=d_R=d_P and the regularization-gap terms are similar | RLHF vs. DPO · Contextual bandit, realizable rewards | suboptimality gap: RLHF: \widetilde{\Theta}(\sqrt{d_R/n}); DPO: \widetilde{\Theta}(d_P/(\beta n)) | Helped |
| Use DPO with \beta=O(1/\sqrt{n}) to control the added error from reward non-realizability | DPO · Contextual bandit, non-realizable rewards | suboptimality gap: \min\{ 2\epsilon_\textnormal{app}, O\left( \beta D_\textnormal{KL}\left( \pi_{\theta^*}||\pi^*\right)\right)\} | Helped |
| Use a low-dimensional reward class with a higher-dimensional policy class | RLHF vs. DPO · Deterministic MDP, realizable rewards | suboptimality gap: RLHF: \widetilde{\Theta}(\Lambda'_R \sqrt{d_R/n}); DPO: \Theta(\Lambda'_M (d_M+1)/(\beta n) + \beta\Lambda_M\lambda (B') | Helped |
Surprising findings
- RLHF can outperform DPO when the policy dimension greatly exceeds the reward dimension, especially with small sample sizes (n < d).
Limitations
- Analysis restricted to finite spaces, linear rewards and loglinear policies; not general function approximation.
- No large-scale empirical validation of the theoretical insights.
- DPO extension to MDPs limited to deterministic MDPs with loglinear occupancy-measure regularization.
Experimental setup
Metrics: suboptimality gap
Baselines: RLHF DPO
Builds on: direct preference optimization reinforcement learning from human feedback Bradley-Terry model PL condition
Closest papers by meaning
- Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference (ICLR 2025 · similarity 0.94)
- Policy-labeled Preference Learning: Is Preference Enough for RLHF? (ICML 2025 · similarity 0.94)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.93)
- Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF (ICLR 2025 · similarity 0.92)
- Adaptive Preference Scaling for Reinforcement Learning with Human Feedback (NeurIPS 2024 · similarity 0.92)
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.