Research map / Reinforcement learning / RL from feedback
Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Transfer Policy Optimization (TPO), which transfers from imperfect source reward models in online RLHF by exploiting KL-regularization-induced policy coverage, plus an empirical win-rate-based variant that integrates with DPO/IPO/XPO and is tested on XSum summarization with T5.
Problem
Online RLHF is sample-inefficient in terms of human preference labels, and existing work does not exploit available but imperfect (misspecified) reward models to accelerate learning.
Approach
KL regularization makes a policy's coverability of the optimal policy controlled by its value gap, so high-value policies (including ones distilled from online data, "self-transfer") can be used for transfer without conflicting with exploration; an empirical variant selects transfer policies by win rate against the learning policy via a UCB bandit.
Key results
- Empirical TPO win rate vs. without-transfer iterative DPO is 52.1% at iter 1 rising to 54.0% at iter 3 (mean of 3 seeds).
- Against purely exploiting T5-Large, empirical TPO achieves 50.6% win rate at iter 3, above 49.5% and 49.1% in earlier iterations.
- Against purely exploiting the ROUGE-Lsum reward, empirical TPO's win rate is 53.1% (iter 1) and 54.5% (iter 2), but drops to 53.3% at iter 3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add win-rate-based transfer policy selection instead of removing transfer | Iterative DPO on T5-small · XSum, Iter 1 vs without transfer | win rate (%): 52.1±1.2 | Helped |
| Add win-rate-based transfer policy selection instead of removing transfer | Iterative DPO on T5-small · XSum, Iter 2 vs without transfer | win rate (%): 53.3±1.6 | Helped |
| Add win-rate-based transfer policy selection instead of removing transfer | Iterative DPO on T5-small · XSum, Iter 3 vs without transfer | win rate (%): 54.0±1.2 | Helped |
| Select transfer policies by win rate instead of purely exploiting ROUGE-Lsum | Iterative DPO on T5-small · XSum, Iter 1 vs purely exploit ROUGE-Lsum | win rate (%): 53.1±1.1 | Helped |
| Select transfer policies by win rate instead of purely exploiting ROUGE-Lsum | Iterative DPO on T5-small · XSum, Iter 2 vs purely exploit ROUGE-Lsum | win rate (%): 54.5±1.3 | Helped |
| Select transfer policies by win rate instead of purely exploiting ROUGE-Lsum | Iterative DPO on T5-small · XSum, Iter 3 vs purely exploit ROUGE-Lsum | win rate (%): 53.3±1.5 | Helped |
| Select transfer policies by win rate instead of purely exploiting T5-Large | Iterative DPO on T5-small · XSum, Iter 1 vs purely exploit T5-Large | win rate (%): 49.5±0.9 | Hurt |
| Select transfer policies by win rate instead of purely exploiting T5-Large | Iterative DPO on T5-small · XSum, Iter 2 vs purely exploit T5-Large | win rate (%): 49.1±0.4 | Hurt |
| Select transfer policies by win rate instead of purely exploiting T5-Large | Iterative DPO on T5-small · XSum, Iter 3 vs purely exploit T5-Large | win rate (%): 50.6±0.3 | Mixed |
Surprising findings
- Exploiting the worst source reward (ROUGE-Lsum) gives higher win rates than exploiting the best one (T5-Large) in iterations 1 and 2, suggesting the best source is not always the best transfer target.
Limitations
- Experiments limited to T5-small policy on summarization with only K=3 iterations.
- Human reward is simulated by a distilled reward model rather than real human labels.
- Theoretical TPO requires solving minimax optimization, which is computationally heavy.
Experimental setup
Datasets: XSum
Benchmarks: XSum
Metrics: win rate
Baselines: iterative DPO without transfer purely exploit ROUGE-Lsum reward purely exploit T5-Large reward
Builds on: XPO RPO DPO IPO iterative DPO
Closest papers by meaning
- Provably Efficient Online RLHF with One-Pass Reward Modeling (NeurIPS 2025 · similarity 0.92)
- Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization (ICML 2024 · similarity 0.92)
- Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference (ICLR 2025 · similarity 0.91)
- Greedy Sampling Is Provably Efficient For RLHF (NeurIPS 2025 · similarity 0.90)
- Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.90)
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.