Research map / Reinforcement learning / RL from feedback
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formulates RLHF as a reverse-KL regularized contextual bandit and gives offline (pessimism), online/hybrid (batch exploration) theory, then practical iterative DPO and multi-step RSO algorithms evaluated on Open-LLaMA-3B-V2 with HH-RLHF and UltraRM gold reward.
Problem
Existing RLHF methods (PPO, DPO) lack rigorous theory under the reverse-KL regularized objective; offline methods need broad data coverage and suffer reward over-optimization.
Approach
Treat RLHF as a KL-regularized contextual bandit with Gibbs policy oracle; derive pessimistic offline learning and enhancer-based online exploration with finite-sample guarantees; implement via DPO with pessimistic margins and multi-step rejection sampling (RSO generalization).
Key results
- Hybrid-GSHF-DPO reaches gold reward 2.67 and gold win rate 0.67 in-domain, versus 2.15 for offline DPO and 2.25 for RSO.
- Multi-step RSO reduces reward gap between in-domain and OOD (Δ 0.18) compared with DPO (0.44) and RSO (0.36).
- Iterative online DPO improves reward-KL frontier across iterations, and online data is more efficient than more offline data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Hybrid-GSHF-DPO instead of DPO | Open-LLaMA-3B-V2 alignment · HH-RLHF test | Gold Reward: 2.15 -> 2.67 | Helped |
| Use Hybrid-GSHF-DPO instead of DPO | Open-LLaMA-3B-V2 alignment · HH-RLHF test | GPT4 Eval: 0.5 -> 0.65 | Helped |
| Use Hybrid-GSHF-DPO instead of DPO | Open-LLaMA-3B-V2 alignment · UltraFeedback OOD | OOD Gold Reward: 1.71 -> 2.46 | Helped |
| Use Multi-step RSO instead of RSO | Open-LLaMA-3B-V2 alignment · HH-RLHF test | Gold Reward: 2.25 -> 2.59 | Helped |
| Use Multi-step RSO instead of RSO | Open-LLaMA-3B-V2 alignment · UltraFeedback OOD | OOD Gold Reward: 1.89 -> 2.41 | Helped |
| Use Multi-step RSO instead of RSO | Open-LLaMA-3B-V2 alignment · UltraFeedback OOD | Difference Δ: 0.36 -> 0.18 | Helped |
| Use Hybrid-GSHF-DPO instead of Multi-step RSO | Open-LLaMA-3B-V2 alignment · HH-RLHF test | Gold Reward: 2.59 -> 2.67 | Helped |
| Use Hybrid-GSHF-DPO instead of Multi-step RSO | Open-LLaMA-3B-V2 alignment · UltraFeedback OOD | OOD GPT4 Eval: 0.60 -> 0.59 | No clear effect |
| Iterate Hybrid-GSHF-DPO with newly collected online data | Hybrid-GSHF-DPO · HH-RLHF and UltraFeedback | Gold reward-KL trade-off | Helped |
| Use pseudo-labeled rejection-sampling data with RSO instead of training directly with DPO | Open-LLaMA-3B-V2 alignment · HH-RLHF test | Gold Reward: 2.15 -> 2.25 | Helped |
Surprising findings
- Models selected by the gold reward at 2-3 DPO epochs outperform the validation-loss minimum, so the loss minimum does not give the best policy.
- Offline RSO (pseudo-labelled by learned reward) outperforms DPO even when DPO is selected with the gold reward.
Limitations
- Uncertainty estimation for neural reward models lacks theory; relies on heuristic ensembles.
- Theory assumes linear reward parameterization with features and bounded norms.
- Length increases across iterations may partly explain reward gains.
Experimental setup
Datasets: Anthropic HH-RLHF UltraFeedback
Benchmarks: HH-RLHF test split UltraFeedback OOD prompts
Metrics: gold reward gold win rate GPT-4 win/tie/lose KL divergence
Baselines: DPO RSO SFT RAFT
Builds on: DPO RSO PPO InfoNCA
Closest papers by meaning
- Avoiding exp(R) scaling in RLHF through Preference-based Exploration (NeurIPS 2025 · similarity 0.93)
- Provably Efficient Online RLHF with One-Pass Reward Modeling (NeurIPS 2025 · similarity 0.93)
- Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference (ICLR 2025 · similarity 0.93)
- Greedy Sampling Is Provably Efficient For RLHF (NeurIPS 2025 · similarity 0.93)
- Doubly Robust Alignment for Large Language Models (NeurIPS 2025 · similarity 0.92)
- Online Preference Alignment for Language Models via Count-based Exploration (ICLR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.