Research map / Reinforcement learning / RL from feedback
Online Iterative Reinforcement Learning from Human Feedback with General Preference Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
placeholder
Problem
RLHF usually relies on a reward function and the Bradley-Terry model, which cannot capture intransitive or population-level human preferences; offline learning also suffers from poor coverage with LLM policies.
Approach
Replace reward-based RLHF with a KL-regularized minimax game between two LLMs under a general preference oracle, find the Nash equilibrium (von Neumann winner), and use pessimism offline and an enhancer policy (maximizing uncertainty relative to the main agent) online, with a practical self-play IPO and rejection-sampling implementation.
Key results
- Online-ELHF-IPO reaches 17.67 AlpacaEval2 win rate vs 9.36 for offline IPO and 9.33 for offline DPO.
- Online-ELHF-IPO has 0.78 gold win rate against offline IPO on the UltraFeedback hold-out set.
- Preference model with LLaMA3-8B achieves higher RewardBench reasoning accuracy than the BT reward model (94.8 vs 87.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace offline IPO with iterative online ELHF using self-play IPO and rejection-sampling exploration | IPO alignment from Zephyr-SFT-7B · Ultra-feedback test set (3K prompts) | Gold WR v.s. IPO: 0.5 -> 0.78 | Helped |
| Replace offline IPO with iterative online ELHF using self-play IPO and rejection-sampling exploration | IPO alignment from Zephyr-SFT-7B · AlpacaEval2 | WR: 9.36 -> 17.67 | Helped |
| Set the IPO KL coefficient to η=0.1 instead of η=0.5 | Offline IPO · Ultra-feedback test set (3K prompts) | win rate vs. SFT: 0.795 -> 0.879 | Helped |
| Set the IPO KL coefficient to η=0.1 instead of η=1.0 | Offline IPO · Ultra-feedback test set (3K prompts) | win rate vs. SFT: 0.710 -> 0.879 | Helped |
| Set the IPO KL coefficient to η=0.1 instead of η=0.5 | Offline IPO · AlpacaEval2 | score: 6.86 -> 9.36 | Helped |
| Set the IPO KL coefficient to η=0.1 instead of η=1.0 | Offline IPO · AlpacaEval2 | score: 6.55 -> 9.36 | Helped |
Surprising findings
- Preference-model-based training with Gemma-2B-it substantially improves reasoning accuracy (80.7 vs 74.2) while other categories stay comparable.
- Estimated KL between Gemma-7B-it and Gemma-2B-it policies is about 456, suggesting offline coverage coefficients explode in practice.
Limitations
- Experiments use a LLaMA3-8B preference model as a simulated oracle rather than real human labelers.
- The enhancer's exploration via rejection sampling has no theoretical guarantee.
- Only three iterations were run and the evaluation uses a single model family (Zephyr-7B).
Experimental setup
Datasets: UltraFeedback UltraChat HH-RLHF SHP HelpSteer UltraInteract distilabel-capybara distilabel-orca
Benchmarks: AlpacaEval2 RewardBench UltraFeedback hold-out test set
Metrics: win rate gold win rate test accuracy
Baselines: Offline DPO Offline IPO SFT
Builds on: IPO DPO Nash learning from human feedback Bradley-Terry model
Closest papers by meaning
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.92)
- Policy-labeled Preference Learning: Is Preference Enough for RLHF? (ICML 2025 · similarity 0.91)
- Greedy Sampling Is Provably Efficient For RLHF (NeurIPS 2025 · similarity 0.91)
- A Minimaximalist Approach to Reinforcement Learning from Human Feedback (ICML 2024 · similarity 0.91)
- Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference (ICLR 2025 · similarity 0.91)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.