Research map / Reinforcement learning / RL from feedback
Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VPO, a unified online/offline RLHF method that adds a value-based optimism (online) or pessimism (offline) bonus to the reward MLE, implemented implicitly as a DPO-style loss with a KL term against a calibration distribution, with regret/sub-optimality guarantees under linear rewards and experiments on TL;DR, ARC-Challenge, and AlpacaEval/MT-Bench.
Problem
Standard RLHF and DPO-style methods ignore reward uncertainty, so they lack principled optimism (online exploration) or pessimism (offline), and explicit confidence intervals are intractable for LLM policies.
Approach
Regularize the reward MLE with the optimal KL-regularized value J*(r), signed to favor high-value rewards online and penalize them offline; through the closed-form policy-reward relation this becomes a DPO-like loss with an added KL(calibration distribution || policy) term, avoiding explicit uncertainty estimation.
Key results
- On AlpacaEval 2.0 with Zephyr-7B-SFT base, VPO Iter 1 reaches 22.53 length-controlled win rate versus 15.41 for Zephyr-7B-DPO and 8.01 for the SFT base.
- On ARC-Challenge offline, VPO is reported to significantly outperform DPO and IPO across Llama2-7b-chat, Llama2-13b-chat and Flan-T5-xl, and avoids the performance drop DPO shows after 1000 iterations.
- Synthetic bandits: VPO with a tuned alpha beats the MLE-MAB baseline on long-term cumulative regret in the online setting and achieves lower sub-optimality gap offline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use iterative VPO training instead of the Zephyr-7B-SFT baseline | Zephyr-7B-SFT · AlpacaEval 2.0 | LC Win Rate: +14.52 | Helped |
| use iterative VPO training instead of the Zephyr-7B-SFT baseline | Zephyr-7B-SFT · MT-Bench | Avg: +2.32 | Helped |
| use VPO instead of DPO at iteration 1 | Zephyr-7B-SFT · AlpacaEval 2.0 | LC Win Rate: 20.53 -> 22.53 | Helped |
| use VPO instead of DPO at iteration 2 | Zephyr-7B-SFT · MT-Bench | Avg: 7.55 -> 7.62 | Helped |
| use VPO instead of DPO at iteration 3 | Zephyr-7B-SFT · AlpacaEval 2.0 | LC Win Rate: 22.19 -> 22.15 | No clear effect |
| use VPO instead of DPO at iteration 3 | Zephyr-7B-SFT · MT-Bench | Avg: 7.46 -> 7.61 | Helped |
| increase the optimistic VPO exploration weight from 0.01 to 0.1 (ablation) | PaLM2-XXS on TL;DR (Buffer) · TL;DR | win rate against SFT | Mixed |
| add the optimistic value-incentivized term | PaLM2-XXS on TL;DR (Buffer) · TL;DR | head-to-head win/tie/loss rate against Online DPO | Helped |
Surprising findings
- Optimistic VPO shows larger win-rate oscillation against SFT than Online DPO early in training, which shrinks as the exploration weight alpha decreases.
Limitations
- Theory relies on linear reward parameterization and a bounded-reward/finite-class argument; LLM experiments do not match the theoretical setting.
- Choice of regularization strength alpha and calibration distribution must be tuned, with no adaptive rule given.
- Optimistic online variant shows greater training oscillation.
Experimental setup
Datasets: ARC-Challenge TL;DR UltraFeedback
Benchmarks: ARC-Challenge TL;DR AlpacaEval 2.0 MT-Bench
Metrics: accuracy win rate win/tie/loss rate length-controlled win rate cumulative regret sub-optimality gap MT-Bench score
Baselines: DPO IPO Online DPO MLE-MAB SPIN DNO SPPO
Builds on: DPO reward-biased maximum likelihood IPO Online DPO
Closest papers by meaning
- Greedy Sampling Is Provably Efficient For RLHF (NeurIPS 2025 · similarity 0.94)
- Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference (ICLR 2025 · similarity 0.93)
- Provably Efficient Online RLHF with One-Pass Reward Modeling (NeurIPS 2025 · similarity 0.92)
- Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.92)
- Avoiding exp(R) scaling in RLHF through Preference-based Exploration (NeurIPS 2025 · similarity 0.92)
- Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF (ICLR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.