Research map / Reinforcement learning / RL from feedback

Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF

ICLR 2025 · Reinforcement learning › RL from feedback · approach: Reward modeling versus direct optimization · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes XPO, a one-line modification of online DPO that adds an optimism bonus (log-likelihood of rejected-side samples from the reference policy) to give provable sample-efficient online exploration for RLHF with general function approximation.

Problem

Online exploration in RLHF is sample-inefficient: passive exploration (online/iterative DPO) cannot discover behaviors poorly covered by the reference policy, and prior provably efficient exploration methods are computationally intractable or limited to bandits/linear settings.

Approach

Interpret DPO as implicit Q*-approximation (Bellman error minimization) in a KL-regularized MDP, and add a global optimism term alpha * sum log pi(tau_tilde) to the DPO objective, where tau_tilde are reference-policy samples; this yields a differentiable, practical exploration bonus that gives coverability-based sample complexity guarantees for deterministic contextual MDPs.

Key results

• XPO-iter3 reaches 29.35 AlpacaEval-2 LC and 27.2 Arena-Hard, versus 27.33 and 24.9 for iterative DPO-iter3 and 31.7 / 23.1 for Llama-3-8B-Flow-Final (which uses 4x more responses).
• XPO-iter3 improves GSM8K over the non-exploratory baseline? No: XPO-iter3 scores 75.97 on GSM8K while DPO-iter3 scores 72.93, and XPO avoids the math regression seen in iterative DPO.
• Theory: XPO achieves suboptimality O~(sqrt(Ccov(Pi) log|Pi| / T)) for deterministic contextual MDPs, avoiding exp(1/beta) dependence that Online DPO suffers.

Measured effects reported in the paper

Surprising findings

• Sampling both responses from the current policy (as in standard online DPO) can lead to degenerate under-exploration in the small-beta regime even when the reference policy has good coverage.
• Online DPO can require exponentially many samples in 1/beta even in a two-action bandit, showing passive exploration is provably insufficient.

Limitations

• Theory restricted to deterministic contextual MDPs with stochastic start states; DPO objective unsuitable for stochastic dynamics.
• Empirical evaluation is preliminary: single-seed runs with reported 3-4% win-rate variance, and the authors defer comprehensive evaluation.
• Optimism objective is non-convex, especially as beta approaches 0, making optimization harder.

Experimental setup

Datasets: iterative-prompt-v1-iter1-20K iterative-prompt-v1-iter2-20K iterative-prompt-v1-iter3-20K

Benchmarks: AGIEval ANLI MMLU GPQA GSM8K AlpacaEval-2 LC Arena-Hard

Metrics: accuracy length-controlled win rate win rate

Baselines: Iterative DPO (alpha=0) Llama-3-8B-Flow-Final Llama-3-8B-it

Builds on: DPO Iterative DPO Online DPO KL-regularized MDP coverability

Closest papers by meaning

• Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF (ICLR 2025 · similarity 0.92)
• Avoiding exp(R) scaling in RLHF through Preference-based Exploration (NeurIPS 2025 · similarity 0.92)
• Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference (ICLR 2025 · similarity 0.92)
• Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.92)
• Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.91)
• Policy-labeled Preference Learning: Is Preference Enough for RLHF? (ICML 2025 · similarity 0.91)