Research map / Reinforcement learning / RL from feedback
Efficient Preference-Based Reinforcement Learning: Randomized Exploration meets Experimental Design
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes randomized preference optimization (RPO) meta-algorithms that reduce RLHF to an RL oracle via Thompson-style reward sampling, with regret and last-iterate guarantees, plus a lazy batched variant using greedy D-optimal design to cut preference queries.
Problem
Active selection of trajectory-level preference queries in general MDPs with Bradley-Terry feedback, aiming for computationally tractable algorithms with regret/PAC guarantees while reducing human annotation workload.
Approach
Replace intractable optimistic exploration bonuses over state distributions with randomized exploration: sample reward parameters from an inflated MLE confidence ellipsoid and call a PAC-RL oracle; add lazy batched updates and greedy D-optimal design to select informative preference pairs in parallel batches.
Key results
- RPO-Regret attains considerably lower regret with less variance than an entropy-exploration RLHF baseline in a 6x6 gridworld.
- On Isaac-Cartpole, RPO-Regret, LRPO-Regret and LRPO-OD-Regret all show flattening regret slopes and performance competitive with PPO under ground-truth reward.
- LRPO-OD-Regret reaches comparable performance to RPO-Regret while considerably reducing the number of preference queries.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RPO-Regret instead of entropy-based exploration | 6x6 gridworld · 6x6 gridworld | cumulative regret | Helped |
| add optimal design to lazy updates (ablation) | LRPO-Regret · Isaac-Cartpole-v0 | cumulative regret | Mixed |
| add optimal design to lazy updates (ablation) | LRPO-Regret · Isaac-Cartpole-v0 | preference queries | Helped |
| use randomized preference optimization methods | Isaac-Cartpole-v0 · Isaac-Cartpole-v0 | cumulative regret | No clear effect |
Surprising findings
- Entropy-based exploration does not necessarily guarantee low regret in MDPs, per the gridworld experiments.
Limitations
- Bradley-Terry preference model may not capture real human feedback.
- Relies on an RL oracle at every RLHF round, which may be computationally demanding.
- Experiments limited to tabular gridworld and simple control with synthetic feedback.
Experimental setup
Datasets: Isaac-Cartpole-v0 synthetic preferences 6x6 gridworld synthetic preferences
Benchmarks: Isaac-Cartpole-v0 (Nvidia Isaac Lab) 6x6 gridworld
Metrics: cumulative regret number of preference queries
Baselines: entropy-regularized exploration RLHF reward-based RL with ground truth reward RPO-Regret LRPO-Regret
Builds on: linear Thompson sampling dueling bandits lazy updates for linear bandits PPO soft value iteration
Closest papers by meaning
- Towards Theoretical Understanding of Sequential Decision Making with Preference Feedback (ICML 2025 · similarity 0.91)
- A Minimaximalist Approach to Reinforcement Learning from Human Feedback (ICML 2024 · similarity 0.91)
- Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries (ICML 2025 · similarity 0.90)
- Query-Policy Misalignment in Preference-Based Reinforcement Learning (ICLR 2024 · similarity 0.90)
- Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF (ICLR 2025 · similarity 0.90)
- CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.