Research map / Reinforcement learning / RL from feedback
DUO: Diverse, Uncertain, On-Policy Query Generation and Selection for Reinforcement Learning from Human Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DUO, a query generation and selection scheme for RLHF that picks on-policy, epistemically uncertain, and diverse trajectory-segment pairs to reduce the number of human preference queries needed.
Problem
RLHF for control requires many pairwise trajectory-segment preference queries to learn a reward function, and existing methods waste queries on off-policy trajectories, heuristic (aleatoric-inclusive) uncertainty, and redundant query batches.
Approach
Combines three filters: priority sampling of trajectories by log-likelihood under the current policy (on-policiness), ranking queries by the epistemic uncertainty measured as the length of the ensemble prediction interval, and K-means clustering in predicted reward-difference space with elbow-chosen K for batch diversity; implemented on top of PEBBLE.
Key results
- On Quadruped walk, DUO significantly outperforms baselines and reaches a level similar to SAC with ground-truth reward.
- On Meta-World Door open, DUO matches baseline final success while using only about 2760 of the 5000 query budget (baselines use nearly all).
- In a user study with 150 queries on Quadruped, DUO's agent always performed the desired standing and waving behavior across 10 evaluations, whereas PEBBLE's hardly stood up.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DUO query generation and selection instead of baseline querying | PEBBLE · DMControl Walker walk | episode reward | Helped |
| Use DUO query generation and selection instead of baseline querying | PEBBLE · DMControl Cheetah run | episode reward | Mixed |
| Use DUO query generation and selection instead of baseline querying | PEBBLE · DMControl Quadruped walk | episode reward | Helped |
| Use DUO query generation and selection instead of baseline querying | PEBBLE · Meta-World Sweep into | success rate | Helped |
| Use DUO query generation and selection instead of baseline querying | PEBBLE · Meta-World Drawer open | success rate | Helped |
| Use DUO query generation and selection instead of baseline querying | PEBBLE · Meta-World Door open | success rate | No clear effect |
| Use DUO query generation and selection instead of baseline querying | PEBBLE · Meta-World Door open | queries used: 2760 vs 5000 | Helped |
| Add on-policy trajectory prioritization (ablation) | DUO · various tasks | performance | Helped |
| Add epistemic uncertainty filtering (ablation) | DUO · various tasks | performance | Helped |
| Add diversity filtering (ablation) | DUO · various tasks | performance | Helped |
Surprising findings
- On Cheetah run, DUO falls slightly behind QPA, which also uses on-policiness.
- Under the mistake irrational teacher, DUO works a bit worse than QPA on Quadruped walk.
Limitations
- Evaluation limited to control and robotic manipulation tasks; no LLM alignment experiments.
- Evaluation relies on simulated scripted teachers apart from a single user study.
- Diversity filter yields an unfixed number of queries per session, extending the querying process.
Experimental setup
Datasets: DMControl (Walker walk, Cheetah run, Quadruped walk) Meta-World (Sweep into, Drawer open, Door open, Window open)
Benchmarks: DMControl Walker walk DMControl Cheetah run DMControl Quadruped walk Meta-World Sweep into Meta-World Drawer open Meta-World Door open Meta-World Window open
Metrics: true episode return success rate
Baselines: PEBBLE SURF RUNE MRN QPA SAC (ground-truth reward)
Builds on: PEBBLE SAC Bradley-Terry model
Closest papers by meaning
- Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization (ICML 2024 · similarity 0.92)
- Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference (ICLR 2025 · similarity 0.91)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.90)
- Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback (ICLR 2024 · similarity 0.90)
- Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.90)
- Dual RL: Unification and New Methods for Reinforcement and Imitation Learning (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.