Research map / Reinforcement learning / RL from feedback
Query-Policy Misalignment in Preference-Based Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies query-policy misalignment in PbRL query selection and proposes QPA, which pairs policy-aligned query selection with hybrid experience replay on top of PEBBLE to improve feedback and sample efficiency on DMControl and MetaWorld.
Problem
Existing PbRL query selection schemes that maximize reward-model quality often query segments outside the current policy's visitation distribution, so the feedback has little effect on policy learning and feedback efficiency is poor.
Approach
Select preference queries only from recent on-policy segments (a small first-in-first-out query buffer) and sample half of each RL update batch from the policy-aligned buffer, so reward learning and Q-learning both focus on the current policy's state-action distribution, using a single reward model instead of an ensemble.
Key results
- QPA outperforms SURF and PEBBLE by a substantial margin on all six DMControl locomotion tasks in learning curves, e.g., Walker_walk-100 feedback: 802.65 vs 610.93 (SURF) and 388.58 (PEBBLE).
- On Cheetah_run-100, QPA reaches 652.56 vs 462.44 (SURF) and 521.44 (PEBBLE); on Door-open-3000, QPA reaches 84.4 vs 50.0 (SURF) and 79.6 (PEBBLE) success.
- QPA can surpass SAC with ground-truth reward during early training on some tasks such as Quadruped_run.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use QPA instead of PEBBLE with disagreement query selection | PEBBLE-based PbRL · DMControl locomotion and MetaWorld manipulation tasks | ground-truth reward or success rate | Helped |
| Add policy-aligned query selection (ablation) | PEBBLE · Walker_walk, Cheetah_run, Walker_run | task performance | Helped |
| Add policy-aligned query selection (ablation) | PEBBLE · QPA ablation tasks | final performance | Helped |
| Add hybrid experience replay to policy-aligned query selection (ablation) | PEBBLE with policy-aligned query selection · QPA ablation tasks | final performance | Helped |
| Add temporal data augmentation (ablation) | QPA · QPA ablation tasks | performance | Mixed |
| Use hybrid experience replay without policy-aligned query selection (ablation) | PEBBLE · Certain QPA ablation tasks | performance | Helped |
| Remove SAC MaxEntropy (ablation) | QPA · DMControl locomotion tasks | policy performance | Hurt |
| Use policy-aligned query selection instead of full-buffer query selection (ablation) | PEBBLE · Walker_walk, Cheetah_run, Walker_run | current-policy log likelihood of queried segments | Helped |
| Replace the full replay buffer with recent policy-aligned experience for queries | QPA · DMControl locomotion and MetaWorld manipulation tasks | feedback efficiency | Helped |
| Use QPA instead of SURF | Off-policy PbRL · DMControl locomotion tasks | ground-truth reward | Helped |
| Use QPA instead of MRN | PbRL · Most evaluated tasks | task performance | Helped |
| Use QPA with continued feedback instead of SAC with ground-truth reward | QPA and SAC · Quadruped_run | task score | Helped |
Surprising findings
- QPA, using a single reward model and random sampling from a small recent buffer, can exceed SAC trained with the ground-truth reward in early training on some tasks.
- Existing disagreement-based query selection often performs only marginally better than uniform random queries.
Limitations
- Policy-aligned query selection is inherently not applicable to on-policy PbRL methods, which already query on-policy segments but are sample-inefficient.
- Experiments use a scripted oracle overseer rather than real human feedback.
- Evaluated only on DMControl and MetaWorld; the motivating example is a simple 2D navigation task.
Experimental setup
Datasets: DMControl locomotion tasks (Walker_walk, Walker_run, Cheetah_run, Quadruped_walk, Quadruped_run, Humanoid_stand) MetaWorld tasks (Door-unlock, Drawer-open, Door-open) scripted oracle preferences
Benchmarks: DMControl MetaWorld
Metrics: episode return (ground truth reward) success rate feedback efficiency sample efficiency
Baselines: PEBBLE SURF PrePPO SAC with ground truth reward MRN
Builds on: PEBBLE B-Pref SAC SURF combined Q-learning
Closest papers by meaning
- Provable Reward-Agnostic Preference-Based Reinforcement Learning (ICLR 2024 · similarity 0.91)
- CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries (ICML 2025 · similarity 0.90)
- Preference-based Reinforcement Learning beyond Pairwise Comparisons: Benefits of Multiple Options (NeurIPS 2025 · similarity 0.90)
- PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback (ICLR 2024 · similarity 0.90)
- Efficient Preference-Based Reinforcement Learning: Randomized Exploration meets Experimental Design (NeurIPS 2025 · similarity 0.90)
- PILAF: Optimal Human Preference Sampling for Reward Modeling (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.