Research map / Reinforcement learning / RL from feedback
Policy-labeled Preference Learning: Is Preference Enough for RLHF?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Policy-labeled Preference Learning (PPL), a DPO-style regret-based preference model that labels each segment with its behavior policy to fix likelihood mismatch, plus a contrastive KL regularizer, evaluated on MetaWorld offline and online.
Problem
Existing RLHF/DPO-style preference learning for sequential decision making treats all trajectories as if generated by an optimal policy, causing likelihood mismatch and suboptimal learning, especially with heterogeneous offline data from diverse policies under environmental stochasticity.
Approach
Replace the optimal-advantage score used in CPL with a negative-regret score that depends on the behavior policy label of each segment, yielding a policy-labeled Bradley-Terry objective; regret decomposes into log-likelihood plus a sequential forward KL term, which is implemented as a contrastive KL regularizer over L-step rollouts.
Key results
- On heterogeneous sparse preference data, PPL reaches 83.8 on Bin-Picking and 83.5 on Button-Press versus 12.2-48.0 for SFT/P-IQL/CPL on those tasks (CPL 18.0 on Bin-Picking).
- On homogeneous dense data PPL achieves best results on Bin-Picking (83.5) and Drawer-Open (69.2) among the compared methods.
- In online from-scratch training across five MetaWorld tasks, PPL reaches performance comparable to PEBBLE while using only 8.8% of its parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use policy-labeled regret-based learning instead of CPL without behavior-policy labels | PPL · MetaWorld heterogeneous offline datasets | success rate | Mixed |
| Use policy-labeled regret-based learning instead of CPL without behavior-policy labels | PPL · MetaWorld heterogeneous offline datasets | success rate: 1.2 $\pm$ 0.8 -> 59.7 $\pm$ 18.6; 49.7 $\pm$ 3.0 -> 73.8 $\pm$ 3.3; 26.0 $\pm$ 2.2 -> 58.5 $\pm$ 3.8 | Mixed |
| Replace policy labels with deterministic pseudo-labels based on observed actions (ablation) | PPL · MetaWorld homogeneous offline datasets | average success rate | Hurt |
| Replace policy labels with deterministic pseudo-labels based on observed actions (ablation) | PPL · MetaWorld heterogeneous offline datasets | average success rate | Hurt |
| Set look-ahead horizon to L=0, removing sequential rollout planning (ablation) | PPL · MetaWorld offline preference learning | success rate | No clear effect |
| Apply PPL online from scratch rather than using PEBBLE with unsupervised pretraining | PPL · Five MetaWorld online tasks | average success rate | No clear effect |
Surprising findings
- PPL performs better with Sparse preference settings than Dense on heterogeneous data, unlike baselines.
- CPL collapses on heterogeneous data (e.g., 1.2 on Bin-Picking), below SFT.
Limitations
- Offline pseudo-labels assume deterministic behavior policies when labels are unknown, and PPL-deterministic degrades on heterogeneous data.
- Online RLHF is sensitive to the sampling of queries from preference data.
- Results are limited to MetaWorld manipulation tasks with a single oracle-based labeling setup.
Experimental setup
Datasets: MetaWorld rollouts (Bin-Picking, Button-Press, Door-Open, Drawer-Open, Plate-Slide, Sweep-Into)
Benchmarks: MetaWorld
Metrics: success rate
Baselines: CPL P-IQL SFT PEBBLE PPL-deterministic
Builds on: Direct Preference Optimization Contrastive Preference Learning PEBBLE Implicit Q-Learning
Closest papers by meaning
- Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.94)
- Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference (ICLR 2025 · similarity 0.92)
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.92)
- Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF (ICLR 2025 · similarity 0.92)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.92)
- PILAF: Optimal Human Preference Sampling for Reward Modeling (ICML 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.