Research map / Reinforcement learning / MDPs and planning
Reinforcement Learning with Pairwise Preferences in Long-Term Decision Problems
Jonathan Colaço Carr, Prakash Panangaden, Doina Precup, Benjamin Van Roy
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use HPI instead of SPPO for pairwise-preference RL | HPI · Mujoco-v5 suite and Mujoco-NT (13 tasks, true preference margin) | AUC confidence interval vs SPPO: 12/13 tasks exceed or overlap SPPO | Improved |
| use HPI-Clip instead of SPPO | HPI-Clip · Mujoco-v5 suite and Mujoco-NT (13 tasks, true preference margin) | AUC confidence interval vs SPPO: 12/13 tasks exceed or overlap SPPO | Improved |
| use HPI-Clip instead of SPPO on nontransitive preference margins | HPI-Clip · Mujoco-NT (Reacher-NT, Walker2d-NT) | AUC of inverted optimality gap: strictly higher CI on both tasks | Improved |
| use HPI-Clip instead of SPPO with learned preference models (ablation) | HPI-Clip · Mujoco-v5 suite (learned preference model) | AUC confidence interval vs SPPO: better on 7 of 11 tasks, worse on 1 of 11 | Improved |
| use HPI instead of SPPO with learned preference models (ablation) | HPI · Mujoco-v5 suite (learned preference model) | AUC confidence interval vs SPPO: better on 8 of 11 tasks, worse on 1 of 11 | Improved |
| use HPI and HPI-Clip instead of SPPO with learned preference models (ablation) | HPI, HPI-Clip · Mujoco-NT (learned preference model) | AUC confidence interval vs SPPO: strictly higher CI on both tasks | Improved |
| increase the comparison horizon TC (average the preference cumulant over more timesteps, towards SPPO) (ablation) | SPPO variants vs HPI-Clip · Mujoco tasks | learning efficiency | Hurt |
Closest papers by meaning
- Efficient Preference-Based Reinforcement Learning: Randomized Exploration meets Experimental Design (NeurIPS 2025 · similarity 0.90)
- Preference-based Reinforcement Learning with Finite-Time Guarantees (NeurIPS 2020 · similarity 0.90)
- Towards Theoretical Understanding of Sequential Decision Making with Preference Feedback (ICML 2025 · similarity 0.89)
- Reward is enough for convex MDPs (NeurIPS 2021 · similarity 0.89)
- Policy-Regret Minimization in Markov Games with Function Approximation (ICML 2025 · similarity 0.88)
- Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
