Research map / Reinforcement learning / RL from feedback
Preference Elicitation for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sim-OPRL, an offline preference-based RL method that elicits preferences over trajectories simulated in a learned environment model, with pessimism for transitions and optimism for reward uncertainty, and gives sample-complexity guarantees.
Problem
Preference elicitation for offline reinforcement learning: learning the best offline policy from an offline dataset of trajectories plus as few human preference queries as possible, without environment interaction.
Approach
Instead of querying preferences over offline dataset trajectories (OPRL), generate trajectory pairs by rolling out two exploratory candidate-optimal policies inside a learned, pessimistic transition model, so preferences target plausibly optimal policies while avoiding out-of-distribution regions.
Key results
- Sim-OPRL needs 4 ± 2 preference queries on Star MDP versus 32 ± 4 (OPRL Uniform) and 30 ± 4 (OPRL Uncertainty) to reach the target gap.
- On the Sepsis Simulation, Sim-OPRL requires 225 ± 46 preference queries versus 642 ± 72 for OPRL Uncertainty, while OPRL Uniform fails to reach the target.
- Sim-OPRL consistently achieves better environment returns than OPRL among fully offline methods across the five environments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use simulated model rollouts instead of uncertainty sampling from the offline trajectories | OPRL Uncertainty · Star MDP | preference queries N_p: 30 $\pm$ 4 -> 4 $\pm$ 2 | Helped |
| Use simulated model rollouts instead of uncertainty sampling from the offline trajectories | OPRL Uncertainty · Gridworld | preference queries N_p: {66 $\pm$ 7} -> 49 $\pm$ 7 | Helped |
| Use simulated model rollouts instead of uncertainty sampling from the offline trajectories | OPRL Uncertainty · MiniGrid-FourRooms | preference queries N_p: 53 $\pm$ 5 -> 41 $\pm$ 5 | Helped |
| Use simulated model rollouts instead of uncertainty sampling from the offline trajectories | OPRL Uncertainty · HalfCheetah-Random | preference queries N_p: 71 $\pm$ 8 -> 50 $\pm$ 10 | Helped |
| Use simulated model rollouts instead of uncertainty sampling from the offline trajectories | OPRL Uncertainty · Sepsis Simulation | preference queries N_p: 642 $\pm$ 72 -> 225 $\pm$ 46 | Helped |
| Remove pessimism from the output policy (ablation) | Sim-OPRL · Star MDP | environment return | Hurt |
| Remove pessimism with respect to the transition model during simulated rollouts (ablation) | Sim-OPRL · Star MDP | environment return | Hurt |
| Remove optimism with respect to reward uncertainty during preference sampling (ablation) | Sim-OPRL · Star MDP | environment return | Hurt |
| Increase offline dataset size to improve transition-model accuracy (ablation) | Sim-OPRL · Star MDP | preference queries N_p | Helped |
| Use an offline dataset with poor coverage of the optimal policy (ablation) | Sim-OPRL · Star MDP | preference queries N_p | Hurt |
Surprising findings
- The offline optimal policy pi*_offline can be suboptimal, and its suboptimality is not a lower bound for the learned policy's suboptimality (counterexample in appendix).
Limitations
- Theoretical guarantees assume realizability and identifiable models with no optimization error.
- Concentrability coefficients can be large for suboptimal or low-coverage offline data, increasing preference needs.
Experimental setup
Datasets: D4RL Sepsis Simulation StarMDP Gridworld MiniGrid-FourRooms HalfCheetah-Random
Benchmarks: StarMDP Gridworld MiniGrid-FourRooms HalfCheetah-Random Sepsis Simulation
Metrics: normalized environment return preference sample complexity N_p to reach suboptimality gap
Baselines: OPRL Uniform OPRL Uncertainty PbOP
Builds on: OPRL Bradley-Terry model PbOP MOPO
Closest papers by meaning
- Adversarial Policy Optimization for Offline Preference-based Reinforcement Learning (ICLR 2025 · similarity 0.91)
- Design Considerations in Offline Preference-based RL (ICML 2025 · similarity 0.91)
- Listwise Reward Estimation for Offline Preference-based Reinforcement Learning (ICML 2024 · similarity 0.90)
- Provable Offline Preference-Based Reinforcement Learning (ICLR 2024 · similarity 0.90)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.