Research map / Reinforcement learning / RL from feedback
Offline Preference-Based Value Optimization
Hyungkyu Kang, Min-hwan Oh
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace TD loss with value alignment loss (ablation) | IQL (giving PVO) · Meta-World medium-replay, 8 tasks, 1000 feedback | success rate (%): IQL 58.76 -> PVO 69.14 (average) | Improved |
| replace TD loss with value alignment loss (ablation) | TD3+BC · Meta-World medium-replay, 1000 feedback | success rate (IQM) | Improved |
| replace TD loss with value alignment loss (ablation) | XQL · Meta-World medium-replay, 1000 feedback | success rate (IQM) | Improved |
| use a learned transition model instead of dataset-sample approximation (ablation) | PVO · Meta-World medium-replay/expert, 500 and 1000 feedback | success rate (%): medium-replay 1000: 69.14 vs 71.18; medium-expert 1000: 73.00 vs 71.16 | No clear effect |
| reduce preference feedback to about 100 samples (ablation) | PVO · Meta-World medium-replay dial-turn | success rate (%) | No clear effect |
| increase proportion of random trajectories in dataset (ablation) | PVO, IQL, APPO · Meta-World dial-turn mixture datasets | success rate (%) | Hurt |
| use PVO instead of IQL with learned reward on button-press-topdown | PVO · Meta-World medium-replay | success rate (%) | Improved |
| use PVO instead of Flow to Better (FTB) | PVO · Meta-World medium-replay, 1000 feedback | success rate (%): FTB 0.00 on 6 tasks; sweep-into PVO 27.84 vs FTB 62.67 | Mixed |
Closest papers by meaning
- Adversarial Policy Optimization for Offline Preference-based Reinforcement Learning (ICLR 2025 · similarity 0.93)
- Preference Elicitation for Offline Reinforcement Learning (ICLR 2025 · similarity 0.92)
- Provable Offline Preference-Based Reinforcement Learning (ICLR 2024 · similarity 0.91)
- Provable Reward-Agnostic Preference-Based Reinforcement Learning (ICLR 2024 · similarity 0.90)
- Query-Policy Misalignment in Preference-Based Reinforcement Learning (ICLR 2024 · similarity 0.90)
- In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
