Research map / Reinforcement learning / Offline RL
Preference-based Policy Optimization from Sparse-reward Offline Dataset
Wenjie Qiu, Guofeng Cui, Shicheng Liu, Yuanlin Duan, He Zhu
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PREFORL degraded-preference contrast instead of CPL's dataset failures | CPL · Sparse-MuJoCo walker2d-medium | success rate (%): 85.3 -> 98.0 | Improved |
| use PREFORL instead of CPL | CPL · Sparse-MuJoCo halfcheetah-medium-expert | success rate (%): 47.3 -> 100.0 | Improved |
| use PREFORL instead of CPL | CPL · Sparse-MuJoCo hopper-medium-expert | success rate (%): 0.0 -> 98.4 | Improved |
| use PREFORL instead of CPL | CPL · D4RL Adroit door-cloned | normalized score: 3.6 -> 16.3 | Improved |
| use PREFORL instead of CPL | CPL · D4RL Adroit pen-human | normalized score: 100.1 -> 119.1 | Improved |
| use PREFORL instead of BC | BC · MetaWorld Peg Insert | success rate (%): 49.0 -> 73.3 | Improved |
| use PREFORL instead of BC | BC · MetaWorld Push Wall | success rate (%): 47.7 -> 84.7 | Improved |
| use PREFORL instead of ReBRAC | ReBRAC · D4RL AntMaze large-diverse-v2 | average score: 64.0 -> 26.1 | Hurt |
| use PREFORL instead of ReBRAC | ReBRAC · D4RL AntMaze large-play-v2 | average score: 52.3 -> 53.2 | Improved |
| use PREFORL instead of ReBRAC | ReBRAC · Maze2D Medium | average successes: 0.71 -> 0.67 | Hurt |
| lower contrastive bias lambda from 0.5 to 0.25 (ablation) | PREFORL · D4RL Adroit pen-human | normalized score: 119.1 -> 113.9 | Hurt |
| raise contrastive bias lambda from 0.5 to 1 (ablation) | PREFORL · D4RL Adroit hammer-expert | normalized score: 128.6 -> 122.5 | Hurt |
Closest papers by meaning
- Preference Elicitation for Offline Reinforcement Learning (ICLR 2025 · similarity 0.92)
- Adversarial Policy Optimization for Offline Preference-based Reinforcement Learning (ICLR 2025 · similarity 0.91)
- Provable Offline Preference-Based Reinforcement Learning (ICLR 2024 · similarity 0.90)
- A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspective (NeurIPS 2024 · similarity 0.90)
- Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning (ICML 2024 · similarity 0.90)
- In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
