Research map / Reinforcement learning / RL from feedback
PAWS: Preference Learning with Advantage-Weighted Segments
Aleksandar Taranovic, Onur Celik, Niklas Freymuth, Ge Li, Serge Thilges, Huy Le, Tai Hoang, Rania Rayyes, Gerhard Neumann
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| update policy on segment-level advantages instead of single state-action advantages (ablation) | PAWS (MLP) · Meta-World 10 tasks, 500 preferences | success rate (%): 67 -> 78 | Improved |
| update policy on segment-level advantages instead of single state-action advantages (ablation) | PAWS (Trans.) · Meta-World 10 tasks, 50 preferences | success rate (%): 44 -> 52 | Improved |
| increase segment length under single-step updates (ablation) | state-action-based update · Meta-World 10 tasks | success rate | Hurt |
| shorten update-time segments from 64 to 8 with fixed advantage model (ablation) | PAWS (Trans.) · Meta-World 10 tasks, 500 preferences | success rate (%): 78.2 -> 64.3 | Hurt |
| lower effective sample size n_eff to 10% with 500 preferences (ablation) | PAWS · Peg Insert Side, Sweep Into, Drawer Open | success rate | Improved |
| use larger n_eff with only 50 preferences (ablation) | PAWS · Peg Insert Side, Sweep Into, Drawer Open | success rate | Improved |
| use segment-based instead of step-based update for rank preservation (ablation) | PAWS · Spearman rank correlation with expert | Spearman rs: 0.05 -> 0.22 | Improved |
| use Transformer instead of MLP advantage function (ablation) | PAWS · Meta-World 10 tasks | success rate (%): 78.3 -> 78.2 (500 prefs) | No clear effect |
| add KL regularization to CPL | CPL · Meta-World 10 tasks | success rate (%): 67.6 -> 67.3 (500 prefs) | No clear effect |
| use PAWS with human-collected preferences | PAWS (MLP) · Meta-World Door Open, 50 preferences | success rate (%): 73.6 -> 86.0 (BC -> PAWS MLP) | Improved |
| use baselines with only 50 preferences vs behavior cloning | CPL, CPL+KL, IPL · Meta-World 10 tasks, 50 preferences | success rate (%): 46.2 -> 42.9 (CPL) | Hurt |
Closest papers by meaning
- Hindsight PRIORs for Reward Learning from Human Preferences (ICLR 2024 · similarity 0.89)
- STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- PEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulation (ICML 2024 · similarity 0.88)
- LIBERO-PeRM: Benchmarking Personalized Robotic Manipulation (NeurIPS 2026 · similarity 0.88)
- Limited Preference Aided Imitation Learning from Imperfect Demonstrations (ICML 2024 · similarity 0.88)
- Efficient Preference-Based Reinforcement Learning: Randomized Exploration meets Experimental Design (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
