Research map / Reinforcement learning / RL from feedback
PEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Transfers preference labels from source to target robotic tasks via Gromov-Wasserstein optimal transport alignment of trajectories (CPA), and learns a robust distributional reward model (RPT) for zero-shot offline preference-based RL.
Problem
Preference-based RL requires large numbers of human preference labels for each new task, and labels cannot be reused across tasks; the paper targets zero-shot preference transfer to target tasks without target-task human labels, plus robustness to noisy transferred labels.
Approach
Use Gromov-Wasserstein optimal transport to align source and target trajectories, then compute target preference labels as correspondence-weighted sums of source preference labels; learn rewards as Gaussian distributions (mean and variance branches, entropy regularizer) to be robust to label noise.
Key results
- RPT+CPA (zero-shot) achieves 78.1% average success across all tasks versus 83.8% for PT with scripted labels, and 38.1% for PT+Sim on Meta-World transfers.
- CPA labels reach 74.9% mean accuracy across all task pairs.
- RPT+CPA outperforms PT with scripted labels at low label counts; e.g., exceeds Oracle PT on Lift-MH for all tested label counts from 0 to 20 oracle labels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add reward-uncertainty modeling (RPT) to PT trained with CPA labels (ablation) | PT+CPA · Average (all settings) | success rate: 71.2 -> 78.1 | Helped |
| transfer preferences with CPA rather than trajectory similarity | Cross-task preference learning · Average (all settings) | success rate | Helped |
| use Cosine instead of Euclidean cost function (ablation) | RPT+CPA · Average | success rate: 81.1 -> 80.3 | Mixed |
| use Cosine instead of Euclidean cost function (ablation) | CPA · Average | CPA accuracy: 73.2 -> 74.0 | Helped |
| increase source trajectories from M=4 to M=8 or M=16 (ablation) | RPT+CPA · Average | success rate: 86.4; 83.9; 86.8 | No clear effect |
| use more target-task scripted preference labels alongside CPA labels | RPT+CPA · Window Open and Lift-MH | success rate | Helped |
| model reward uncertainty rather than only scalar rewards under label noise | RPT · Sweep Into, Window Open, and Lift-MH | success rate | Helped |
| increase regularization weight alpha (ablation) | RPT+CPA · Sweep Into, Window Open, and Lift-MH | success rate | Hurt |
| vary the RPT margin eta (ablation) | RPT+CPA · Sweep Into, Window Open, and Lift-MH | success rate | No clear effect |
| vary robust-term weight lambda (ablation) | RPT+CPA · Sweep Into, Window Open, and Lift-MH | success rate | No clear effect |
| use a more similar source task | CPA · Cross-task preference transfer | CPA accuracy: -0.43; -0.67 | Helped |
Surprising findings
- RPT+CPA exceeds PT trained on scripted labels on the Lift-MH task in the zero-shot setting.
- Cosine cost in the GW alignment gives CPA 91.0% accuracy on Window Open, above PT's scripted-label success rate on that task.
Limitations
- Not well-suited for high-dimensional inputs due to OT computational cost.
- Requires source and target tasks to share the same action space.
- Performance degrades when source and target tasks have large domain gaps (e.g., D4RL locomotion).
Experimental setup
Datasets: Meta-World Robomimic D4RL
Benchmarks: Meta-World Robomimic
Metrics: success rate CPA label accuracy
Baselines: PT PT+Semi IPL PT+Sim PEBBLE
Builds on: Preference Transformer IPL SURF PEBBLE Implicit Q-Learning
Closest papers by meaning
- Robust Reward Alignment via Hypothesis Space Batch Cutting (ICML 2025 · similarity 0.89)
- Adaptive Preference Scaling for Reinforcement Learning with Human Feedback (NeurIPS 2024 · similarity 0.88)
- LIBERO-PeRM: Benchmarking Personalized Robotic Manipulation (NeurIPS 2026 · similarity 0.87)
- Policy-labeled Preference Learning: Is Preference Enough for RLHF? (ICML 2025 · similarity 0.87)
- RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences (ICML 2024 · similarity 0.87)
- What Matters to You? Towards Visual Representation Alignment for Robot Learning (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.