Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Predictive Preference Learning from Human Interventions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PPL, an interactive imitation learning method that bootstraps each human intervention into a preference horizon of predicted future states and trains with a CPO-plus-BC loss, reducing expert data and interventions in MetaDrive and Robosuite.
Problem
Active human-intervention imitation learning corrects only the current state, so the agent repeats similar mistakes in future states, requiring many expert demonstrations and constant expert monitoring.
Approach
Roll out the agent's intended action with a trajectory predictor, let the human review the predicted future, and convert each intervention into preference pairs (predicted future state, human action, agent action) over a preference horizon L, trained with a contrastive preference loss plus behavior cloning.
Key results
- In MetaDrive with a real human expert, PPL reaches 0.76 success rate using 2.9K human data versus 0.46 for PVP (4.9K human data) at 10K total steps.
- With a neural expert in MetaDrive, PPL achieves 0.80 success rate with 1.2K expert data and 6K total data, versus 0.56 for PVP (2.5K expert data, 10K total).
- In Nut Assembly, PPL reaches 0.51 success rate versus 0.35 for PVP at 200K total data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use predictive preference learning from interventions instead of PVP | Interactive imitation learning with a real human expert · MetaDrive safety | success rate: 0.46 -> 0.76 | Helped |
| use predictive preference learning from interventions instead of PVP | Interactive imitation learning with a neural expert · MetaDrive | success rate: 0.56 -> 0.80 | Helped |
| use predictive preference learning from interventions instead of PVP | Interactive imitation learning with a neural expert · Table Wiping | success rate: 0.58 -> 0.80 | Helped |
| use predictive preference learning from interventions instead of PVP | Interactive imitation learning with a neural expert · Nut Assembly | success rate: 0.35 -> 0.51 | Helped |
| add contrastive preference pairs instead of only behavior-cloning positive actions (ablation) | PPL · MetaDrive, neural expert | success rate: 0.36 -> 0.81 | Helped |
| use the recorded positive actions rather than random positive actions (ablation) | PPL · MetaDrive, neural expert | success rate: 0.45 -> 0.81 | Helped |
| use the recorded negative actions rather than random negative actions (ablation) | PPL · MetaDrive, neural expert | success rate: 0.38 -> 0.81 | Helped |
| replace CPO with DPO (ablation) | PPL · MetaDrive, neural expert | success rate: 0.81 -> 0.80 | No clear effect |
| use CPO rather than IPO (ablation) | PPL · MetaDrive, neural expert | success rate: 0.35 -> 0.81 | Helped |
| use CPO rather than SLiC-HF (ablation) | PPL · MetaDrive, neural expert | success rate: 0.32 -> 0.81 | Helped |
| add the preference loss to BC (ablation) | PPL · MetaDrive, neural expert | success rate: 0.42 -> 0.81 | Helped |
| add the BC loss to CPO (ablation) | PPL · MetaDrive, neural expert | success rate: 0.04 -> 0.81 | Helped |
Surprising findings
- Using DPO instead of the CPO-based objective gave comparable or slightly better results (0.80 vs 0.81 success in ablation), though DPO needs a pretrained reference policy.
- Discarding the BC loss and training with only the preference loss collapsed performance (0.04 success rate).
Limitations
- Assumes the expert always knows and accurately demonstrates the optimal corrective action, while real demonstrations may be suboptimal or inconsistent.
- All experiments are in simulation; effectiveness on real robots is untested.
- Trajectory predictor assumes other traffic participants stay stationary, and preference labels degrade for long horizons.
Experimental setup
Datasets: MetaDrive safety benchmark Robosuite Table Wiping Robosuite Nut Assembly
Benchmarks: MetaDrive safety benchmark Robosuite Table Wiping Robosuite Nut Assembly
Metrics: success rate episodic return route completion rate human data usage overall intervention rate
Baselines: BC GAIL Ensemble-DAgger Thrifty-DAgger PVP IWR EIL HACO
Builds on: CPO HG-DAgger PVP DPO PPO-Lagrangian
Closest papers by meaning
- Direct Post-Training Preference Alignment for Multi-Agent Motion Generation Model Using Implicit Feedback from Pre-training Demonstrations (ICLR 2025 · similarity 0.89)
- Limited Preference Aided Imitation Learning from Imperfect Demonstrations (ICML 2024 · similarity 0.89)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.89)
- A Minimaximalist Approach to Reinforcement Learning from Human Feedback (ICML 2024 · similarity 0.89)
- Policy-labeled Preference Learning: Is Preference Enough for RLHF? (ICML 2025 · similarity 0.88)
- Improving Reward Model Generalization from Adversarial Process Enhanced Preferences (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.