Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Promptable Behaviors: Personalizing Multi-Objective Rewards from Human Preferences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a single multi-objective navigation policy conditioned on reward weight vectors (single-policy MORL) and infers user preferences via demonstrations, trajectory-comparison feedback (including group comparisons), or ChatGPT-translated language instructions, without fine-tuning the policy.
Problem
Personalizing embodied agents (robot navigation) to diverse human preferences over multiple objectives without retraining the policy for each preference.
Approach
Condition a DD-PPO navigation policy on a randomly sampled reward weight vector over K objectives during training, then at deployment infer the user's weight vector from demonstrations, (group) preference comparisons, or LLM-parsed language, keeping the policy frozen.
Key results
- Promptable Behaviors achieves 0.680 success with house exploration prioritized on ProcTHOR ObjectNav, 11.3% higher than EmbCLIP, and 0.650 success with path efficiency prioritized, 19.3% higher SPL than EmbCLIP.
- Group trajectory comparison with two trajectories per group reaches 93.5% cosine similarity to expert-designed weights using 25 feedback queries, versus 50.8% from... (pairwise M=1 reaches 0.897 at 500 queries)
- Human evaluation win rate is 0.650 for group comparison (M=2) versus 0.552 for pairwise comparison, a 17.8% improvement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Prioritize house exploration with the reward weights | Promptable Behaviors · ProcTHOR ObjectNav | success rate: 11.3% higher than EmbCLIP | Helped |
| Prioritize path efficiency with the reward weights | Promptable Behaviors · ProcTHOR ObjectNav | SPL: 19.3% | Helped |
| Prioritize path efficiency with the reward weights | Promptable Behaviors · ProcTHOR ObjectNav | path efficiency reward: 56.1% | Helped |
| Use group trajectory comparisons instead of pairwise comparisons | Promptable Behaviors reward-weight prediction · ProcTHOR ObjectNav human evaluation | win rate: 17.8% | Helped |
| Use group comparisons with two trajectories per group | Promptable Behaviors reward-weight prediction · ProcTHOR ObjectNav | cosine similarity: 0.935 | Helped |
| Use ChatGPT with chain-of-thought reasoning | Promptable Behaviors reward-weight prediction · ProcTHOR ObjectNav | cosine similarity: 0.614 | Helped |
Surprising findings
- Pairwise comparison yields the most peaked predicted weights, while human demonstrations yield the least peaked weights, with no direct correlation between peakedness and prediction accuracy.
Limitations
- Assumes static and linear human preferences.
- Evaluated only in simulated navigation tasks (AI2-THOR), not manipulation or real robots.
- Human evaluation involves only five participants and five scenarios.
Experimental setup
Datasets: ProcTHOR RoboTHOR
Benchmarks: ProcTHOR ObjectNav ProcTHOR FleeNav RoboTHOR ObjectNav RoboTHOR FleeNav
Metrics: success rate SPL PLOPL sub-rewards cosine similarity generalized Gini index win rate
Baselines: EmbCLIP Prioritized EmbCLIP Ask4Help
Builds on: DD-PPO EmbCLIP Ask4Help Bradley-Terry model PPO
Closest papers by meaning
- LIBERO-PeRM: Benchmarking Personalized Robotic Manipulation (NeurIPS 2026 · similarity 0.91)
- AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model (ICLR 2024 · similarity 0.89)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.88)
- Predictive Preference Learning from Human Interventions (NeurIPS 2025 · similarity 0.88)
- Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries (ICML 2025 · similarity 0.87)
- Behavioral Exploration: Learning to Explore via In-Context Adaptation (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.