Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Learning to Assist Humans without Inferring Rewards
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Empowerment via Successor Representations (ESR), learning contrastive successor representations to estimate effective empowerment and using them as an RL reward to train assistive agents without inferring human rewards.
Problem
Assisting humans without inferring their reward functions, since reward inference is hard in high-dimensional settings and prior empowerment estimators (e.g., AvE via random rollouts) do not scale.
Approach
Estimate effective empowerment (mutual information between the human's current action and discounted future states under current behavior policies) using two contrastive representations whose inner-product difference gives a log probability ratio; maximize it with SAC as an intrinsic reward for the assistant.
Key results
- On Overcooked Cramped Room, ESR achieves 91.33 ± 4.08 vs. AvE 5.13 ± 1.31 and random 69.26.
- On Asymmetric Advantages, ESR reaches 72.00 ± 5.37 vs. Reward Inference 60.33 ± 0.26, AvE 36.71 ± 1.71, random 59.36.
- On gridworld with harder variants (N=5,7,10 obstacles), ESR significantly outperforms AvE, which performs worse than random on moderate/high complexity tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ESR instead of Reward Inference | Overcooked assistant · Overcooked Asymmetric Advantages | reward: 60.33 ± 0.26 -> 72.00 ± 5.37 | Helped |
| Use ESR instead of Reward Inference | Overcooked assistant · Overcooked Coordination Ring | reward: 5.96 ± 0.20 -> 8.40 ± 0.69 | Helped |
| Use ESR instead of Random | Overcooked assistant · Overcooked Coordination Ring | reward: 6.02 -> 8.40 ± 0.69 | Helped |
| Use ESR instead of Random | Overcooked assistant · Overcooked Cramped Room | reward: 69.26 -> 91.33 ± 4.08 | Helped |
| Use ESR instead of Reward Inference | Overcooked assistant · Overcooked Cramped Room | reward: 39.24 ± 0.35 -> 91.33 ± 4.08 | Helped |
| Add conditioning on the robot action to the learned representations (ablation) | ESR · Obstacle Gridworld | reward | Helped |
| Use a non-greedy policy with γ = 0.9 instead of a greedy policy with γ = 0 (ablation) | ESR · 5x5 Gridworld with 10 obstacles | reward | Helped |
Surprising findings
- AvE performs worse than a random controller on the moderate and high complexity gridworld tasks and on Overcooked.
- A greedy empowerment policy (gamma=0) vastly underperforms the gamma=0.9 policy.
Limitations
- Assumes the assistant can observe the human's actions.
- On-policy learning requiring trial and error; not tested with real humans.
- Assumes shared state space between agents; partial observability not tested.
Experimental setup
Datasets: Obstacle gridworld (N=2,5,7,10 blocks) Overcooked (Asymmetric Advantages, Coordination Ring, Cramped Room)
Benchmarks: Obstacle gridworld Overcooked
Metrics: episode reward
Baselines: AvE Reward Inference Goal Inference Random
Builds on: empowerment (AvE) InfoNCE soft actor-critic assistance games
Closest papers by meaning
- AssistanceZero: Scalably Solving Assistance Games (ICML 2025 · similarity 0.85)
- What Is Worth Representing? Representational Empowerment for Continual Model Construction (NeurIPS 2026 · similarity 0.85)
- Behavioral Exploration: Learning to Explore via In-Context Adaptation (ICML 2025 · similarity 0.84)
- Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance (ICLR 2025 · similarity 0.84)
- Plasticity as the Mirror of Empowerment (NeurIPS 2025 · similarity 0.84)
- Learning to Cooperate with Humans using Generative Agents (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.