Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Designs for Enabling Collaboration in Human-Machine Teaming via Interactive and Explainable Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes interpretable tree-based (IDCT) teammates trained with RL in Overcooked-AI that humans can modify via a GUI across repeated episodes, and reports a 50-subject study with design guidelines for human-machine teaming.
Problem
Learned human-machine teaming agents (imitation/RL) are brittle, rigid, and individualistic, and opaque black-box policies prevent users from understanding or iteratively correcting them during repeated teaming.
Approach
Interpretable Discrete Control Trees trained with PPO and crispification, pruned post hoc, plus a GUI letting users modify the tree between teaming episodes (human-led policy modification), compared against AI-led modification and static white-box/black-box conditions.
Key results
- In Forced Coordination, Fictitious Co-Play significantly outperformed the tree-based conditions on max reward (Kruskal-Wallis chi2(4)=20.146, p<0.001).
- In Optional Collaboration, Human-Led Policy Modification significantly outperformed static white-box conditions (AI-led, static interpretable, static black-box) and showed significant within-condition improvement across iterations (p<0.01).
- All approaches underperformed the simple collaborative heuristic in Optional Collaboration (408 vs. at most ~295 for trained policies in training).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the collaborative heuristic instead of the individualized heuristic | Overcooked-AI agents · Optional Collaboration | cumulative team score: 306 -> 408 | Helped |
| Use the neural network policy instead of the IDCT policy | Collaborative agent policies · Forced Coordination | average reward: 315.22 ± 14.59 -> 403.16 ± 16.08 | Helped |
| Use the neural network policy instead of the IDCT policy | Collaborative agent policies · Optional Collaboration | average reward: 171.46 ± 18.89 -> 295.02 ± 1.86 | Helped |
| Replace the learned policy with the collaborative heuristic | Learned agent policies · Optional Collaboration | team score: 408 vs. 295.02 and 171.46 | Helped |
| Add human-led policy modification to the static interpretable policy (ablation) | IDCT teammate · Optional Collaboration | maximum team performance | Helped |
| Add human-led policy modification to the static black-box policy (ablation) | IDCT teammate · Optional Collaboration | maximum team performance | Helped |
| Add policy adaptation to a static interpretable policy (ablation) | IDCT teammate · Optional Collaboration | team performance across iterations | Helped |
| Add AI-led policy adaptation to the static interpretable policy (ablation) | IDCT teammate · Optional Collaboration | team performance across iterations | Helped |
| Add policy visualization to the static black-box condition (ablation) | Static IDCT teammate · Forced Coordination and Optional Collaboration | objective team performance | No clear effect |
| Use Fictitious Co-Play instead of human-led policy modification (ablation) | Human-machine team · Forced Coordination | maximum team performance | Helped |
| Use Fictitious Co-Play instead of AI-led modification, static interpretable, or static black-box policies (ablation) | Human-machine team · Optional Collaboration | maximum team performance | Helped |
| Use human-led policy modification instead of AI-led modification, static interpretability, or static black-box (ablation) | Human-machine team · Optional Collaboration | maximum team performance | Helped |
Surprising findings
- Interpretability alone (viewing the tree) gave no objective performance benefit over a black-box policy.
- All learned approaches underperformed a simple hand-coded collaborative heuristic, a negative result for learning-based HMT.
Limitations
- Study population was university students, mostly engineering, limiting generalization.
- Only two-agent, low-dimensional, short-horizon Overcooked-AI was tested.
- Interpretable trees underperform neural network policies in initial simulated performance.
Experimental setup
Datasets: Overcooked-AI (modified simultaneous-move version)
Benchmarks: Overcooked-AI Forced Coordination Overcooked-AI Optional Collaboration
Metrics: team reward (max and performance-round score) improvement across iterations NASA-TLX Human-Robot Collaborative Fluency Godspeed Inclusion of Other in the Self
Baselines: Fictitious Co-Play Carroll et al. human-like agents collaborative heuristic static IDCT policy AI-led policy modification
Builds on: Interpretable Discrete Control Trees differentiable decision trees PPO Fictitious Co-Play PantheonRL HA-PPO
Closest papers by meaning
- NestRL: A Nested Training Regime for Mutual Adaptation in Human–AI Teaming (NeurIPS 2026 · similarity 0.88)
- Adaptive Agents for Mixed-Initiative Human-AI Collaborations (AAAI 2025 · similarity 0.87)
- Learning “Partner-Aware” Collaborators in Multi-Party Collaboration (NeurIPS 2025 · similarity 0.87)
- Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learning (ICLR 2026 · similarity 0.87)
- Autonomous Policy Explanations for Effective Human-Machine Teaming (AAAI 2024 · similarity 0.87)
- Beyond Single Stationary Policies: Meta-Task Players as Naturally Superior Collaborators (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.