Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challenge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CHAIC, an accessibility-focused embodied social intelligence benchmark in ThreeDWorld where a helper agent must infer and assist constrained human-like partners, and proposes an LLM agent augmented with a video-based behavior modeling module.
Problem
Embodied helper agents struggle to perceive a human partner's goals and physical constraints from egocentric visual observations and to plan cooperative assistance tailored to them, especially for people with accessibility needs and in outdoor/emergency scenarios.
Approach
A challenge with four constrained agent types (child, wheelchair, bicycle, frail) across eight indoor/outdoor tasks, plus an LLM-based helper (CoELA-style) augmented with a TSN action-recognition behavior modeling module that grounds the partner's observed actions to objects for goal inference.
Key results
- LLM (GPT-4) + BM helper achieves the highest transport rate in 6 of 8 tasks and the largest efficiency improvement in 7 of 8 tasks (e.g., 0.49 TR in High Container vs 0.37 without helper).
- Goal inference accuracy of all baselines remains far below the Oracle helper (e.g., 0.25-0.47 vs ~0.9 in indoor tasks), indicating difficulty inferring partner goals from raw RGB-D.
- The fine-tuned action recognition model achieves 86.1% top-1 accuracy on validation; RL and SmartHelp baselines often perform no better than working without a helper in most tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LLM+BM instead of no helper | CHAIC helper agent · No constraint | transport rate: 0.53 -> 0.65 | Helped |
| Use LLM+BM instead of no helper | CHAIC helper agent · High Target | transport rate: 0.30 -> 0.38 | Helped |
| Use LLM+BM instead of no helper | CHAIC helper agent · High Container | transport rate: 0.37 -> 0.49 | Helped |
| Use LLM+BM instead of no helper | CHAIC helper agent · High Goalplace | transport rate: 0.28 -> 0.36 | Helped |
| Use LLM+BM instead of no helper | CHAIC helper agent · Low Target | transport rate: 0.51 -> 0.70 | Helped |
| Use LLM+BM instead of RHP | CHAIC helper agent · Obstacle | transport rate: 0.44 -> 0.42 | Hurt |
| Use LLM+BM instead of no helper | CHAIC helper agent · Shopping | transport rate: 0.37 -> 0.58 | Helped |
| Use LLM+BM instead of VLM | CHAIC helper agent · Furniture | transport rate: 0.70 -> 0.69 | Hurt |
| Use LLM+BM instead of RHP | CHAIC helper agent · Shopping | emergency rate: 0.30 -> 0.38 | Hurt |
Surprising findings
- VLM Helper (GPT-4o) can underperform both the random helper and the no-helper baseline in some tasks because it consistently follows the constrained agent without transporting objects.
- LLM- and VLM-based helpers handle shopping emergencies less efficiently than the rule-based RHP helper.
Limitations
- Constrained agents follow rule-based planners, lacking human behavioral diversity.
- Simulated constraints cannot cover all real-world accessibility needs.
- LLM helpers struggle with spatial reasoning from text and underestimate goal travel cost.
Experimental setup
Datasets: CHAIC training episodes (96 across 8 tasks) Kinetics-400
Benchmarks: CHAIC
Metrics: transport rate efficiency improvement goal inference accuracy emergency rate completion ratio
Baselines: Random Helper Rule-Based Hierarchical Plan Helper (RHP) VLM Helper (GPT-4o) Oracle Helper RL (PPO) SmartHelp
Builds on: CoELA Watch-and-Help SmartHelp ThreeDWorld
Closest papers by meaning
- PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks (ICLR 2025 · similarity 0.86)
- Position: Towards Bidirectional Human-AI Alignment (NeurIPS 2025 · similarity 0.86)
- Habitat 3.0: A Co-Habitat for Humans, Avatars, and Robots (ICLR 2024 · similarity 0.85)
- CaPo: Cooperative Plan Optimization for Efficient Embodied Multi-Agent Cooperation (ICLR 2025 · similarity 0.85)
- LLawCo: Learning Laws of Cooperation for Modeling Embodied Multi-Agent Behavior (ICML 2026 · similarity 0.85)
- Infer Human’s Intentions Before Following Natural Language Instructions (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.