Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Habitat 3.0: A Co-Habitat for Humans, Avatars, and Robots
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Habitat 3.0, a simulator for humanoid avatars and robots in home scenes with a human-in-the-loop evaluation tool, and benchmarks social navigation and social rearrangement with learned and heuristic baselines.
Problem
Studying collaborative human-robot tasks in home environments is hard on real hardware (safety, cost, scalability, standardized benchmarks), and existing simulators either lack realistic humanoids or lack robot-humanoid interaction.
Approach
A simulator that combines SMPL-X-based skinned humanoid avatars with cached body shapes and LBS for fast simulation, a hierarchical humanoid behavior model (path-planned walking and VPoser-interpolated reaching), and a client-server human-in-the-loop tool letting real people control avatars alongside robot policies.
Key results
- Heuristic expert reaches 100% finding success vs 97% for end-to-end RL policy in social navigation; RL matches collision rate (0.51 vs 0.52).
- Plan-Pop3/Plan-Pop4 achieve highest zero-shot coordination success rate of 71.7% in social rearrangement, vs 50.9% for Learn-Single on ZSC partners.
- In a 30-participant human-in-the-loop study, Learn-Single and Plan-Pop3 raise relative efficiency to 134% and 123% over solo humans.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove humanoid GPS from the End-to-end RL policy (ablation) | End-to-end RL · Social Navigation | Finding Success (S): 0.97_{\pm 0.00} -> 0.76_{\pm 0.02} | Hurt |
| remove humanoid GPS from the End-to-end RL policy (ablation) | End-to-end RL · Social Navigation | Finding Success Weighted by Path Steps (SPS): 0.65_{\pm 0.00} -> 0.34_{\pm 0.01} | Hurt |
| remove humanoid detector from the End-to-end RL policy (ablation) | End-to-end RL · Social Navigation | Collision Rate (CR): 0.51_{\pm 0.03} -> 0.64_{\pm 0.05} | Hurt |
| remove arm depth from the End-to-end RL policy (ablation) | End-to-end RL · Social Navigation | Following Rate (F): 0.44_{\pm 0.01} -> 0.19_{\pm 0.01} | Hurt |
| remove arm depth from the End-to-end RL policy (ablation) | End-to-end RL · Social Navigation | Collision Rate (CR): 0.51_{\pm 0.03} -> 0.71_{\pm 0.08} | Hurt |
| replace oracle low-level skills with learned skills without retraining the high-level policy (ablation) | Plan-Pop3 · Social Rearrangement, train-pop-eval | Success Rate (SR): 77.79% -> 41.96% | Hurt |
| use RGB instead of depth (ablation) | Social Rearrangement robot policy · Social Rearrangement | Success Rate (SR) | No clear effect |
| remove humanoid GPS (ablation) | Social Rearrangement robot policy · Social Rearrangement | Success Rate (SR) | Hurt |
| remove humanoid GPS (ablation) | Social Rearrangement robot policy · Social Rearrangement | Relative Efficiency (RE) | Hurt |
| train with Plan-Pop3/4 rather than a single-partner population | Social Rearrangement robot policy · Social Rearrangement, ZSC-pop-eval | Success Rate (SR): 71.7% | Helped |
| pair the human with a learned robot policy | Learn-Single · Human-in-the-Loop Social Rearrangement | Relative Efficiency (RE): 100.0 -> 133.80 | Helped |
| pair the human with a population-trained robot policy | Plan-Pop3 · Human-in-the-Loop Social Rearrangement | Relative Efficiency (RE): 100.0 -> 123.46 | Helped |
Surprising findings
- Learn-Single achieves the highest relative efficiency with train partners but drops from 159.2 to 106.0 on unseen collaborators, while Learn-Pop with more policies still generalizes poorly.
- Learned low-level skills substantially degrade social rearrangement success (77.79% to 41.96% on train-pop), since the high-level policy was trained with oracle skills.
Limitations
- Pick/place are teleport-based oracle skills and VPoser reaching fails for hand positions outside the precomputed range.
- Humanoid behaviors are limited to a restricted set of motions (walking and reaching) and only 12 base avatar models with 4 cached shapes per gender.
- Learned ZSC agents do not capture human-robot interaction dynamics well, as humans were more reactive and success was 1 across human episodes.
Experimental setup
Datasets: Habitat Synthetic Scenes Dataset (HSSD) AMASS SMPL-X body models
Benchmarks: Habitat 3.0 social navigation Habitat 3.0 social rearrangement human-in-the-loop study (30 participants)
Metrics: success rate SPS following rate collision rate relative efficiency frames per second
Baselines: Heuristic Expert Learn-Single Plan-Pop Learn-Pop Solo
Builds on: Habitat DD-PPO SMPL-X VPoser population play
Closest papers by meaning
- LocoVR: Multiuser Indoor Locomotion Dataset in Virtual Reality (ICLR 2025 · similarity 0.87)
- PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks (ICLR 2025 · similarity 0.87)
- CooHOI: Learning Cooperative Human-Object Interaction with Manipulated Object Dynamics (NeurIPS 2024 · similarity 0.86)
- LabUtopia: High-Fidelity Simulation and Hierarchical Benchmark for Scientific Embodied Agents (NeurIPS 2025 · similarity 0.86)
- Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with Dynamic Human Interactions (NeurIPS 2024 · similarity 0.86)
- Embodied Web Agents: Bridging Physical-Digital Realms for Integrated Agent Intelligence (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.