Research map / Large language models / Agents, code and math
Unified Human-Scene Interaction via Prompted Chain-of-Contacts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniHSI, which uses an LLM planner to convert language commands into Chain-of-Contacts plans and a unified AMP-based physics controller to execute them, trained on a new ScenePlan dataset.
Problem
Human-scene interaction (HSI) control is typically limited to specific interactions with separate policies per task and relies on annotated motion data; a user-friendly, versatile, language-driven interface is lacking.
Approach
Defines interactions as Chain of Contacts (ordered steps of human joint-object part contact pairs), generated by prompting an LLM planner from language commands, and executes all of them with one unified AMP-based controller via a TaskParser that converts contacts into uniform observations and rewards, without interaction annotations.
Key results
- On ScenePlan simple plans UniHSI reaches 85-90% success range for simple tasks while the PartNet reference reaches 85.5%; success falls to 40.5% on hard plans for the PartNet reference (UniHSI hard-level figures are reported in Table 2 of the paper).
- Sit task success improves from 77.3% (AMP) to 94.3% and Lie Down from 21.3% (AMP) to 81.5%, with Reach at 97.5%.
- GPT-4 as LLM planner achieves 57.3% execution success rate versus 35.6% for GPT-3.5 and 73.2% for humans.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Adaptive Weights to the controller (ablation) | UniHSI · ScenePlan PartNet | Success Rate (%): 21.2 -> 85.5 (Simple); 5.3 -> 67.9 (Mid); 0.1 -> 40.5 (Hard) | Helped |
| Add Ego-centric Heightmap (ablation) | UniHSI · ScenePlan PartNet | Success Rate (%): 61.6 -> 85.5 (Simple); 45.7 -> 67.9 (Mid); 0.0 -> 40.5 (Hard) | Helped |
| Use UniHSI's unified task representation and controller instead of vanilla task combination | AMP · Sit | Success Rate (%): 62.5 -> 94.3 | Helped |
| Use UniHSI's unified task representation and controller instead of vanilla task combination | AMP · Lie Down | Success Rate (%): 20.1 -> 81.5 | Helped |
| Use UniHSI's unified task representation and controller instead of vanilla task combination | AMP · Reach | Success Rate (%): 90.3 -> 97.5 | Helped |
| Use GPT-4 instead of GPT-3.5 for planning | LLM Planner · Language-command task planning | Execution Success Rate (ESR, %): 35.6 -> 57.3 | Helped |
| Replace GPT-3.5 with human planning | LLM Planner · Language-command task planning | Execution Success Rate (ESR, %): 35.6 -> 73.2 | Helped |
Surprising findings
- Naively combining tasks in one AMP policy (vanilla combination) causes a large performance drop, e.g. Lie Down falls to 20.1%.
Limitations
- Only controls humanoids interacting with fixed objects; movable objects are not handled.
- LLM is not integrated into training; plans are pre-generated offline.
- LLM planners sometimes produce spatially wrong or out-of-distribution plans.
Experimental setup
Datasets: ScenePlan PartNet ScanNet SAMP CIRCLE
Benchmarks: ScenePlan ScanNet scenes single-task sit/lie down/reach comparisons
Metrics: success rate contact error success steps planning correctness naturalness score
Baselines: AMP NSM SAMP InterPhys AMP-Vanilla Combination PartNet scenes ScanNet scenes
Builds on: AMP InterPhys PPO IsaacGym
Closest papers by meaning
- HSI-GPT: A General-Purpose Large Scene-Motion-Language Model for Human Scene Interaction (CVPR 2025 · similarity 0.89)
- TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenization (CVPR 2025 · similarity 0.88)
- OpenHOI: Open-World Hand-Object Interaction Synthesis with Multimodal Large Language Model (NeurIPS 2025 · similarity 0.86)
- Scaling Up Dynamic Human-Scene Interaction Modeling (CVPR 2024 · similarity 0.86)
- CG-HOI: Contact-Guided 3D Human-Object Interaction Generation (CVPR 2024 · similarity 0.85)
- InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsing (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.