Research map / 3D vision / 3D generation
InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InteractAnything, a zero-shot text-driven 3D human-object interaction synthesis framework that uses LLM feedback for relation initialization, 2D diffusion inpainting for open-set object affordance parsing, and multi-view SDS plus force-closure optimization for fine-grained contacts.
Problem
Generating novel 3D human-object interactions from text for open-set objects is hard because it requires precise human-object relation reasoning, affordance parsing for arbitrary objects, and detailed pose synthesis aligned with object geometry.
Approach
Uses an LLM to select relations from predefined options for initialization, derives object contact affordance from 2D inpainting masks and OpenPose-based distance probabilities projected to 3D, and optimizes SMPL-H pose with multi-view SDS, spatial constraints, and a SMPL-based force-closure loss, without training on HOI datasets.
Key results
- CLIP score 0.2968 for Ours vs 0.2757 for DreamHOI (Table 1).
- GPT-4V selection Contact score 52.1 for Ours vs 19.5 for DreamHOI (Table 2).
- Removing LLM-guided initialization drops GPT-4V Overall selection from 45.6 to 26.1 (Table 3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use InteractAnything instead of DreamHOI | HOI synthesis · HOI evaluation prompts | CLIP similarity: 0.2757 -> 0.2968 | Helped |
| Use InteractAnything instead of DreamHOI | HOI synthesis · HOI evaluation outputs | GPT-4V selection: Overall: 26.0 -> 45.6 | Helped |
| Use InteractAnything instead of DreamHOI | HOI synthesis · HOI evaluation outputs | GPT-4V selection: Contact: 19.5 -> 52.1 | Helped |
| Add fine-grained HOI refinement (ablation) | InteractAnything · HOI evaluation outputs | GPT-4V selection: Overall: 39.1 -> 45.6 | Helped |
| Add LLM-guided initialization instead of randomly initializing human and object positions (ablation) | InteractAnything · HOI evaluation outputs | GPT-4V selection: Overall: 26.1 -> 45.6 | Helped |
| Add fine-grained optimization terms (ablation) | InteractAnything · grasping interaction with a table | contact plausibility | Helped |
Limitations
- Relies on 2D priors as primary guidance, lacking multi-view consistency.
- Uses SMPL-H, limiting adaptation to non-human agents.
- Physical plausibility only enforced via force closure, not simulator-verified.
Experimental setup
Datasets: BEHAVE PartNet
Benchmarks: CLIP score on 5 prompts and 10 object categories GPT-4V selection
Metrics: CLIP similarity score GPT-4V selection rate (Overall, Contact)
Baselines: DreamFusion Magic3D DreamHOI DreamFusion* InterFusion
Builds on: DreamFusion DreamHOI InterFusion Magic3D SMPL-H
Closest papers by meaning
- CG-HOI: Contact-Guided 3D Human-Object Interaction Generation (CVPR 2024 · similarity 0.91)
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.91)
- GenZI: Zero-Shot 3D Human-Scene Interaction Generation (CVPR 2024 · similarity 0.90)
- InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction (NeurIPS 2024 · similarity 0.90)
- Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction (CVPR 2024 · similarity 0.90)
- InteractVLM: 3D Interaction Reasoning from 2D Foundational Models (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.