Research map / Vision-language and multimodal / Human and embodied modeling
InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates text-conditioned 3D dynamic human-object interactions without paired text-interaction data by combining LLM high-level planning, a text-to-motion model, rule-based contact-map interaction retrieval, and a vertex-conditioned world model trained on BEHAVE.
Problem
Text-guided 3D whole-body human-object interaction generation is hard because large-scale text-interaction paired data is lacking; the paper asks what can be achieved without any text supervision for HOI.
Approach
Decouple interaction semantics (handled by pretrained LLM planning and a pretrained text-to-motion model plus retrieval of initial object pose) from low-level dynamics (learned by a world model from motion-capture HOI data, conditioned on contact vertices of the human), with an optimization step refining the rollout.
Key results
- Adding high-level planning improves MotionGPT on BEHAVE from FID 1.062 to 1.337 (planning) with Top-1 R-Precision 0.234 vs 0.233; planning consistently improves FID/R-Precision across MDM, MotionDiffuse, ReMoDiffuse, MotionGPT on BEHAVE and OMOMO.
- Vertex-based contact-vertex actions in the dynamics model reduce CMD to 0.151 versus 0.219 for InterDiff contact-marker actions and 0.424 without action.
- Planning raises text similarity to HumanML3D on out-of-distribution descriptions from 0.838 to 0.927.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use contact vertices as actions instead of no action (ablation) | InterDreamer dynamics model · Text-to-Interaction | CMD: 0.424 -> 0.151 | Helped |
| use contact vertices as actions instead of human motion as actions (ablation) | InterDreamer dynamics model · Text-to-Interaction | Pene. (10−2%): 957 -> 443 | Helped |
| add high-level planning (ablation) | MotionGPT on BEHAVE · BEHAVE | FID: 5.497±0.106 -> 4.751±0.121 | Helped |
| add high-level planning (ablation) | MotionGPT on BEHAVE · BEHAVE | R-Precision Top 1: 0.233±0.003 -> 0.234±0.004 | No clear effect |
| add high-level planning (ablation) | MotionGPT on OMOMO · OMOMO | FID: 18.472±0.528 -> 17.512±0.498 | Helped |
| add high-level planning (ablation) | MotionGPT on OMOMO · OMOMO | R-Precision Top 1: 0.061±0.005 -> 0.064±0.007 | Helped |
| add high-level planning (ablation) | text descriptions processed for HumanML3D similarity · Text similarity to HumanML3D (average) | Similarity: 0.913 -> 0.932 | Helped |
| add high-level planning (ablation) | out-of-distribution text descriptions · Text similarity to HumanML3D (out-of-distribution) | Similarity: 0.838 -> 0.927 | Helped |
Surprising findings
- Human motion as action performs worse than no action on several interaction metrics in the dynamics ablation (CMD 0.325 vs 0.424 for no action is better, but penetration is worse at 957 vs 533).
Limitations
- Results may be physically implausible with artifacts such as foot skating.
- Hand poses are rough because hand articulation is missing from the dataset.
- Dynamics modeling is simple and could be enhanced; no physics simulator is used.
Experimental setup
Datasets: BEHAVE OMOMO CHAIRS HumanML3D
Benchmarks: BEHAVE OMOMO CHAIRS
Metrics: FID R-Precision MM Dist Multimodality Diversity CMD penetration translation error rotation error
Baselines: MDM MotionDiffuse ReMoDiffuse MotionGPT InterDiff
Builds on: MotionGPT InterDiff MDM ControlNet
Closest papers by meaning
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.92)
- CG-HOI: Contact-Guided 3D Human-Object Interaction Generation (CVPR 2024 · similarity 0.92)
- Auto-Regressive Diffusion for Generating 3D Human-Object Interactions (AAAI 2025 · similarity 0.92)
- HOI-Dyn: Learning Interaction Dynamics for Human-Object Motion Diffusion (NeurIPS 2025 · similarity 0.91)
- HOIAnimator: Generating Text-prompt Human-object Animations using Novel Perceptive Diffusion Models (CVPR 2024 · similarity 0.91)
- InterMask: 3D Human Interaction Generation via Collaborative Masked Modeling (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.