Research map / Vision-language and multimodal / Robot action and navigation
OpenHOI: Open-World Hand-Object Interaction Synthesis with Multimodal Large Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OpenHOI, which fine-tunes a 3D MLLM (ShapeLLM-7B) for affordance grounding and instruction decomposition, then uses an affordance-conditioned diffusion model with training-free physics refinement to synthesize long-horizon hand-object interactions for unseen objects.
Problem
Generating 3D hand-object interaction sequences from open-vocabulary instructions for unseen objects; existing methods work on closed-set objects and predefined tasks and lack physical plausibility and long-horizon coherence.
Approach
Couples a 3D MLLM that outputs multi-step affordance masks via a segmentation token with an affordance- and sub-task-conditioned diffusion model, plus a training-free spherical-Gaussian-constrained loss guidance for affordance alignment, penetration avoidance and sub-sequence transition smoothness.
Key results
- On GRAB unseen split, OpenHOI reports MPJPE 51.34 and FID 28.29 versus Text2HOI's 60.67 and 36.96.
- On ARCTIC seen split, OpenHOI reports MPJPE 45.15 and FID 19.74 versus Text2HOI's 52.16 and 23.35.
- On the H2O extreme-case transfer test, OpenHOI reports MPJPE 75.78 on GRAB-trained models versus 80.25 for the best baseline Text2HOI.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OpenHOI instead of Text2HOI | HOI synthesis · GRAB seen | MPJPE: 56.29$\pm$2.13 -> 47.64$\pm$1.03 | Helped |
| Use OpenHOI instead of Text2HOI | HOI synthesis · GRAB unseen | FID: 36.96$\pm$0.77 -> 28.29$\pm$0.62 | Helped |
| Use OpenHOI instead of Text2HOI | HOI synthesis · ARCTIC seen | FOL: 0.33$\pm$0.01 -> 0.25$\pm$0.04 | Helped |
| Use OpenHOI instead of Text2HOI | HOI synthesis · ARCTIC unseen | MPJPE: 57.83$\pm$1.61 -> 47.25$\pm$0.39 | Helped |
| Add MLLM-based affordance maps (ablation) | OpenHOI · GRAB seen | MPJPE: 56.04 $\pm$ 1.22 -> 47.64$\pm$1.03 | Helped |
| Add classifier-free guidance (ablation) | OpenHOI · GRAB unseen | FID: 37.05 $\pm$ 1.38 -> 28.29$\pm$0.62 | Helped |
| Add penetration loss-guided refinement (ablation) | OpenHOI · GRAB seen | FOL: 0.42 $\pm$ 0.01 -> 0.26$\pm$0.02 | Helped |
| Add affordance loss-guided refinement (ablation) | OpenHOI · GRAB seen | FOL: 0.43 $\pm$ 0.02 -> 0.26$\pm$0.02 | Helped |
| Add physical refinement (ablation) | OpenHOI · Seen | Physical realism: 0.89$\pm$0.07 -> 0.93$\pm$0.02 | Helped |
| Add physical refinement (ablation) | OpenHOI · Unseen | IV: 12.27$\pm$0.48 -> 10.35$\pm$0.82 | Helped |
| Use coarse-grained affordance tuning instead of no fine-tuning (ablation) | 3D MLLM · Affordance dataset | AUC: 68.77 -> 84.65 | Helped |
| Add fine-grained HOI tuning after coarse-grained tuning (ablation) | 3D MLLM · Affordance dataset | AUC: 84.65 -> 87.02 | Helped |
Limitations
- Performance degrades after more than three consecutive actions (over 450 frames) due to error accumulation.
- Cannot resolve specific-instance references (e.g., 'the second cabinet') because of limited 3D QA training.
- Fine-grained dynamics such as fluid pouring and compositional long-horizon tasks like cooking remain challenging.
Experimental setup
Datasets: GRAB ARCTIC H2O
Benchmarks: GRAB ARCTIC H2O
Metrics: MPJPE FOL FID Diversity MModality Physical realism IV
Baselines: MDM TM2T MotionGPT Text2HOI
Builds on: ShapeLLM LISA PixelLM DSG classifier-free guidance
Closest papers by meaning
- FlowHOI: Flow-based Semantics-Grounded Generation of Hand-Object Interactions for Dexterous Robot Manipulation (NeurIPS 2026 · similarity 0.89)
- InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsing (CVPR 2025 · similarity 0.88)
- HOIGPT: Learning Long-Sequence Hand-Object Interaction with Language Models (CVPR 2025 · similarity 0.88)
- LatentHOI: On the Generalizable Hand Object Motion Generation with Latent Hand Diffusion. (CVPR 2025 · similarity 0.88)
- OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints (CVPR 2025 · similarity 0.88)
- AffordBot: 3D Fine-grained Embodied Reasoning via Multimodal Large Language Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.