Research map / 3D vision / 3D generation
Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Text2HOI generates 3D hand-object interaction sequences from a text prompt and canonical object mesh by predicting a text-conditioned contact map with a VAE and denoising motion with a Transformer diffusion model, followed by a hand refiner.
Problem
Generating physically plausible 3D hand-object interaction sequences from text is hard because existing hand-object motion datasets are small and lack diversity in interaction types and object categories.
Approach
Decompose text-to-HOI generation into text-guided, scale-aware contact map prediction on object point clouds (category-agnostic local geometry) followed by a Transformer diffusion model conditioned on the contact map, plus a feed-forward hand refiner that reduces penetration and improves contact.
Key results
- On GRAB, Text2HOI reaches top-3 accuracy 0.9218 and FID 0.3017, versus MDM at 0.5127 and 0.6023.
- On H2O, physical realism is 0.7574 for Ours versus 0.5572 for MDM (baseline).
- Ablations on GRAB show removing the hand refiner drops physical realism from 0.8839 to 0.8312, and removing the contact loss drops it to 0.1467.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add frame-wise and agent-wise positional encoding (ablation) | Text2HOI without refiner · GRAB | Physical realism: 0.8024 ± 0.0007 -> 0.8312 ± 0.0005 | Helped |
| add agent-wise positional encoding (ablation) | Text2HOI without refiner · GRAB | Physical realism: 0.8221 ± 0.0009 -> 0.8312 ± 0.0005 | Helped |
| add distance map loss and relative orientation loss (ablation) | Text2HOI without refiner · GRAB | Physical realism: 0.7490 ± 0.0006 -> 0.8312 ± 0.0005 | Helped |
| add relative orientation loss (ablation) | Text2HOI without refiner · GRAB | Physical realism: 0.8247 ± 0.0011 -> 0.8312 ± 0.0005 | Helped |
| add distance map loss (ablation) | Text2HOI without refiner · GRAB | Physical realism: 0.6410 ± 0.0010 -> 0.8312 ± 0.0005 | Helped |
| add contact map and object scale conditions (ablation) | Text2HOI without refiner · GRAB | Physical realism: 0.7815 ± 0.0006 -> 0.8312 ± 0.0005 | Helped |
| add object scale condition (ablation) | Text2HOI without refiner · GRAB | Physical realism: 0.8018 ± 0.0005 -> 0.8312 ± 0.0005 | Helped |
| add contact map condition (ablation) | Text2HOI without refiner · GRAB | Physical realism: 0.7892 ± 0.0009 -> 0.8312 ± 0.0005 | Helped |
| add hand refinement module (ablation) | Text2HOI · GRAB | Physical realism: 0.8312 ± 0.0005 -> 0.8839 ± 0.0005 | Helped |
| add penetration loss and contact loss (ablation) | Text2HOI hand refiner · GRAB | Physical realism: 0.6249 ± 0.0008 -> 0.8839 ± 0.0005 | Helped |
| add contact loss (ablation) | Text2HOI hand refiner · GRAB | Physical realism: 0.1467 ± 0.0005 -> 0.8839 ± 0.0005 | Helped |
| add penetration loss (ablation) | Text2HOI hand refiner · GRAB | Physical realism: 0.8782 ± 0.0006 -> 0.8839 ± 0.0005 | Helped |
Surprising findings
- Removing the contact loss L_contact collapses physical realism on GRAB to 0.1467, far below removing other losses.
Limitations
- Physical realism is judged by a learned physics model following ManipNet, not by real-world simulation.
- Text labels for H2O and GRAB are auto-generated from action labels, limiting prompt diversity.
Experimental setup
Datasets: H2O GRAB ARCTIC
Benchmarks: H2O GRAB ARCTIC
Metrics: accuracy (top-3) FID diversity multimodality physical realism
Baselines: T2M MDM IMOS
Builds on: Motion Diffusion Model (MDM) IMOS CLIP ManipNet GrabNet-style contact prediction
Closest papers by meaning
- How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactions (CVPR 2025 · similarity 0.92)
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.91)
- CG-HOI: Contact-Guided 3D Human-Object Interaction Generation (CVPR 2024 · similarity 0.91)
- BOTH2Hands: Inferring 3D Hands from Both Text Prompts and Body Dynamics (CVPR 2024 · similarity 0.90)
- InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsing (CVPR 2025 · similarity 0.90)
- HOIGPT: Learning Long-Sequence Hand-Object Interaction with Language Models (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.