Research map / 3D vision / 3D generation
FlowHOI: Flow-based Semantics-Grounded Generation of Hand-Object Interactions for Dexterous Robot Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FlowHOI, a two-stage conditional flow-matching model that generates semantically grounded hand-object interaction sequences (grasp then manipulation) from egocentric observation, text, and 3DGS scene context, with a reconstructed EgoDex HOI prior.
Problem
Generating long-horizon, contact-rich hand-object interaction sequences that are physically plausible, semantically grounded in language and 3D scene context, and fast enough for real-time robot planning, with scarce high-fidelity HOI supervision.
Approach
Decouples geometry-centric grasping from semantics-centric manipulation, each as a flow-matching model with x-prediction; the manipulation stage conditions on compact Perceiver-compressed hybrid 3D scene tokens and a TMR-style motion-text alignment loss, with grasp-prefix inpainting and hard transition constraint; grasping prior pretrained on reconstructed egocentric (EgoDex) HOI data.
Key results
- On GRAB, FlowHOI reaches 55.96% simulation success rate versus 33.03% for DiffH2O (1.7x), with the best action recognition accuracy among compared methods.
- Inference takes 0.16 s per sequence versus 6.34 s for DiffH2O and 3.57 s for LatentHOI, a claimed up to 40x speedup.
- On GRAB, IV is 10.93 cm^3 versus 13.11 for DiffH2O and 13.76 for LatentHOI, the lowest among generative methods.
Surprising findings
- LatentHOI achieves the highest heuristic Phy score on GRAB (96.40%) yet has much lower simulation success rate (28.44%), showing heuristic plausibility metrics do not capture dynamic stability.
- On HOT3D, FlowHOI's action recognition is highest but its physics success rate (5.33%) is below the real mocap reference's 6.67%, and its interpenetration volume is higher than LatentHOI.
Limitations
- Assumes accurate initial hand and object state estimation; degrades under heavy occlusion or unreliable reconstruction.
- Generated trajectories are kinematic; dynamics and compliance depend on downstream controllers.
- Failure cases include object size mismatch and incorrect contact establishment during grasping.
Experimental setup
Datasets: GRAB HOT3D EgoDex
Benchmarks: GRAB HOT3D
Metrics: interpenetration volume interpenetration depth contact ratio IVU action recognition accuracy sample diversity overall diversity physical plausibility success rate holding time
Baselines: DiffH2O LatentHOI Real Mocap
Builds on: flow matching DiffH2O TMR DiT Perceiver
Closest papers by meaning
- OpenHOI: Open-World Hand-Object Interaction Synthesis with Multimodal Large Language Model (NeurIPS 2025 · similarity 0.89)
- Scaffolding Dexterous Manipulation with Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- LatentHOI: On the Generalizable Hand Object Motion Generation with Latent Hand Diffusion. (CVPR 2025 · similarity 0.89)
- HOIGPT: Learning Long-Sequence Hand-Object Interaction with Language Models (CVPR 2025 · similarity 0.88)
- ForceVLA: Enhancing VLA Models with a Force-aware MoE for Contact-rich Manipulation (NeurIPS 2025 · similarity 0.88)
- HandEdit: A Unified Benchmark for Egocentric Human-to-Robot Dexterous Hand Image Editing (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.