Research map / Image generation / Autoregressive and latent generation
LatentHOI: On the Generalizable Hand Object Motion Generation with Latent Hand Diffusion.
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LatentHOI, a latent diffusion model over a GraspVAE latent space that decouples global hand-object motion from per-frame grasp poses to generalize hand-object motion generation to unseen objects.
Problem
Generating 3D hand-object interaction motion for objects unseen during training; direct diffusion over hand-object sequences overfits on small HOI datasets, causing penetration, tracking loss and implausible grasps.
Approach
Factorize P(H,O|C) into a latent diffusion model that generates object/hand-wrist trajectories and a per-frame latent grasp code, with a GraspVAE decoding fine-grained hand pose conditioned on object point cloud and wrist position; diffusion is trained in the regularized VAE latent space with a shared noise sample across frames.
Key results
- On unseen GRAB objects, right-hand interpenetration volume drops to 6.38 cm3 vs 9.12 for MDM and 9.62 for MLD, with contact ratio 11.94% vs 8.21% for MDM.
- On unseen OakInk objects, LatentHOI achieves IVU 0.14 vs 0.20 for MDM and physical plausibility 71.24 vs 60.89 for MDM.
- On DexYCB, LatentHOI improves contact ratio to 11.98 vs 8.87 for MDM with comparable penetration (IV 7.70 vs 7.78).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LatentHOI instead of MDM | HOI generation · GRAB unseen objects | IVU: 0.19 -> 0.10 | Helped |
| Use LatentHOI instead of MDM | HOI generation · GRAB unseen objects | Phy: 89.81 -> 96.16 | Helped |
| Use LatentHOI instead of MDM | HOI generation · OakInk unseen objects | IVU: 0.20 -> 0.14 | Helped |
| Use LatentHOI instead of MDM | HOI generation · OakInk unseen objects | Phy: 60.89 -> 71.24 | Helped |
| Use LatentHOI instead of MDM | HOI generation · DexYCB unseen objects | IVU: 0.18 -> 0.13 | Helped |
| Use LatentHOI instead of MDM | HOI generation · DexYCB unseen objects | Phy: 86.22 -> 88.52 | Helped |
| Add random rotation augmentation during GraspVAE training (ablation) | LatentHOI · Ablation evaluation | IV: 4.17 -> 3.78 | Helped |
| Add random rotation augmentation during GraspVAE training (ablation) | LatentHOI · Ablation evaluation | IVU: 0.14 -> 0.10 | Helped |
| Train diffusion in the GraspVAE latent representation (ablation) | LatentHOI · Ablation evaluation | IV: 4.99 -> 3.78 | Helped |
| Train diffusion in the GraspVAE latent representation (ablation) | LatentHOI · Ablation evaluation | IVU: 0.13 -> 0.10 | Helped |
| Use shared η across frames in Zi = μi + σi · η (ablation) | LatentHOI · Ablation evaluation | IV: 4.51 -> 3.78 | Helped |
| Use shared η across frames in Zi = μi + σi · η (ablation) | LatentHOI · Ablation evaluation | IVU: 0.12 -> 0.10 | Helped |
Surprising findings
- On DexYCB, the authors report only small gains over MDM and attribute this to having only 16 training objects, with the user-study agreement (Cohen's kappa 0.127) being low.
Limitations
- Struggles to generalize to objects with sizes and geometries very different from training objects.
- Cannot generalize to motion intents absent from the training dataset.
- Small evaluation sets (e.g., 17 pairs on GRAB unseen split) and heuristic physics metric rather than simulation.
Experimental setup
Datasets: GRAB OakInk DexYCB
Benchmarks: GRAB OakInk DexYCB
Metrics: interpenetration volume interpenetration depth contact ratio interpenetration volume per contact unit physical plausibility sample diversity overall diversity user preference study
Baselines: MDM MLD IMoS Text2HOI
Builds on: MDM MLD CLIP MANO
Closest papers by meaning
- G-HOP: Generative Hand-Object Prior for Interaction Reconstruction and Grasp Synthesis (CVPR 2024 · similarity 0.90)
- Hand-Centric Motion Refinement for 3D Hand-Object Interaction via Hierarchical Spatial-Temporal Modeling (AAAI 2024 · similarity 0.90)
- How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactions (CVPR 2025 · similarity 0.90)
- MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generation (NeurIPS 2025 · similarity 0.90)
- GEARS: Local Geometry-aware Hand-object Interaction Synthesis (CVPR 2024 · similarity 0.89)
- DiffGrasp: Whole-Body Grasping Synthesis Guided by Object Motion Using a Diffusion Model (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.