Research map / 3D vision / 3D generation
EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reconstructs hand-object interactions from a single in-the-wild image by chaining off-the-shelf large models (LISA, diffusion inpainting, SAM, InstantMesh, HaMeR) with a three-stage prior-guided optimization for physically plausible contact.
Problem
Single-view hand-object interaction reconstruction is ill-posed: occlusions, diverse hand poses, and unknown object shapes make joint object and hand recovery inaccurate and physically implausible, and prior methods need templates or lack generalization.
Approach
Combine foundation models for segmentation, inpainting, object and hand reconstruction with a prior-guided optimizer that aligns cameras via Sinkhorn+IoU mask loss, aligns hand-object contact via ray-cast contact points and ICP, then refines hand pose with penetration, contact, and regularization losses.
Key results
- Object reconstruction: lowest Chamfer distance on Arctic (0.155 vs 0.072 for MOHO is not lowest; reported ours achieves lowest simulation displacement and intersection volume across datasets, e.g. Arctic S.D. 2.25 cm and I.V. 4.67 cm3).
- Hand accuracy: with predicted object, MPVPE/MPJPE are slightly worse than HaMeR (e.g. OakInk 1.19 vs 1.06 cm MPVPE); with GT object geometry, optimization improves HaMeR (OakInk MPVPE 0.86 cm).
- Ablation: full loss set gives best simulation displacement (3.08) and intersection volume (4.11) on OakInk; removing L_reg raises MPVPE to 4.57 cm.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add hand-mask loss Lhand-mask during hand parameter refinement (ablation) | EasyHOI · OakInk | Simulation Displacement (cm): 3.21 -> 3.08 | Helped |
| add penetration loss Lpenetr during hand parameter refinement (ablation) | EasyHOI · OakInk | Intersection Volume (cm³): 9.62 -> 4.11 | Helped |
| add regularization loss Lreg during hand parameter refinement (ablation) | EasyHOI · OakInk | MPVPE (cm): 4.57 -> 1.19 | Helped |
| add contact loss Lcontact during hand parameter refinement (ablation) | EasyHOI · OakInk | Simulation Displacement (cm): 3.94 -> 3.08 | Helped |
| perform camera system setup, then HOI contact alignment and hand parameter refinement (ablation) | EasyHOI · Arctic, OakInk, and DexYCB | Simulation Displacement (cm): 5.79 -> 2.25; 5.81 -> 3.08; 6.32 -> 2.43 | Helped |
Surprising findings
- Object Chamfer distance for the proposed method is not the lowest on every dataset: MOHO attains lower Chamfer distance on Arctic (0.072 F5 vs ours 0.155 is reported with different metric columns), so gains are mainly in stability and interpenetration metrics.
- Hand pose accuracy is slightly worse than HaMeR alone when using the predicted object, because imperfect LRM object geometry compromises hand optimization.
Limitations
- Performance depends heavily on the adopted large models; failures in them propagate to the output.
- Object geometry from the LRM is imperfect, which limits hand accuracy after optimization.
- Single-view ambiguity in depth and scale is only partly addressed by the optimization.
Experimental setup
Datasets: Arctic OakInk DexYCB
Benchmarks: Arctic OakInk DexYCB
Metrics: Chamfer distance F-score F5 F-score F10 simulation displacement intersection volume MPVPE MPJPE
Baselines: IHOI AlignSDF gSDF MOHO HaMeR
Builds on: InstantMesh HaMeR LISA MANO Sinkhorn distance
Closest papers by meaning
- HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video (CVPR 2024 · similarity 0.91)
- HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions (CVPR 2024 · similarity 0.89)
- BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splatting (CVPR 2025 · similarity 0.89)
- Learning Explicit Contact for Implicit Reconstruction of Hand-Held Objects from Monocular Images (AAAI 2024 · similarity 0.89)
- UniHOPE: A Unified Approach for Hand-Only and Hand-Object Pose Estimation (CVPR 2025 · similarity 0.88)
- SiMHand: Mining Similar Hands for Large-Scale 3D Hand Pose Pre-training (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.