Research map / 3D vision / Face and shape
Learning Explicit Contact for Implicit Reconstruction of Hand-Held Objects from Monocular Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Predicts 3D hand-object contact probabilities from a single RGB image with coarse-to-fine graph-based transformers, then diffuses them via sparse convolutions into a contact code volume that conditions an implicit SDF decoder for hand-held object reconstruction.
Problem
Reconstructing hand-held objects from monocular RGB images is hard because the 3D geometry is ambiguous; prior implicit methods ignore explicit hand-object contact, producing less realistic object meshes.
Approach
Explicitly predict part- and vertex-level contact probabilities on the MANO hand mesh with cascaded graph-based transformers, then diffuse these discrete contact states into the 3D volume with sparse convolutions so an implicit SDF decoder can query contact features at any point.
Key results
- On HO3D with ground-truth hands, F@5mm improves from 0.361/0.351 (IHOI) to 0.393 and chamfer distance drops to 0.646 mm versus IHOI's 0.656 mm.
- On OakInk with predicted hands, F@5mm rises to 0.459 versus IHOI's 0.432 and chamfer distance falls from 0.491 to 0.260.
- Multi-level vertex contact prediction on OakInk reaches F1 0.262 versus single-vertex baseline; part-level contact F1 on OakInk is 0.747 versus 0.728 single-part.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add vertex sub-mesh loss (ablation) | OakInk vertex-level contact prediction · OakInk | F1: 0.189 -> 0.210 | Helped |
| predict contact only, without reconstructing the hand mesh (ablation) | OakInk vertex-level contact prediction · OakInk | precision: 0.285 -> 0.309 | Helped |
| use weighted BCE loss (ablation) | OakInk vertex-level contact prediction · OakInk | F1: 0.213 -> 0.262 | Helped |
| use multi-level rather than single-level vertex prediction (ablation) | HO3D contact prediction · HO3D | F1: 0.416 -> 0.436 | Helped |
| use multi-level rather than single-level part prediction (ablation) | Part-level contact prediction · HO3D | F1: 0.672 -> 0.685 | Helped |
| use sparse convolution instead of nearest-neighbor diffusion (ablation) | HO3D object reconstruction · HO3D | F@5mm: 0.261 -> 0.393 | Helped |
| replace estimated contacts with an all-one contact vector (ablation) | HO3D object reconstruction · HO3D | F@5mm: 0.393 -> 0.361 | Hurt |
| use four contact code volume scales instead of only V3 + V4 (ablation) | HO3D object reconstruction · HO3D | F@5mm: 0.371 -> 0.393 | Helped |
| add full multi-scale contact code volumes (ablation) | HO3D object reconstruction · HO3D | F@5mm: 0.327 -> 0.348 -> 0.386 -> 0.393 | Helped |
| use estimated contact features instead of an all-one contact vector (ablation) | Object reconstruction · HO3D | F@5mm: 0.361 -> 0.393 | Helped |
| use estimated contact features instead of an all-one contact vector (ablation) | Object reconstruction · OakInk | F@10mm: 0.651 -> 0.716 | Helped |
| use part-level rather than vertex-level predicted contacts (ablation) | Object reconstruction · HO3D | F@5mm: 0.374 -> 0.393 | Helped |
Surprising findings
- Vertex-level contact prediction is far less accurate than part-level prediction (F1 0.262 vs 0.728 on OakInk), so the final method uses part-level predictions mapped to vertices.
- Naive nearest-neighbor diffusion with four scales performs worse than the IHOI baseline without contacts (F@5mm 0.261 vs 0.351 on HO3D).
Limitations
- Currently focuses only on hand-held object reconstruction without integrating hand reconstruction.
- Vertex-level contact predictions are less accurate than part-level ones.
- Ground-truth contact annotations on OakInk are noisy, limiting gains from ground-truth contacts.
Experimental setup
Datasets: OakInk HO3D
Benchmarks: HO3D OakInk
Metrics: precision recall F1 F@5mm F@10mm chamfer distance penetration depth intersection volume
Baselines: IHOI HO GF BSTRO
Builds on: Graphormer IHOI Neural Body BSTRO DeepSDF
Closest papers by meaning
- HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video (CVPR 2024 · similarity 0.91)
- Hand-held Object Reconstruction from RGB Video with Dynamic Interaction (CVPR 2025 · similarity 0.89)
- In-Hand 3D Object Reconstruction from a Monocular RGB Video (AAAI 2024 · similarity 0.89)
- SRHand: Super-Resolving Hand Images and 3D Shapes via View/Pose-aware Neural Image Representations and Explicit Meshes (NeurIPS 2025 · similarity 0.89)
- Single-view Image to Novel-view Generation for Hand-Object Interactions (AAAI 2025 · similarity 0.89)
- EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.