Research map / 3D vision / 3D generation
In-Hand 3D Object Reconstruction from a Monocular RGB Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reconstructs a hand-held object from a monocular RGB video by combining NeuS-style SDF rendering with amodal mask completion (refined via a semantic head) and hand-object contact (penetration/attraction) losses.
Problem
In-hand 3D object reconstruction from a monocular RGB video suffers from hand occlusion, which leaves the hand-object contact region poorly reconstructed by implicit neural rendering methods.
Approach
Adds a 2D amodal mask completion network (hourglass over hand and object segmentation maps) whose masks are refined jointly with the SDF via a rendered semantic head for multi-view consistency, plus differentiable penetration and attraction losses on the hand-object contact region.
Key results
- On HO3D, CD of 0.282 vs 0.591 for HHOR with post-processing (52% improvement).
- On HOD, CD of 0.277 vs 0.347 for HHOR with post-processing (about 20% improvement).
- Ablation on HO3D: adding amodal masks cuts CD from 1.094 to 0.448; contact constraints and mask refinement bring it to 0.282.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add amodal mask supervision (ablation) | NeuS · HO3D | CD: 1.094 -> 0.448 | Helped |
| Add contact constraints (ablation) | NeuS with amodal masks · HO3D | CD: 0.448 -> 0.333 | Helped |
| Add mask refinement (ablation) | NeuS with amodal masks and contact constraints · HO3D | CD: 0.333 -> 0.282 | Helped |
| Add attraction loss (ablation) | Ours · HO3D | CD: 0.376 -> 0.282 | Helped |
| Add attraction loss (ablation) | Ours · HO3D | Vol: 0.108 -> 0.327 | Hurt |
| Add penetration loss (ablation) | Ours · HO3D | CD: 0.319 -> 0.282 | Helped |
| Add penetration loss (ablation) | Ours · HO3D | Vol: 0.614 -> 0.327 | Helped |
| Replace contact-only attraction with the proposed attraction loss covering nearby non-contact regions (ablation) | Ours · HO3D | CD: 0.341 -> 0.282 | Helped |
| Replace contact-only attraction with the proposed attraction loss covering nearby non-contact regions (ablation) | Ours · HO3D | Vol: 0.439 -> 0.327 | Helped |
| Add mask refinement to the amodal completion predictions (ablation) | Amodal completion network · HO3D | mIOU%: 98.15 -> 99.02; 93.15 -> 94.37; 92.84 -> 94.28; 92.51 -> 95.17; 95.29 -> 96.52 | Helped |
| Use Ours instead of HHOR** | Object reconstruction · HO3D | CD: 0.591 -> 0.282; 52% | Helped |
| Use Ours instead of HHOR** | Object reconstruction · HOD | CD: 20% | Helped |
Surprising findings
- Penetration loss alone minimizes intersection volume but gives worse reconstruction quality than the full method.
Limitations
- Training is slow (about 14 hours per sequence), which the authors plan to address.
- Assumes the object is firmly grasped so object pose is fixed relative to the hand.
- Only 14 of 35 HOD objects have ground-truth meshes for evaluation.
Experimental setup
Datasets: HO3D HOD ObMan
Benchmarks: HO3D HOD
Metrics: Chamfer Distance Intersection Volume IoU PA-MPJPE PA-MPVPE ATE
Baselines: HHOR NeuS IHOI
Builds on: NeuS HHOR MANO HandOccNet
Closest papers by meaning
- Hand-held Object Reconstruction from RGB Video with Dynamic Interaction (CVPR 2025 · similarity 0.90)
- Learning Explicit Contact for Implicit Reconstruction of Hand-Held Objects from Monocular Images (AAAI 2024 · similarity 0.89)
- O^2-Recon: Completing 3D Reconstruction of Occluded Objects in the Scene with a Pre-trained 2D Diffusion Model (AAAI 2024 · similarity 0.88)
- Free-Moving Object Reconstruction and Pose Estimation with Virtual Camera (AAAI 2025 · similarity 0.87)
- SiMA-Hand: Boosting 3D Hand-Mesh Reconstruction by Single-to-Multi-View Adaptation (AAAI 2024 · similarity 0.86)
- HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.