Research map / 3D vision / 3D generation
HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reconstructs an articulated hand and an unknown object jointly from a monocular interaction video using a compositional neural implicit SDF model refined with hand-object contact constraints, without 3D hand-object supervision.
Problem
Category-agnostic 3D reconstruction of articulated hands and held objects from monocular RGB video, without object templates or 3D hand-object annotations.
Approach
A compositional articulated implicit model (HOLD-Net) with separate canonical SDF/texture networks for the MANO-driven hand and the rigid object, plus a background NeRF++-style network, trained with 2D losses and refined via contact and mask-alignment pose optimization before final training.
Key results
- On HO3D, HOLD reaches 24.2 mm MPJPE, 0.4 cm² CD and 96.5% F10, outperforming iHOI and DiffHOI which use 3D supervision.
- On unseen object categories, DiffHOI's CD rises to 6.5 cm² while HOLD stays at 0.3 cm² CD.
- Versus Hampali et al. in-hand scanning, HOLD improves CD from 1.4 to 0.5 cm² and F5 from 57.4% to 84.3%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add joint hand modeling to the object reconstruction (ablation) | HOLD · HO3D-v3 | CD [cm2]: 0.41 -> 0.38 | Helped |
| Add contact-based hand-object pose refinement (ablation) | HOLD · HO3D-v3 | CDh [cm2]: 122.1 -> 11.3 | Helped |
Surprising findings
- The self-supervised method without any 3D hand-object annotations outperforms fully supervised baselines iHOI and DiffHOI.
Limitations
- Reconstruction of thin or textureless objects is limited by detector-based SfM pose initialization.
- Reliance on RGB supervision may hinder reconstruction of rarely observed object regions.
- Training time is long per sequence.
Experimental setup
Datasets: HO3D-v3 HOLD dataset (self-captured in-the-wild videos)
Benchmarks: HO3D-v3 HOLD dataset
Metrics: MPJPE Chamfer distance F-score hand-relative Chamfer distance
Baselines: HOMan iHOI DiffHOI Hampali et al. 2022 in-hand scanning
Builds on: NeuS NeRF++ MANO HLoc Hampali et al. in-hand scanning
Closest papers by meaning
- BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splatting (CVPR 2025 · similarity 0.92)
- EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild (CVPR 2025 · similarity 0.91)
- Hand-held Object Reconstruction from RGB Video with Dynamic Interaction (CVPR 2025 · similarity 0.91)
- Learning Explicit Contact for Implicit Reconstruction of Hand-Held Objects from Monocular Images (AAAI 2024 · similarity 0.91)
- HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions (CVPR 2024 · similarity 0.90)
- HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.