Research map / 3D vision / Human pose estimation
Hand-held Object Reconstruction from RGB Video with Dynamic Interaction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reconstructs rigid hand-held objects (one or two hands) from monocular RGB video by initializing poses with a text-to-3D generated prior aligned via DINO semantic consistency, then jointly optimizing an SDF implicit representation and poses with a Sampson-distance pose outlier voting step.
Problem
Accurately reconstructing the 3D geometry of rigid objects manipulated dynamically by one or both hands from monocular RGB video, where SfM and hand priors fail on textureless, thin, or two-hand interaction cases.
Approach
Uses a generated 3D textured mesh (Genie, selected via OpenShape) as a prior for object pose initialization, aligning rendered and input frames with DINO semantic feature similarity in addition to masks; then jointly optimizes NeuS-style SDF and camera poses, and detects/reinitializes outlier poses using a Sampson-distance-based inter-view consistency vote.
Key results
- On HO3D single-hand, CD of 0.51 cm² and F5 of 78.2% vs HOLD's 1.82 cm² and 63.6%.
- On HOD, CD 0.44 vs HOLD 1.27 (unit size).
- On two-hand HOPE, CD 0.29 cm² and F5 85.4% vs COLMAP-BARF 0.96 and 62.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use generated-prior pose initialization and joint optimization with pose outlier voting | Hand-held object reconstruction · HO3D | CD (cm2): 1.82 -> 0.51 | Helped |
| Use generated-prior pose initialization and joint optimization with pose outlier voting | Hand-held object reconstruction · HOD | CD: 1.27 -> 0.44 | Helped |
| Use generated-prior pose initialization and joint optimization with pose outlier voting | Hand-held object reconstruction · HOPE two-hand interactions | CD (cm2): 0.96 -> 0.29 | Helped |
| Use pose initialization instead of initializing all poses from identity (ablation) | Joint shape-pose optimization · HO3D | CD (cm2): 7.948 -> 0.65 | Helped |
| Add semantic consistency Lsem to pose initialization (ablation) | Generated-prior pose initialization · HO3D | CD (cm2): 4.17 -> 0.65 | Helped |
| Add geometric constraints Lgeo to joint optimization (ablation) | Hand-held object reconstruction · HO3D | CD (cm2): 0.65 -> 0.62 | Helped |
| Add pose reinitialization using PSNR (ablation) | Hand-held object reconstruction · HO3D | CD (cm2): 0.62 -> 0.58 | Helped |
| Use Sampson-distance pose outlier voting instead of PSNR-only reinitialization (ablation) | Hand-held object reconstruction · HO3D | CD (cm2): 0.58 -> 0.51 | Helped |
| Use Genie-generated prior rather than ground-truth templates for pose initialization (ablation) | Hand-held object reconstruction · HO3D | CD (cm2): 0.43 -> 0.51 | Hurt |
| Use Shap-e-generated prior rather than ground-truth templates for pose initialization (ablation) | Hand-held object reconstruction · HO3D | CD (cm2): 0.43 -> 1.11 | Hurt |
Surprising findings
- Shap-e priors (lower quality) still give reasonable initialization; reconstruction with Shap-e still beats SOTA on HO3D.
- Removing the semantic loss in pose initialization severely degrades reconstruction (CD 4.17 vs 0.51).
Limitations
- Effectiveness depends on quality of generated priors; unique objects may not be generated well.
- DINO features can be nearly identical for different object sides, causing pose errors.
- Pose outlier voting relies on threshold and PSNR heuristics; PSNR-based detection can be biased by neighboring views.
Experimental setup
Datasets: HO3D HOD HOPE
Benchmarks: HO3D HOD HOPE
Metrics: Chamfer Distance F-score F5 F-score F10 ATE RPE translation RPE rotation
Baselines: HOLD IHOI Hampali et al. (incremental training) COLMAP-BARF BARF
Builds on: NeuS BARF HOLD DINO Genie
Closest papers by meaning
- HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video (CVPR 2024 · similarity 0.91)
- Free-Moving Object Reconstruction and Pose Estimation with Virtual Camera (AAAI 2025 · similarity 0.90)
- ComPose: When to Trust Hands for Object Pose Tracking (NeurIPS 2026 · similarity 0.90)
- In-Hand 3D Object Reconstruction from a Monocular RGB Video (AAAI 2024 · similarity 0.90)
- Learning Explicit Contact for Implicit Reconstruction of Hand-Held Objects from Monocular Images (AAAI 2024 · similarity 0.89)
- BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splatting (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.