Research map / 3D vision / 3D generation
HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Synthesizes realistic hand-object images with a conditional diffusion model (normal, texture, orientation conditions) and samples novel grasp poses via farthest-pose and cross-distribution sampling to boost 3D hand-mesh reconstruction on DexYCB and HO3D.
Problem
3D hand-mesh reconstruction from a single RGB image is limited by low diversity of real hand-object datasets, and existing synthetic data suffers from a syn-to-real gap.
Approach
Train a content-aware conditional diffusion model (normal map, texture map, quaternion hand orientation) to synthesize realistic hand-object images with free 3D labels, and use a novel condition creator with farthest-pose sampling and similarity-weighted cross-distribution sampling to produce novel, diverse grasp poses.
Key results
- On HO3D (root-relative), HandOccNet + HandBooster improves V-PE from 24.2 to 20.5 mm and F@5 from 26.0 to 28.7.
- On DexYCB S1 (unseen subjects, Procrustes), HandOccNet + HandBooster improves V-PE from 6.3 to 5.9 mm and F@5 from 72.1 to 75.4.
- Generated data FID vs DexYCB training set is 2.91 compared with 8.52 for ObMan renderings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train HandOccNet with HandBooster synthetic data | HandOccNet · DexYCB S0 | root-relative J-PE (mm): 14.0 -> 11.9 | Helped |
| Train MobRecon with HandBooster synthetic data | MobRecon · DexYCB S0 | root-relative J-PE (mm): 14.2 -> 13.2 | Helped |
| Train H2ONet with HandBooster synthetic data | H2ONet · DexYCB S0 | root-relative J-PE (mm): 14.0 -> 12.9 | Helped |
| Train HandOccNet with HandBooster synthetic data | HandOccNet · DexYCB S1 | root-relative J-PE (mm): 17.4 -> 15.8 | Helped |
| Train MobRecon with HandBooster synthetic data | MobRecon · DexYCB S1 | root-relative J-PE (mm): 18.3 -> 17.5 | Helped |
| Train H2ONet with HandBooster synthetic data | H2ONet · DexYCB S1 | root-relative J-PE (mm): 17.6 -> 17.0 | Helped |
| Train HandOccNet with HandBooster synthetic data | HandOccNet · HO3D | root-relative J-PE (mm): 24.9 -> 21.1 | Helped |
| Train MobRecon with HandBooster synthetic data | MobRecon · HO3D | root-relative J-PE (mm): 25.2 -> 23.4 | Helped |
| Train H2ONet with HandBooster synthetic data | H2ONet · HO3D | root-relative J-PE (mm): 26.3 -> 24.0 | Helped |
| Add ObMan rendered synthetic data (ablation) | MobRecon · DexYCB | root-relative J-PE (mm): 14.20 -> 14.00 | Helped |
| Use depth and segmentation conditions (ablation) | HandBooster with MobRecon · DexYCB | root-relative J-PE (mm): 14.20 -> 14.91 | Hurt |
| Use normal and texture conditions (ablation) | HandBooster with MobRecon · DexYCB | root-relative J-PE (mm): 14.20 -> 13.93 | Helped |
Surprising findings
- Rendering-based ObMan data only slightly improves root-relative metrics, while the generated data improves all metrics even without the other techniques.
Limitations
- Relies on a pre-trained reconstruction model to filter edge-case synthetic images, which requires an existing model trained on the same data.
- Gains for MobRecon on DexYCB are small due to its mobile-friendly design and limited capacity.
- Grasp poses are generated per YCB object with simulation, so diversity is bounded by object set.
Experimental setup
Datasets: DexYCB HO3D ObMan MOW YCB objects
Benchmarks: DexYCB S0 DexYCB S1 HO3D v2 MOW
Metrics: J-PE V-PE J-AUC V-AUC F@5 F@15 FID
Baselines: HandOccNet MobRecon H2ONet ObMan Deformer HO3D baseline
Builds on: classifier-free diffusion DDPM GraspIt-style grasp simulation (Ye et al.) HOGAN
Closest papers by meaning
- HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video (CVPR 2024 · similarity 0.90)
- Fine-Grained Multi-View Hand Reconstruction Using Inverse Rendering (AAAI 2024 · similarity 0.90)
- EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild (CVPR 2025 · similarity 0.89)
- Monocular 3D Hand Mesh Recovery via Dual Noise Estimation (AAAI 2024 · similarity 0.89)
- Reconstructing Hands in 3D with Transformers (CVPR 2024 · similarity 0.89)
- SiMA-Hand: Boosting 3D Hand-Mesh Reconstruction by Single-to-Multi-View Adaptation (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.