Research map / 3D vision / Human pose estimation
Analyzing the Synthetic-to-Real Domain Gap in 3D Hand Pose Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematic study of the synthetic-to-real gap in 3D hand pose estimation using a NIMBLE-based rendering pipeline, decomposing arm, amplitude augmentation, object occlusion, pose and skeleton topology effects, and showing synthetic-only training nearly matches real-data training on FreiHAND.
Problem
Synthetic-only training for 3D hand pose estimation still lags real-data training due to a large synthetic-to-real gap, and it is unclear which factors cause it.
Approach
Build a controllable rendering pipeline (NIMBLE hand model, HDRI backgrounds, diverse textures, dynamic lighting) so that individual hand-image components (arm, object, background, texture, pose, skeleton) can be varied and decomposed to attribute the synthetic-to-real gap; add a VAE-based object occlusion prior for refinement.
Key results
- Models trained only on SynFrei reach 97% (S2HAND), 91% (CMR), 88% (METRO), 91% (MeshGraphormer), 84% (simpleHand) of real-FreiHAND performance on the FreiHAND test set.
- Mixing SynDex with Dex-YCB improves S2HAND cross-dataset PA-MPJPE on FreiHAND from 1.63 to 1.49 cm and on HO3D-v2 from 1.25 to 1.15 cm.
- Removing amplitude spectrum augmentation raises SynFrei PA-MPJPE by 0.09 cm; removing the arm raises it by 0.05 to 0.09 cm range in Table 4 rows (i) and (v).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train S2HAND on SynFrei instead of real FreiHAND | S2HAND · FreiHAND test | PA-MPJPE: 0.99 -> 1.02 | Hurt |
| Add the arm to synthetic training images (ablation) | S2HAND trained on SynFrei · FreiHAND test | PA-MPJPE: 1.07 -> 1.02 | Helped |
| Add the arm to synthetic training images (ablation) | S2HAND trained on SynDex · Dex-YCB test | PA-MPJPE: 0.90 -> 0.87 | Helped |
| Add amplitude spectrum augmentation (ablation) | S2HAND trained on SynFrei · FreiHAND test | PA-MPJPE: 1.11 -> 1.02; 0.09 cm | Helped |
| Add amplitude spectrum augmentation (ablation) | S2HAND trained on SynDex · Dex-YCB test | PA-MPJPE: 0.89 -> 0.87; 0.02 cm | Helped |
| Add interacting objects to synthetic training images (ablation) | S2HAND trained on SynFrei · FreiHAND test | PA-MPJPE: 1.07 -> 1.02; 0.05 cm | Helped |
| Add interacting objects to synthetic training images (ablation) | S2HAND trained on SynDex · Dex-YCB test | PA-MPJPE: 0.95 -> 0.87; 0.08 cm | Helped |
| Replace realistic arm and object masks with randomized RGB values (ablation) | S2HAND trained on synthetic data · FreiHAND and Dex-YCB tests | PA-MPJPE: 1.02 -> 1.04; 0.87 -> 0.92 | Hurt |
| Use 5% of real hand poses rather than the full pose set for synthesis (ablation) | S2HAND trained on synthetic hand images · FreiHAND test | PA-MPJPE: around 80% of the final performance | Hurt |
| Add label adaptation between NIMBLE and real skeleton topologies (ablation) | S2HAND trained with NIMBLE joints · FreiHAND test | PA-MPJPE: 1.28 -> 1.02 | Helped |
| Add a VAE object occlusion prior to refine predicted joints (ablation) | S2HAND trained on SynFrei · HO3D-v2 validation, by occlusion level | PA-MPJPE | Helped |
| Mix SynFrei with real FreiHAND training data | S2HAND · FreiHAND test | PA-MPJPE: 0.99 -> 0.95 | Helped |
Surprising findings
- Stronger backbones (e.g., simpleHand) show larger synthetic-to-real drops, which the authors attribute to overfitting to synthetic data.
- Using only 40% of real hand poses for synthesis reaches 97% of the full-pose performance, and randomly filled arm/object occlusions still reach 95% and 93% of real performance.
Limitations
- Gap remains (about 3-16% relative) for most architectures; synthetic-only training not fully matching real data for all methods.
- Analysis mainly uses S2HAND; other models analyzed less thoroughly.
- Arm and object components are composed from real images via Grounding DINO and SAM rather than fully rendered.
Experimental setup
Datasets: FreiHAND Dex-YCB ObMan HO3D-v2 MOW RHD HDRI scenes from Poly Haven-style collection [50] DART poses (DARTset comparison)
Benchmarks: FreiHAND Dex-YCB HO3D-v2 RHD MOW DARTset comparison
Metrics: PA-MPJPE PA-MPVPE
Baselines: S2HAND CMR METRO MeshGraphormer simpleHand DARTset RenderIH
Builds on: NIMBLE PASTA DARTset S2HAND MeshGraphormer
Closest papers by meaning
- Efficiently Creating 3D Training Data for Fine Hand Pose Estimation (CVPR 2016 · similarity 0.89)
- Cross-Domain 3D Hand Pose Estimation With Dual Modalities (CVPR 2023 · similarity 0.88)
- HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions (CVPR 2024 · similarity 0.87)
- HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video (CVPR 2024 · similarity 0.86)
- SiMHand: Mining Similar Hands for Large-Scale 3D Hand Pose Pre-training (ICLR 2025 · similarity 0.86)
- HOISDF: Constraining 3D Hand-Object Pose Estimation with Global Signed Distance Fields (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.