Research map / 3D vision / Human pose estimation
Reconstructing Hands in 3D with Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents HaMeR, a fully transformer-based (ViT-H) monocular 3D hand mesh recovery model trained on 2.7M examples from ten hand datasets, plus the HInt 2D keypoint benchmark of in-the-wild hands.
Problem
Monocular 3D hand mesh reconstruction is limited by small, controlled training datasets and weak evaluation on in-the-wild images, where robustness matters.
Approach
Scale both the training data (consolidating ~2.7M examples from multiple 2D/3D hand datasets) and the network capacity (ViT-H backbone with transformer decoder head regressing MANO parameters and camera), with a simple design and 2D reprojection plus adversarial priors.
Key results
- On HInt, HaMeR reaches 49.4 PCK@0.05 on New Days versus 16.1 for FrankMocap, roughly 2-3x improvement on in-the-wild data.
- On HO3Dv2, HaMeR achieves AUCJ 0.846 and PA-MPJPE 7.7mm, outperforming AMVUR (0.835, 8.3mm).
- Ablation shows large data and large model each help, with the combination giving the best HInt results (e.g., 44.4 PCK@0.05 on VISOR).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HaMeR instead of the strongest listed baseline | 3D hand mesh recovery · FreiHAND | PA-MPVPE: 5.8 -> 5.7 | Helped |
| Use HaMeR instead of the strongest listed baseline | 3D hand mesh recovery · HO3Dv2 | AUCJ: 0.835 -> 0.846 | Helped |
| Use HaMeR instead of the strongest listed baseline | 3D hand mesh recovery · HInt New Days, all joints | PCK@0.05: 16.8 -> 49.4 | Helped |
| Use HaMeR instead of the strongest listed baseline | 3D hand mesh recovery · HInt VISOR, all joints | PCK@0.05: 19.1 -> 44.4 | Helped |
| Use HaMeR instead of the strongest listed baseline | 3D hand mesh recovery · HInt Ego4D, all joints | PCK@0.05: 14.7 -> 40.3 | Helped |
| Increase training examples by 4× (ablation) | HaMeR base design · HInt New Days, all joints | PCK@0.05: 16.9 -> 31.3 | Helped |
| Replace ResNet50 with ViT-H (ablation) | HaMeR base design · HInt New Days, all joints | PCK@0.05: 16.9 -> 25.9 | Helped |
| Combine large training data with the ViT-H model (ablation) | HaMeR · HInt New Days, all joints | PCK@0.05: 16.9 -> 49.4 | Helped |
| Add HInt training data (ablation) | HaMeR · HInt New Days, all joints | PCK@0.05: 49.4 -> 51.7 | Helped |
| Add HInt training data (ablation) | HaMeR · HInt VISOR, all joints | PCK@0.05: 44.4 -> 56.5 | Helped |
| Add HInt training data (ablation) | HaMeR · HInt Ego4D, all joints | PCK@0.05: 40.3 -> 47.2 | Helped |
Surprising findings
- Gains on controlled benchmarks FreiHAND and HO3D are smaller and more saturated than gains on in-the-wild HInt.
Limitations
- Single-frame approach; temporal smoothness only observed qualitatively on video.
- HInt provides only 2D keypoint annotations, so 3D accuracy in the wild is not directly benchmarked.
- Only the right hand is trained; left hands are handled by mirroring.
Experimental setup
Datasets: FreiHAND HO3D MTC RHD InterHand2.6M H2O3D DexYCB COCO-WholeBody Halpe MPII NZSL Hands23 New Days VISOR
Benchmarks: FreiHAND HO3Dv2 HInt
Metrics: PA-MPJPE PA-MPVPE AUCJ AUCV F@5mm F@15mm PCK@0.05 PCK@0.1 PCK@0.15
Baselines: FrankMocap METRO Mesh Graphormer HandOccNet MobRecon AMVUR I2L-MeshNet Pose2Mesh
Builds on: HMR2.0 FrankMocap MANO ViT
Closest papers by meaning
- Fine-Grained Multi-View Hand Reconstruction Using Inverse Rendering (AAAI 2024 · similarity 0.89)
- HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions (CVPR 2024 · similarity 0.89)
- Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Camera (CVPR 2025 · similarity 0.89)
- WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild (CVPR 2025 · similarity 0.88)
- Hamba: Single-view 3D Hand Reconstruction with Graph-guided Bi-Scanning Mamba (NeurIPS 2024 · similarity 0.88)
- HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.