Research map / 3D vision / Human pose estimation
HandOS: 3D Hand Reconstruction in One Stage
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HandOS, an end-to-end one-stage 3D hand mesh reconstruction model that builds on a frozen Grounding DINO detector with side-tuning and an interactive 2D-3D decoder, removing the separate left-right classification stage.
Problem
Hand mesh reconstruction is usually done in multiple stages (detection, left-right classification, pose estimation), which adds redundant computation and cumulative errors when earlier stages fail.
Approach
Freeze a DETR-like detector (Grounding DINO) and attach a learnable side network plus an interactive 2D-3D decoder with instance-to-joint query expansion, 2D-to-3D query lifting (initialized from MANO skinning weights), and hierarchical attention, so 2D joints, 3D vertices and camera translation are predicted jointly; left/right is read from mesh normals.
Key results
- Achieves 5.0 mm PA-MPJPE on FreiHand, versus 5.7-6.0 for prior methods such as HaMeR (6.0) and Hamba (5.8).
- Reaches 64.6% PCK@0.05 on HInt-Ego4D using 2D predictions, with 204K training samples versus roughly 2.7M for HaMeR.
- Obtains 5.2 mm PA-MPJPE on DexYCB, outperforming Zhou et al. (5.5) and HandOccNet (5.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use instance-specific queries instead of unified queries (ablation) | HandOS · HInt New Days | PCK@0.1: 80.7 -> 82.5 | Helped |
| use Fe6 instead of Fe4 for deformable-attention values (ablation) | HandOS · HInt New Days | PCK@0.1: 79.5 -> 82.5 | Helped |
| use side tuning rather than a scratch-trained SwinT network (ablation) | HandOS · HInt New Days | PCK@0.1: 75.6 -> 82.5 | Helped |
| use hierarchical attention rather than independent attention (ablation) | HandOS · FreiHand | PA-MPVPE (mm): 6.4 -> 5.6 | Helped |
| use hierarchical attention rather than unidirectional attention (ablation) | HandOS · FreiHand | PA-MPVPE (mm): 5.7 -> 5.6 | Helped |
| use hierarchical attention rather than full attention (ablation) | HandOS · FreiHand | PA-MPVPE (mm): 5.9 -> 5.6 | Helped |
| train without image flipping (ablation) | HandOS · FreiHand with left-hand training data | PA-MPJPE (mm): 5.3 -> 5.0 | Helped |
| train without image flipping (ablation) | HandOS · FreiHand with left-hand training data | PA-MPVPE (mm): 5.6 -> 5.3 | Helped |
| use direct 2D predictions rather than projected 3D predictions | HandOS · HInt visible joints | PCK@0.05: 65.7 vs 78.1 | Mixed |
| use projected 3D predictions rather than direct 2D predictions | HandOS · HInt occluded joints | PCK@0.05: 35.8 vs 35.8 | Mixed |
| use HandOS rather than Hamba | Hand reconstruction · FreiHand | PA-MPJPE (mm): 5.8 -> 5.0 | Helped |
| use HandOS rather than Zhou et al. | Hand reconstruction · DexYCB | PA-MPJPE (mm): 5.5 -> 5.2 | Helped |
Surprising findings
- Ablation shows that flipping images to add left-hand training data on FreiHand slightly degrades results (PA-MPJPE 5.0 to 5.3), though it enables unified left-right learning.
- Reported errors include detection and left-right failures; the authors argue prior works' use of ground-truth boxes overstates real-world performance.
Limitations
- Left-right awareness and detection errors are mixed into reported metrics, so comparisons with prior work that assume perfect detection are not direct.
- Left-hand training via image flipping slightly hurts accuracy on FreiHand.
Experimental setup
Datasets: FreiHand HO3Dv3 DexYCB HInt COCO-WholeBody Onehand10K
Benchmarks: FreiHand HO3Dv3 DexYCB HInt
Metrics: PA-MPJPE PA-MPVPE F-score PCK AUC
Baselines: METRO MeshGraphormer MobRecon PointHMR HaMeR Hamba AMVUR SPMHand HandOccNet H2ONet FrankMocap ViTPose
Builds on: Grounding DINO EDPose Deformable DETR MANO HaMeR
Closest papers by meaning
- HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions (CVPR 2024 · similarity 0.89)
- HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video (CVPR 2024 · similarity 0.88)
- SiMA-Hand: Boosting 3D Hand-Mesh Reconstruction by Single-to-Multi-View Adaptation (AAAI 2024 · similarity 0.88)
- SRHand: Super-Resolving Hand Images and 3D Shapes via View/Pose-aware Neural Image Representations and Explicit Meshes (NeurIPS 2025 · similarity 0.88)
- EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild (CVPR 2025 · similarity 0.87)
- Reconstructing Hands in 3D with Transformers (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.