Research map / 3D vision / Human pose estimation
WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents WiLoR, a full-stack pipeline pairing a real-time anchor-free hand detector trained on a new 2M-image in-the-wild hand dataset (WHIM) with a ViT-based 3D hand reconstruction model using a multi-scale image-aligned refinement module.
Problem
Real-time, robust hand detection in the wild (especially multi-hand scenes) is lacking, and single-image 3D hand reconstruction suffers from poor image alignment and temporal jitter in video.
Approach
Combines a fast fully convolutional anchor-free hand localizer trained on a large pseudo-labeled in-the-wild dataset with a coarse-to-fine ViT regressor whose coarse MANO estimate is refined by sampling multi-scale image features projected from the mesh.
Key results
- Proposed-M detector reaches 138 FPS and 53.79 mAP on WHIM, versus 49.44 mAP for ContactHands (3 FPS).
- On FreiHAND, WiLoR achieves PA-MPJPE 5.5 mm and PA-MPVPE 5.1 mm, beating HaMeR (6.0/5.7).
- On HO3D, WiLoR reaches AUCJ 0.851 and PA-MPJPE 7.5 mm, and reduces dynamic jitter (MPFVE x100 4.43 vs 10.60 for HaMeR).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the large-scale WHIM training data rather than train on Oxford-Hands (ablation) | WiLoR hand detector · Oxford-Hands and WHIM | mAP: 40.34 -> 48.98; 35.29 -> 53.79 | Helped |
| add training data, increasing the training set from 0.25M to 0.5M images (ablation) | WiLoR hand detector · Oxford-Hands and WHIM | mAP: 26.69 -> 35.15; 38.48 -> 42.11 | Helped |
| add training data, increasing the training set from 0.5M to 1M images (ablation) | WiLoR hand detector · Oxford-Hands and WHIM | mAP: 35.15 -> 43.04; 42.11 -> 47.92 | Helped |
| replace the proposed backbone with ResNet50 (ablation) | WiLoR hand detector · Oxford-Hands and WHIM | mAP: 48.98 -> 47.34; 53.79 -> 51.85 | Mixed |
| replace the proposed backbone with HRNet (ablation) | WiLoR hand detector · Oxford-Hands and WHIM | mAP: 48.98 -> 49.83; 53.79 -> 54.12 | Mixed |
| remove data augmentations (ablation) | WiLoR hand detector · Oxford-Hands and WHIM | mAP: 48.98 -> 42.17; 53.79 -> 49.93 | Hurt |
| remove landmark regression loss (ablation) | WiLoR hand detector · Oxford-Hands and WHIM | mAP: 48.98 -> 45.96; 53.79 -> 51.44 | Hurt |
| replace the ViT backbone with FastViT (ablation) | WiLoR hand reconstruction · FreiHAND | PA-MPJPE: 5.5 -> 6.5 | Hurt |
| remove ViTPose pre-trained weights and train the backbone from scratch (ablation) | WiLoR hand reconstruction · FreiHAND | PA-MPJPE: 5.5 -> 5.9 | Hurt |
| replace multi-scale refinement with single-scale refinement (ablation) | WiLoR hand reconstruction · FreiHAND | PA-MPJPE: 5.5 -> 6.0 | Hurt |
| remove the refinement module (ablation) | WiLoR hand reconstruction · FreiHAND | PA-MPJPE: 5.5 -> 6.1 | Hurt |
| train only on FreiHAND instead of the full training set (ablation) | WiLoR hand reconstruction · FreiHAND | PA-MPJPE: 5.5 -> 6.1 | Hurt |
Surprising findings
- Training the detector on WHIM alone generalizes better to OxfordHands than training on OxfordHands itself (per the paper's observation).
Limitations
- Training labels for WHIM are produced by an ensemble of automatic detectors and model fitting rather than manual annotation.
- Temporal coherence is obtained without a temporal module, relying on per-frame detection stability; dynamic evaluation is limited to HO3D.
Experimental setup
Datasets: WHIM COCO-WholeBody OxfordHands FreiHAND HO3D MTC RHD InterHand2.6M H2O3D DEX YCB Halpe MPII NZSL
Benchmarks: WHIM COCO-WholeBody OxfordHands FreiHAND HO3D
Metrics: AP0.5 mAP FPS model size PA-MPJPE PA-MPVPE F@5 F@15 AUCJ AUCV
Baselines: MediaPipe OpenPose ContactHands ViTDet METRO Mesh Graphormer AMVUR MobRecon HaMeR SimpleHand I2L-MeshNet Pose2Mesh
Builds on: HaMeR ViTPose PANet YOLO-style one-stage detection MANO
Closest papers by meaning
- Reconstructing Hands in 3D with Transformers (CVPR 2024 · similarity 0.88)
- Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Camera (CVPR 2025 · similarity 0.88)
- HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videos (CVPR 2025 · similarity 0.87)
- EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild (CVPR 2025 · similarity 0.87)
- SiMHand: Mining Similar Hands for Large-Scale 3D Hand Pose Pre-training (ICLR 2025 · similarity 0.86)
- HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.