Research map / 3D vision / Face and shape
Authentic Hand Avatar from a Phone Scan via Universal Hand Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents UHM, a universal 3D hand model trained with simultaneous tracking and modeling plus an optical-flow image matching loss, which can be adapted to a person's hand from a ~15-second phone scan to produce an authentic animatable avatar.
Problem
Universal 3D hand models rely on separate tracking and modeling stages, so tracking errors accumulate, and vertices slide semantically during fitting; existing personalized hand avatars need studio captures rather than short phone scans.
Approach
A VAE-based universal hand model (ID-dependent skeleton and vertex correctives plus pose-and-ID vertex correctives) trained end-to-end with tracking and modeling jointly, an optical-flow-based image matching loss to prevent skin sliding, and a learned ShadowNet with adaptation from a phone scan using the pretrained priors.
Key results
- UHM achieves P2S error 0.72 mm on the first test set versus 1.20 mm for Handy and 1.21 mm for NIMBLE.
- On the in-house phone scan test set, UHM reaches PSNR 31.82 and LPIPS 0.076, versus 29.89 and 0.092 for HARP.
- Adaptation from a 15-second phone scan takes about 2 hours, versus 6 hours for HARP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UHM instead of Handy for universal hand modeling | Universal 3D hand modeling · MANO test set | P2S error (mm): 1.20 -> 0.72 | Helped |
| Use UHM instead of Handy for universal hand modeling | Universal 3D hand modeling · DHM test set | P2S error (mm): 1.11 -> 0.59 | Helped |
| Use UHM instead of LISA | 3D hand modeling · DHM test set | P2S error (mm), 1 view: 3.68 -> 1.63 | Helped |
| Use UHM instead of HARP for phone-scan avatar adaptation | Phone-scan hand avatar adaptation · Our phone scan test set | PSNR: 29.89 -> 31.82 | Helped |
| Use UHM instead of HARP for phone-scan avatar adaptation | Phone-scan hand avatar adaptation · HARP test set | PSNR: 27.50 -> 32.55 | Helped |
| Add the optical-flow image matching loss during tracking and modeling (ablation) | UHM · Tracking and modeling image correspondences | L2 norm of optical flow | Helped |
| Add ShadowNet for phone-scan shadow removal (ablation) | UHM phone-scan adaptation · Phone scan | Shadow removal | Helped |
Limitations
- Universal model trained on a proprietary studio dataset of 177 captures, which limits reproducibility.
- Image matching loss is less effective on the back of the hand, where textures are not distinctive and optical flow gives weaker correspondences.
Experimental setup
Datasets: studio dataset (177 training captures) MANO test set (50 scans, 6 subjects) DHM dataset HARP dataset phone scan dataset (18 scans)
Benchmarks: MANO test set DHM dataset HARP dataset in-house phone scan dataset
Metrics: P2S error (mm) PSNR SSIM LPIPS
Baselines: MANO NIMBLE Handy HARP LISA DHM
Builds on: MANO STAR HARP RAFT
Closest papers by meaning
- OHTA: One-shot Hand Avatar via Data-driven Implicit Priors (CVPR 2024 · similarity 0.85)
- HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos (CVPR 2025 · similarity 0.84)
- Analyzing the Synthetic-to-Real Domain Gap in 3D Hand Pose Estimation (CVPR 2025 · similarity 0.84)
- URHand: Universal Relightable Hands (CVPR 2024 · similarity 0.83)
- MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generation (NeurIPS 2025 · similarity 0.83)
- BiTT: Bi-directional Texture Reconstruction of Interacting Two Hands from a Single Image (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.