Research map / 3D vision / Human pose estimation
VTON-HandFit: Virtual Try-on for Arbitrary Hand Pose Guided by Hand Priors Embedding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VTON-HandFit, a diffusion-based virtual try-on method that uses hand priors (HaMeR 3D hand parameters and DINOv2 appearance features) and a hand-canny loss to handle garments occluded by hands, and introduces the Handfit-3K hand-occlusion test set.
Problem
Diffusion-based virtual try-on degrades when the hand occludes the garment region, since existing methods rely on hand parsing that fails on small fingers and leaves artifacts or background residue.
Approach
Reconstructs hand structure and appearance by disentangling 3D hand priors (HaMeR MANO-related structure embeddings and DINOv2 visual embeddings) and injecting them via a Hand-Pose Aggregation Net (ControlNet-style) and mask-restricted cross-attention, plus a hand-canny edge loss, instead of post-hoc hand refinement.
Key results
- On Handfit-3K (paired), VTON-HandFit reports FID 5.6027 and KID 0.4932 versus best baseline IDM-VTON FID 7.7159 and KID 2.1287, with MPJPE 25.14 vs 48.64.
- On VITON-HD unpaired, FID 8.8178 and KID 0.9951, the best among compared methods (CatVTON FID 9.1434).
- Ablation on VITON-HD shows FID falling from 6.9549 (baseline) to 5.9564 with full components and hand-canny loss not separately ablated.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VTON-HandFit instead of CatVTON | Virtual try-on · DressCode paired | FID: 3.2755 -> 2.8666 | Helped |
| use VTON-HandFit instead of CatVTON | Virtual try-on · DressCode unpaired | FID: 5.4220 -> 5.2314 | Helped |
| use VTON-HandFit instead of CatVTON | Virtual try-on · VITON-HD paired | FID: 6.1394 -> 5.9564 | Helped |
| use VTON-HandFit instead of CatVTON | Virtual try-on · VITON-HD unpaired | FID: 9.1434 -> 8.8178 | Helped |
| use VTON-HandFit instead of IDM-VTON | Virtual try-on with hand occlusion · Handfit-3K paired | MPJPE: 48.6378 -> 25.1376 | Helped |
| use VTON-HandFit instead of IDM-VTON | Virtual try-on with hand occlusion · Handfit-3K paired | FID-H: 11.5403 -> 7.0574 | Helped |
| use VTON-HandFit instead of IDM-VTON | Virtual try-on with hand occlusion · Handfit-3K unpaired | MPJPE: 57.3447 -> 26.6892 | Helped |
| use VTON-HandFit instead of IDM-VTON | Virtual try-on with hand occlusion · Handfit-3K unpaired | FID-H: 12.7721 -> 9.0003 | Helped |
| add Hand-Pose Aggregation Net using Densepose only (ablation) | Baseline virtual try-on model · VITON-HD | FID: 6.9549 -> 6.8924 | Helped |
| add Depth and DWpose to the Densepose-based Hand-Pose Aggregation Net (ablation) | Baseline+HPA (Densepose) · VITON-HD | FID: 6.8924 -> 6.3390 | Helped |
| add hand structural embedding (ablation) | Baseline+HPA · VITON-HD | FID: 6.3390 -> 6.2225 | Helped |
| add hand appearance embedding (ablation) | Baseline+HPA+Cstruct · VITON-HD | FID: 6.2225 -> 6.0515 | Helped |
Surprising findings
- Post-processing with HandRefiner (IDM-VTON*) worsened several metrics versus plain IDM-VTON, failing to reconstruct incomplete hands and altering correct ones.
Limitations
- Handfit-3K is self-collected and only released as a test set without stated public release, limiting reproducibility of hand-occlusion evaluation.
- Relies on external hand reconstruction (HaMeR) and pose estimators, so errors in those priors propagate to try-on output.
- Hand-canny loss and components were not ablated individually in the reported ablation table.
Experimental setup
Datasets: VITON-HD DressCode Handfit-3K
Benchmarks: VITON-HD DressCode Handfit-3K
Metrics: SSIM LPIPS FID KID FID-H KID-H MPJPE
Baselines: LaDI-VTON StableVTON OOTDiffusion IDM-VTON CatVTON IDM-VTON* (HandRefiner post-processing)
Builds on: ControlNet Stable Diffusion HaMeR DINOv2 IDM-VTON
Closest papers by meaning
- OHTA: One-shot Hand Avatar via Data-driven Implicit Priors (CVPR 2024 · similarity 0.86)
- SiMHand: Mining Similar Hands for Large-Scale 3D Hand Pose Pre-training (ICLR 2025 · similarity 0.85)
- The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction (NeurIPS 2026 · similarity 0.85)
- SiMA-Hand: Boosting 3D Hand-Mesh Reconstruction by Single-to-Multi-View Adaptation (AAAI 2024 · similarity 0.84)
- HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions (CVPR 2024 · similarity 0.84)
- Single-view Image to Novel-view Generation for Hand-Object Interactions (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.