Research map / Image generation / Autoregressive and latent generation
FoundHand: Large-Scale Domain-Specific Learning for Controllable Hand Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds FoundHand-10M, a 10M-image hand dataset with automatically extracted 2D keypoints and masks, and trains FoundHand, a 2D-keypoint-conditioned latent diffusion transformer for controllable single/dual hand image generation with zero-shot applications.
Problem
Image generation models struggle to produce realistic, controllable hands due to articulation, viewpoint variation, and occlusion, and no large unified hand dataset with consistent annotations exists.
Approach
Use 2D keypoint heatmaps as a universal hand representation encoding both articulation and camera viewpoint, and train an image-to-image diffusion transformer on frame/view pairs from a large merged hand dataset, with 3D self-attention between reference and target frames.
Key results
- Gesture transfer: FoundHand reaches PSNR 30.96, SSIM 0.82, LPIPS 0.20, FID 2.58, beating CosHand (26.21 PSNR, FID 3.60).
- Novel view synthesis on InterHand2.6M: PSNR 27.72 vs 19.97 for ImageDream, about 39% higher PSNR per the paper.
- Fine-tuning HaMeR on FoundHand domain-transferred data lowers PA-MPJPE from 14.41 to 13.68 and PA-MPVPE from 7.52 to 6.29.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FoundHand for reference-conditioned hand gesture transfer | Hand image generation · Gesture transfer | PSNR: 26.21 -> 30.96 | Helped |
| Use FoundHand for reference-conditioned hand gesture transfer | Hand image generation · Gesture transfer | SSIM: 0.75 -> 0.82 | Helped |
| Use FoundHand for reference-conditioned hand gesture transfer | Hand image generation · Gesture transfer | LPIPS: 0.22 -> 0.20 | Helped |
| Use FoundHand for reference-conditioned hand gesture transfer | Hand image generation · Gesture transfer | FID: 3.60 -> 2.58 | Helped |
| Use FoundHand's learned hand-specific priors for novel view synthesis | Novel view synthesis · InterHand2.6M | PSNR: 19.97 -> 27.72 | Helped |
| Use FoundHand for hand video synthesis without explicit video training | Hand video synthesis · 12 in-the-wild hand videos | PSNR: 17.64 -> 24.08 | Helped |
| Fine-tune HaMeR on domain-transferred data synthesized by FoundHand | HaMeR · 3D hand recovery on target domain | PA-MPJPE: 14.41 -> 13.68 | Helped |
| Fine-tune HaMeR on domain-transferred data synthesized by FoundHand | HaMeR · 3D hand recovery on target domain | PA-MPVPE: 7.52 -> 6.29 | Helped |
Surprising findings
- Model trained only on image pairs (no video) outperforms pose-conditioned video diffusion models ControlNeXt and AnimateAnyone on zero-shot video synthesis.
Limitations
- Latent resolution of 32x32 restricts inputs and outputs to 256x256.
- Relies on automatic 2D keypoint and mask annotations from MediaPipe and SAM rather than manual labels.
Experimental setup
Datasets: DexYCB ARCTIC ReInterHand InterHand Ego4D AssemblyHand HOI4D RHD RenderIH DART HAGRID WLASL
Benchmarks: in-the-wild web-sourced hand test set InterHand2.6M 12 in-the-wild hand videos
Metrics: PSNR SSIM LPIPS FID PA-MPJPE PA-MPVPE F@5 F@15
Baselines: GestureGAN Uni-ControlNet AnyControl CosHand Zero123 ZeroNVS ImageDream ControlNeXt AnimateAnyone HandFixer [47] [101]
Builds on: DiT Stable Diffusion VAE classifier-free guidance REPA SAM MediaPipe
Closest papers by meaning
- HanDiffuser: Text-to-Image Generation With Realistic Hand Appearances (CVPR 2024 · similarity 0.90)
- The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction (NeurIPS 2026 · similarity 0.89)
- RHanDS: Refining Malformed Hands for Generated Images with Decoupled Structure and Style Guidance (AAAI 2025 · similarity 0.89)
- InterHandGen: Two-Hand Interaction Generation via Cascaded Reverse Diffusion (CVPR 2024 · similarity 0.89)
- Hand1000: Generating Realistic Hands from Text with Only 1,000 Images (AAAI 2025 · similarity 0.88)
- SRHand: Super-Resolving Hand Images and 3D Shapes via View/Pose-aware Neural Image Representations and Explicit Meshes (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.