Research map / Image generation / Text-to-image diffusion
HanDiffuser: Text-to-Image Generation With Realistic Hand Appearances
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HanDiffuser, a two-stage diffusion pipeline that first generates SMPL-H body and MANO hand parameters from text and then synthesizes images conditioned on those hand parameters and text via a Text+Hand encoder, to produce realistic hands.
Problem
Text-to-image diffusion models generate unrealistic hands (irregular poses and shapes, wrong finger counts, implausible finger orientations).
Approach
Generate parametric SMPL-H/MANO hand parameters from text with a diffusion model, then inject hand joint positions, rotations, and vertices into Stable Diffusion via a jointly trained Text+Hand encoder that extends CLIP with spatial positional tokens and MLPs for hand geometry.
Key results
- HanDiffuser achieves FID 13.918 and FID-H 27.550 versus 29.005 and 34.372 for off-the-shelf Stable Diffusion.
- Hand detection confidence rises to 0.978 for HanDiffuser versus 0.887 for Stable Diffusion and 0.969 for ControlNet.
- User study (35 participants, 700 responses) gives plausibility 3.51 for HanDiffuser vs 2.74 for fine-tuned Stable Diffusion.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HanDiffuser instead of ControlNet | Text-to-image generation with hand conditioning · Test set | FID: 18.694 -> 13.918 | Helped |
| Use HanDiffuser instead of ControlNet | Text-to-image generation with hand conditioning · Test set | KID: 5.93×10−3 -> 4.07×10−3 | Helped |
| Use HanDiffuser instead of ControlNet | Text-to-image generation with hand conditioning · Test set | FID-H: 28.091 -> 27.550 | Helped |
| Use HanDiffuser instead of ControlNet | Text-to-image generation with hand conditioning · Test set | KID-H: 2.19×10−2 -> 2.11×10−2 | Helped |
| Use HanDiffuser instead of ControlNet | Text-to-image generation with hand conditioning · Test set | Hand Conf.: 0.969 -> 0.978 | Helped |
| Add 2D hand joint locations to hand embeddings (ablation) | HanDiffuser · Test set | FID: 16.839 -> 13.918 | Helped |
| Add 3D joint rotations and hand vertices to hand embeddings (ablation) | HanDiffuser · Test set | FID: 14.586 -> 13.918 | Helped |
Surprising findings
- Fine-tuned Stable Diffusion evaluated with hand-detector crops is optimistic because the detector cannot localize bad hands, so its hand metrics are biased.
- ControlNet, even with ground-truth skeleton control images at test time, performs worse than HanDiffuser.
Limitations
- Fails when the action in the prompt is unclear, when hand pose descriptions are not followed exactly, or with small thin objects and object affordances.
- Evaluated only on single-person images; multi-person, complex hand-object interactions and hand-hand interactions are left for future work.
- Hand-detector-based metrics for baselines are optimistic, so comparisons rely on partly biased evaluation.
Experimental setup
Datasets: LAION-5B (curated subset)
Benchmarks: curated 30K held-out text-image test split
Metrics: FID KID FID-H KID-H hand detection confidence user study Likert plausibility and relevance
Baselines: Stable Diffusion Stable Diffusion Fine-tuned ControlNet HanDiffuser w/o 2D hand joints HanDiffuser w/o 3D joint rotation and vertices
Builds on: Stable Diffusion ControlNet MDM CLIP SMPL-H
Closest papers by meaning
- Hand1000: Generating Realistic Hands from Text with Only 1,000 Images (AAAI 2025 · similarity 0.91)
- FoundHand: Large-Scale Domain-Specific Learning for Controllable Hand Image Generation (CVPR 2025 · similarity 0.90)
- RHanDS: Refining Malformed Hands for Generated Images with Decoupled Structure and Style Guidance (AAAI 2025 · similarity 0.88)
- HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Data (CVPR 2024 · similarity 0.87)
- Single-view Image to Novel-view Generation for Hand-Object Interactions (AAAI 2025 · similarity 0.87)
- InterHandGen: Two-Hand Interaction Generation via Cascaded Reverse Diffusion (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.