Research map / Image generation / Editing and style transfer
RHanDS: Refining Malformed Hands for Generated Images with Decoupled Structure and Style Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RHanDS, a conditional diffusion inpainting framework that refines malformed generated hands using a 3D hand-mesh depth map for structure and the malformed hand itself (via CLIP style encoder) for style, trained in two stages on new multi-style hand datasets.
Problem
Diffusion models often generate malformed hands with wrong structure (finger count, shape); existing refinement methods struggle to keep the hand's style (e.g., anime, sculpture, oil painting) while correcting structure.
Approach
Decouple structure and style guidance: a ControlNet-style structure encoder conditioned on a depth rendering of the reconstructed hand mesh corrects structure, while a CLIP-based style encoder with the malformed hand as reference preserves style; a two-stage training (paired same-person hands for style, then SMPL-H mesh-rendered multi-style hands for structure) reduces mutual interference.
Key results
- On the Multi-Style Malformed Hand-Mesh Dataset, RHanDS reaches FID 32.59 and style loss 5.04 versus HandRefiner's 38.52 and 10.13.
- On the HandRefiner Image2Image Dataset (hand-region metrics), RHanDS achieves FID 22.18 and MPJPE 9.86 versus HandRefiner's 33.84 and 12.02.
- Removing stage-1 style training raises FID from 32.59 to 47.76 on the multi-style test set.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply RHanDS rather than HandRefiner | Hand refinement · Image2Image Dataset, hand region | FID: 33.84 -> 22.18 | Helped |
| Apply RHanDS rather than HandRefiner | Hand refinement · Image2Image Dataset, hand region | MPJPE: 12.02 -> 9.86 | Helped |
| Apply RHanDS rather than HandRefiner | Hand refinement · Image2Image Dataset, hand region | Conf.: 0.94 -> 0.96 | Helped |
| Use paired hands with matching style but different gestures for first-stage training (ablation) | RHanDS · Multi-Style Malformed Hand-Mesh Dataset | MPJPE: 29.48 -> 27.23 | Helped |
| Use paired hands with matching style but different gestures for first-stage training (ablation) | RHanDS · Multi-Style Malformed Hand-Mesh Dataset | Style Loss: 5.35 -> 5.04 | Helped |
| Use the multi-style hand-mesh dataset alongside SGD in second-stage training (ablation) | RHanDS · Multi-Style Malformed Hand-Mesh Dataset | MPJPE: 26.17 -> 27.23 | Hurt |
| Use the multi-style hand-mesh dataset alongside SGD in second-stage training (ablation) | RHanDS · Multi-Style Malformed Hand-Mesh Dataset | FID: 33.56 -> 32.59 | Helped |
| Use the multi-style hand-mesh dataset alongside SGD in second-stage training (ablation) | RHanDS · Multi-Style Malformed Hand-Mesh Dataset | Style Loss: 5.23 -> 5.04 | Helped |
| Train with both style-guidance and structure-guidance stages instead of training without stage 1 (ablation) | RHanDS · Multi-Style Malformed Hand-Mesh Dataset | FID: 15.17 | Helped |
| Train with both style-guidance and structure-guidance stages instead of training without stage 1 (ablation) | RHanDS · Multi-Style Malformed Hand-Mesh Dataset | Style Loss: 2.07 | Helped |
| Use structure guidance rather than disabling the structure encoder (ablation) | RHanDS · Multi-Style Malformed Hand-Mesh Dataset | MPJPE: 67.19 -> 27.23 | Helped |
| Use structure weight w=0.6 rather than w=0.0 (ablation) | RHanDS · Multi-Style Malformed Hand-Mesh Dataset | MPJPE: 67.19 -> 27.23 | Helped |
Surprising findings
- Using the malformed hand itself as the style reference during stage 1 without paired-hand training leaks structure, which degrades structure metrics (stage-1* MPJPE 29.48 vs 27.23).
Experimental setup
Datasets: Multi-Style Paired Hand Dataset Multi-Style Hand-Mesh Dataset Static Gesture Dataset (SGD) HumanArt HAGRID
Benchmarks: HandRefiner Text2Image Dataset HandRefiner Image2Image Dataset Multi-Style Malformed Hand-Mesh Dataset (HumanArt-based)
Metrics: FID MPJPE Keypoint detection confidence Style loss
Baselines: HandRefiner HandRefiner* Stable Diffusion
Builds on: Stable Diffusion Inpainting ControlNet IP-Adapter HandRefiner SDEdit
Closest papers by meaning
- FoundHand: Large-Scale Domain-Specific Learning for Controllable Hand Image Generation (CVPR 2025 · similarity 0.89)
- SRHand: Super-Resolving Hand Images and 3D Shapes via View/Pose-aware Neural Image Representations and Explicit Meshes (NeurIPS 2025 · similarity 0.88)
- Fine-Grained Multi-View Hand Reconstruction Using Inverse Rendering (AAAI 2024 · similarity 0.88)
- HanDiffuser: Text-to-Image Generation With Realistic Hand Appearances (CVPR 2024 · similarity 0.88)
- HHMR: Holistic Hand Mesh Recovery by Enhancing the Multimodal Controllability of Graph Diffusion Models (CVPR 2024 · similarity 0.87)
- HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.