Research map / Image generation / GANs
RealisHuman: A Two-Stage Approach for Refining Malformed Human Parts in Generated Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RealisHuman, a two-stage post-processing framework that regenerates malformed hands and faces using the original parts as references (Part Detail Encoder plus DINOv2 and 3D mesh/depth guidance), then repaints the surrounding area to blend them back.
Problem
Diffusion models often generate malformed human parts such as hands and faces; existing post-processing (HandRefiner) loses skin tone/texture consistency, struggles with small regions, and distorts other areas.
Approach
Two-stage refinement: a reference-based part generator (Part Detail Encoder that concatenates reference self-attention keys/values into SD self-attention, plus DINOv2 embeddings and depth-based pose conditioning) produces rectified parts, then an SD-inpainting model repaints the boundary region after pasting the part back, with mask erosion to avoid edge artifacts.
Key results
- Applied to MagicAnimate, hand FID dropped from 57.73 to 55.18 and hand detection confidence rose from 0.90 to 0.94 on UBC Fashion.
- Applied to AnimateAnyone, face FID dropped from 20.55 to 15.44 and face detection confidence rose from 0.82 to 0.90.
- Qualitative comparisons show better skin-tone consistency and fewer distortions in other regions than HandRefiner, though no quantitative HandRefiner comparison is reported.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply RealisHuman refinement | AnimateAnyone · UBC Fashion hand regions | FID: 14.26 -> 13.02 | Helped |
| Apply RealisHuman refinement | AnimateAnyone · UBC Fashion hand regions | Det. Conf.: 0.86 -> 0.91 | Helped |
| Apply RealisHuman refinement | AnimateAnyone · UBC Fashion face regions | FID: 20.55 -> 15.44 | Helped |
| Apply RealisHuman refinement | AnimateAnyone · UBC Fashion face regions | Det. Conf.: 0.82 -> 0.90 | Helped |
| Apply RealisHuman refinement | Champ · UBC Fashion hand regions | FID: 27.28 -> 25.58 | Helped |
| Apply RealisHuman refinement | Champ · UBC Fashion hand regions | Det. Conf.: 0.87 -> 0.92 | Helped |
| Apply RealisHuman refinement | Champ · UBC Fashion face regions | FID: 20.11 -> 16.74 | Helped |
| Apply RealisHuman refinement | Champ · UBC Fashion face regions | Det. Conf.: 0.85 -> 0.92 | Helped |
| Apply RealisHuman refinement | MagicAnimate · UBC Fashion hand regions | FID: 57.73 -> 55.18 | Helped |
| Apply RealisHuman refinement | MagicAnimate · UBC Fashion hand regions | Det. Conf.: 0.90 -> 0.94 | Helped |
| Apply RealisHuman refinement | MagicAnimate · UBC Fashion face regions | FID: 43.12 -> 38.81 | Helped |
| Apply RealisHuman refinement | MagicAnimate · UBC Fashion face regions | Det. Conf.: 0.87 -> 0.92 | Helped |
Limitations
- Struggles to reconstruct hand-object interactions.
- Fails to keep consistency when objects are present.
- Cannot estimate correct pose when the original hand is severely distorted.
Experimental setup
Datasets: UBC Fashion in-house collected hand and face crops (~58k hands, ~38k faces)
Benchmarks: UBC Fashion
Metrics: FID keypoint detection confidence (hand and face detectors)
Baselines: HandRefiner AnimateAnyone Champ MagicAnimate
Builds on: Stable Diffusion HandRefiner ControlNet SD-inpainting DINOv2
Closest papers by meaning
- HyperHuman: Hyper-Realistic Human Generation with Latent Structural Diffusion (ICLR 2024 · similarity 0.87)
- HumanNorm: Learning Normal Diffusion Model for High-quality and Realistic 3D Human Generation (CVPR 2024 · similarity 0.86)
- Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation (CVPR 2024 · similarity 0.85)
- HanDiffuser: Text-to-Image Generation With Realistic Hand Appearances (CVPR 2024 · similarity 0.85)
- RHanDS: Refining Malformed Hands for Generated Images with Decoupled Structure and Style Guidance (AAAI 2025 · similarity 0.85)
- ShowMaker: Creating High-Fidelity 2D Human Video via Fine-Grained Diffusion Modeling (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.