Research map / 3D vision / 3D generation
HumanNorm: Learning Normal Diffusion Model for High-quality and Realistic 3D Human Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HumanNorm, which fine-tunes Stable Diffusion into normal- and depth-adapted models and a normal-aligned (ControlNet) texture model to drive a two-stage text-to-3D human generation with progressive geometry and multi-step SDS.
Problem
Text-to-3D human generation with diffusion-guided SDS yields Janus artifacts, smooth geometry, fake 3D details and over-saturated, cartoon-like textures because text-to-image models lack 3D geometry understanding.
Approach
Fine-tune a T2I diffusion model to generate normal maps (and depth maps) conditioned on view-dependent and body-aware text, use normal-conditioned ControlNet for texture, and optimize with normal/depth SDS, progressive DMTET positional-encoding masking plus SDF loss, and a multi-step SDS with perceptual loss.
Key results
- HumanNorm achieves FID 92.5 versus 111.3 for DreamHuman and 120.0 for TADA on 30 prompts.
- HumanNorm reaches CLIP score 31.70, above TADA (30.65) and DreamHuman (30.15).
- Generation takes about 2 hours per prompt on a single RTX 3090 (15K geometry + 10K texture iterations).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use normal-adapted and depth-adapted diffusion models instead of a text-to-image diffusion model for geometry generation (ablation) | HumanNorm · 3D human geometry generation | geometry quality | Helped |
| Use a normal-aligned diffusion model instead of a text-to-image diffusion model for texture generation (ablation) | HumanNorm · 3D human texture generation | texture detail and geometry alignment | Helped |
| Use multi-step SDS loss instead of only SDS loss for texture generation (ablation) | HumanNorm · 3D human texture generation | appearance realism | Helped |
| Add depth SDS loss to normal SDS loss (ablation) | HumanNorm · 3D human geometry generation | geometry artifacts | Helped |
Limitations
- Generated humans need a rigged skeleton for animation; not directly animatable without SMPL-X integration.
- Finer body details such as fingers are not well captured.
- Generated texture may contain undesired shading, lacking PBR material estimation.
Experimental setup
Datasets: 3D human scan renderings (normal and depth maps)
Benchmarks: 30 text prompts (in-house)
Metrics: FID CLIP score user study
Baselines: DreamFusion LatentNeRF TEXTure Fantasia3D DreamHuman TADA
Builds on: DreamFusion Fantasia3D DMTET ControlNet Stable Diffusion
Closest papers by meaning
- HyperHuman: Hyper-Realistic Human Generation with Latent Structural Diffusion (ICLR 2024 · similarity 0.90)
- Human-3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Models (NeurIPS 2024 · similarity 0.89)
- PaintHuman: Towards High-Fidelity Text-to-3D Human Texturing via Denoised Score Distillation (AAAI 2024 · similarity 0.89)
- En3D: An Enhanced Generative Model for Sculpting 3D Humans from 2D Synthetic Data (CVPR 2024 · similarity 0.89)
- HumanRef: Single Image to 3D Human Generation via Reference-Guided Diffusion (CVPR 2024 · similarity 0.89)
- MagicMan: Generative Novel View Synthesis of Humans with 3D-Aware Diffusion and Iterative Refinement (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.