Research map / Image generation / Autoregressive and latent generation
HyperHuman: Hyper-Realistic Human Generation with Latent Structural Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HyperHuman, a two-stage framework that jointly denoises RGB, depth, and surface-normal latents with structural expert branches (plus a structure-guided SDXL refiner) and a 340M-image human dataset, HumanVerse, for hyper-realistic pose-conditioned human generation.
Problem
Text-to-image models such as Stable Diffusion and DALL·E 2 often produce human images with incoherent body parts and unnatural poses, and existing pose-controllable methods ignore finer structural cues like depth and surface normals.
Approach
Jointly denoise RGB, depth, and surface-normal latents in one UNet using replicated structural expert branches around a shared backbone, with zero-terminal-SNR noise schedule, shared timesteps, and v-prediction; then a structure-guided SDXL refiner composes the predicted conditions for high-resolution output with random condition dropout for robustness.
Key results
- On zero-shot MS-COCO 2014 human validation at 512x512, HyperHuman reaches FID 17.18 and KID 4.11, versus 22.98 for the next best (SD 2.0) and 23.54 for T2I-Adapter.
- Pose accuracy AP 30.38 and AR 37.84, against 27.54 AP for T2I-Adapter, 26.71 for HumanSD and 20.46 for ControlNet.
- In a user study with 25 participants, users preferred HyperHuman over SDXL 60.45% of the time and over SD 2.1 89.24% of the time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add depth and normal as simultaneous denoising targets instead of denoising RGB only (ablation) | Latent Structural Diffusion Model · MS-COCO 2014 validation human | FID: 21.68 -> 17.18 | Helped |
| Add depth denoising to RGB denoising (ablation) | Latent Structural Diffusion Model · MS-COCO 2014 validation human | FID: 21.68 -> 19.89 | Helped |
| Replicate the full first DownBlock and last UpBlock rather than half of each (ablation) | Structural expert branches · MS-COCO 2014 validation human | FID: 22.85 -> 17.18 | Helped |
| Use the default expert-branch design rather than replicating two DownBlocks and two UpBlocks (ablation) | Structural expert branches · MS-COCO 2014 validation human | FID: 17.94 -> 17.18 | Helped |
| Use zero-terminal SNR with v-prediction instead of default SNR with epsilon-prediction (ablation) | Latent Structural Diffusion Model · MS-COCO 2014 validation human | FID: 17.70 -> 17.18 | Helped |
| Use the same timestep for all modalities instead of different timesteps (ablation) | Latent Structural Diffusion Model · MS-COCO 2014 validation human | FID: 29.36 -> 17.18 | Helped |
| Add random condition dropout during training (ablation) | Structure-Guided Refiner · 512×512 Structure-Guided Refiner ablation | FID: 25.69 -> 12.38 | Helped |
| Add structural conditions to text-only conditioning (ablation) | Structure-Guided Refiner · 512×512 Structure-Guided Refiner ablation | FID: 23.99 -> 12.38 | Helped |
| Add depth and normal conditions to pose-only conditioning (ablation) | Structure-Guided Refiner · 512×512 Structure-Guided Refiner ablation | FID: 20.97 -> 12.38 | Helped |
| Add pose and normal conditions to depth-only conditioning (ablation) | Structure-Guided Refiner · 512×512 Structure-Guided Refiner ablation | FID: 14.97 -> 12.38 | Helped |
| Add pose and depth conditions to normal-only conditioning (ablation) | Structure-Guided Refiner · 512×512 Structure-Guided Refiner ablation | FID: 12.67 -> 12.38 | Helped |
| Add normal conditioning to pose-and-depth conditioning (ablation) | Structure-Guided Refiner · 512×512 Structure-Guided Refiner ablation | FID: 14.98 -> 12.38 | Helped |
Surprising findings
- Denoising RGB together with depth and normal without the full expert-branch design can be worse; fewer replicated layers give better spatial alignment but insufficient per-modality capacity, while excessive replication reduces feature fusion.
- Conditioning the refiner only on surface normals outperformed conditioning on both pose skeleton and depth map in the appendix ablation.
Limitations
- Fails to generate subtle details such as fingers and eyes, limited by existing pose, depth and normal estimators for in-the-wild humans.
- Still requires a body skeleton as input; text-to-pose generation is left for future work.
- Human preference metrics (PickScore, HPS V2) appear biased toward 1024-resolution images, so their gains are marginal.
Experimental setup
Datasets: LAION-2B-en COYO-700M MS-COCO 2014 validation (human subset, 8,236 images) Human-Art
Benchmarks: MS-COCO 2014 validation human subset
Metrics: FID KID FID_CLIP CLIP score AP AR AP_clean AR_clean PickScore HPS V2
Baselines: Stable Diffusion 1.5 Stable Diffusion 2.0 Stable Diffusion 2.1 SDXL DeepFloyd-IF ControlNet T2I-Adapter HumanSD
Builds on: Stable Diffusion SDXL ControlNet v-prediction DDIM
Closest papers by meaning
- HumanNorm: Learning Normal Diffusion Model for High-quality and Realistic 3D Human Generation (CVPR 2024 · similarity 0.90)
- PSHuman: Photorealistic Single-image 3D Human Reconstruction using Cross-Scale Multiview Diffusion and Explicit Remeshing (CVPR 2025 · similarity 0.89)
- PaintHuman: Towards High-Fidelity Text-to-3D Human Texturing via Denoised Score Distillation (AAAI 2024 · similarity 0.88)
- DiffHuman: Probabilistic Photorealistic 3D Reconstruction of Humans (CVPR 2024 · similarity 0.88)
- SyncHuman: Synchronizing 2D and 3D Generative Models for Single-view Human Reconstruction (NeurIPS 2025 · similarity 0.88)
- HumanRef: Single Image to 3D Human Generation via Reference-Guided Diffusion (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.