Research map / 3D vision / 3D generation
HumanRef: Single Image to 3D Human Generation via Reference-Guided Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HumanRef, a coarse-to-fine SDS-based framework that generates 3D clothed humans from a single image using reference-guided SDS (Ref-SDS) with region-aware attention on Stable Diffusion 1.5.
Problem
Generating a 3D clothed human from a single reference image requires plausible invisible-view textures and geometry while staying consistent with the reference; 3D-data-trained models produce blurry textures and SDS-based text-guided methods lose reference texture details.
Approach
Injects reference-image features into the diffusion U-Net's self-attention during SDS (Ref-SDS) and adds region-aware attention masks from human parsing so that body regions attend to corresponding reference regions, allowing a lower CFG scale and more photorealistic, view-consistent textures.
Key results
- On 50 in-the-wild human images, HumanRef achieves CLIP score 90.0% and contextual distance 1.969 versus 85.3% and 2.882 for TeCH.
- On THuman2.0, HumanRef reaches PSNR 25.70 and SSIM 0.927 versus 24.80 and 0.914 for TeCH, with comparable normal error to ECON.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HumanRef instead of TeCH | 3D clothed human generation · 2D human images | LPIPS: 0.044 -> 0.032 | Helped |
| Use HumanRef instead of TeCH | 3D clothed human generation · 2D human images | Contextual distance: 2.882 -> 1.969 | Helped |
| Use HumanRef instead of TeCH | 3D clothed human generation · 2D human images | CLIP Score: 85.3% -> 90.0% | Helped |
| Use HumanRef instead of TeCH | 3D human generation · CAPE | PSNR: 27.4912 -> 28.1453 | Helped |
| Use HumanRef instead of TeCH | 3D human generation · CAPE | LPIPS: 0.0749 -> 0.0613 | Helped |
| Use HumanRef instead of ECON | 3D human generation · THuman2.0 | Normal: 0.0629 -> 0.0648 | Hurt |
| Use HumanRef instead of TeCH | 3D human generation · THuman2.0 | PSNR: 24.8015 -> 25.6963 | Helped |
| Replace Ref-SDS with SDS (ablation) | HumanRef · novel views | texture consistency and realism | Hurt |
| Replace Ref-SDS with VSD (ablation) | HumanRef · novel views | texture consistency with input | Hurt |
| Add region-aware attention (ablation) | HumanRef Ref-SDS · novel views | texture plausibility | Helped |
Limitations
- Limitations are deferred to the supplementary material, which is not included in the provided text.
- Relies on estimated text captions, SMPL-X fitting, and normal estimates, so errors in these preprocessing steps may propagate.
Experimental setup
Datasets: CAPE THuman2.0
Benchmarks: CAPE THuman2.0
Metrics: LPIPS Contextual distance CLIP score L2 normal error PSNR SSIM
Baselines: PIFu PaMIR TeCH FOF D-IF ECON SDS VSD
Builds on: DreamFusion ProlificDreamer Stable Diffusion ECON Make-It-3D
Closest papers by meaning
- GaussianIP: Identity-Preserving Realistic 3D Human Generation via Human-Centric Diffusion Prior (CVPR 2025 · similarity 0.89)
- HumanNorm: Learning Normal Diffusion Model for High-quality and Realistic 3D Human Generation (CVPR 2024 · similarity 0.89)
- HyperHuman: Hyper-Realistic Human Generation with Latent Structural Diffusion (ICLR 2024 · similarity 0.88)
- ConTex-Human: Free-View Rendering of Human from a Single Image with Texture-Consistent Synthesis (CVPR 2024 · similarity 0.88)
- HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splatting (CVPR 2024 · similarity 0.87)
- SiTH: Single-view Textured Human Reconstruction with Image-Conditioned Diffusion (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.