Research map / 3D vision / Face and shape
DreamAvatar: Text-and-Shape Guided 3D Human Avatar Generation via Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates 3D human avatars from text prompts and SMPL shape/pose using a NeRF optimized via VSD with Stable Diffusion, a dual canonical/posed observation space, and joint full-body and zoomed-in head losses.
Problem
Text-guided 3D generation methods lack human body structure constraints, producing malformed geometry, poor textures, and multi-face Janus artifacts when generating controllable-pose 3D human avatars.
Approach
Combines SMPL-derived density fields and an SMPL-based learnable deformation field to jointly optimize a canonical (A-pose) space and a posed space through a shared NeRF, plus a zoomed-in head VSD loss with landmark ControlNet and a back-view token to reduce Janus artifacts.
Key results
- User study with 25 volunteers over 20 examples ranks DreamAvatar best on geometry quality, texture quality, and text consistency against eight baselines.
- Trains each prompt for 10,000 iterations on a single NVIDIA A40 GPU.
- Ablations show removing SMPL-derived density, head VSD loss, or the dual observation space degrades geometry, texture, and Janus behavior qualitatively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SMPL-derived density fields (ablation) | DreamAvatar · 3D human avatar generation | geometry and texture quality | Helped |
| Add the zoomed-in head VSD loss (ablation) | DreamAvatar · 3D human avatar generation | head quality | Helped |
| Use dual observation spaces instead of only the canonical space (ablation) | DreamAvatar · 3D human avatar generation | overall generation performance | Helped |
| Use dual observation spaces instead of only the posed space without a deformation field (ablation) | DreamAvatar · 3D human avatar generation | overall generation performance | Helped |
| Use DreamAvatar rather than the compared SOTA methods | DreamAvatar · User study | geometry quality | Helped |
| Use DreamAvatar rather than the compared SOTA methods | DreamAvatar · User study | texture quality | Helped |
| Use DreamAvatar rather than the compared SOTA methods | DreamAvatar · User study | text consistency | Helped |
Limitations
- No animation support in the current implementation.
- Inherits biases from the pretrained diffusion model, performing better on frequently seen subjects.
- Evaluation is mainly qualitative and user-study based, without quantitative metrics reported in the text.
Experimental setup
Datasets: SMPL parametric model
Benchmarks: text-prompt avatar generation set (qualitative comparisons)
Metrics: user study rating (1-8) for geometry quality, texture qualit
Baselines: DreamFusion Latent-NeRF Fantasia3D Magic3D ProlificDreamer DreamWaltz AvatarCLIP TADA!
Builds on: DreamFusion ProlificDreamer Magic3D HumanNeRF threestudio
Closest papers by meaning
- AvatarVerse: High-Quality & Stable 3D Avatar Creation from Text and Pose (AAAI 2024 · similarity 0.90)
- Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation (CVPR 2024 · similarity 0.89)
- AvatarGO: Zero-shot 4D Human-Object Interaction Generation and Animation (ICLR 2025 · similarity 0.88)
- En3D: An Enhanced Generative Model for Sculpting 3D Humans from 2D Synthetic Data (CVPR 2024 · similarity 0.88)
- Arc2Avatar: Generating Expressive 3D Avatars from a Single Image via ID Guidance (CVPR 2025 · similarity 0.88)
- Human-3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.