Research map / 3D vision / 3D generation
Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Integrates a 3D morphable model (FLAME/SMPL-X/bilinear) into the SyncDreamer multi-view diffusion model via a morphable noise volume and view-frustum conditioning, enabling 3D-consistent, animatable novel view and expression synthesis of humans from a single image.
Problem
Single-image photorealistic human avatar creation that is both 3D-consistent and controllable by facial expression or body pose; existing view-consistent diffusion models cannot be explicitly controlled and 2D-rasterization conditioning lacks multi-view consistency.
Approach
Conditions a multi-view diffusion UNet on a 3D morphable mesh by unprojecting noisy image features onto mesh vertices, processing them with a sparse 3D ConvNet into a 3DMM-aware feature volume, and sampling view-frustum features injected via depth-wise cross-attention; a cross-expression training scheme disentangles reconstruction from animation.
Key results
- On FaceScape novel view synthesis, the method reaches LPIPS 0.1653 and PCK@0.2 95.80, versus SyncDreamer's 0.1854 and 94.07.
- On THuman 2.0 full-body novel view synthesis, FID improves to 30.25 from SyncDreamer's 41.33 and LPIPS to 0.0625 from 0.0960.
- On held-out 'jaw right' novel expression synthesis, LPIPS 0.1693 and FID 14.34 beat DiffusionRig* (42.93 FID) and MoFaNeRF (32.92 FID).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Condition diffusion on a 3D morphable model | SyncDreamer for human face novel-view synthesis · FaceScape test subjects | LPIPS: 0.1854 -> 0.1653 | Helped |
| Condition diffusion on a 3D morphable model | SyncDreamer for human face novel-view synthesis · FaceScape test subjects | FID: 6.05 -> 6.73 | Hurt |
| Condition diffusion on a 3D morphable model | SyncDreamer for human face novel-view synthesis · FaceScape test subjects | PCK@0.2: 94.07 -> 95.80 | Helped |
| Condition diffusion on a 3D morphable model | SyncDreamer for full-body novel-view synthesis · THuman 2.0 test set | LPIPS: 0.0960 -> 0.0625 | Helped |
| Condition diffusion on a 3D morphable model | SyncDreamer for full-body novel-view synthesis · THuman 2.0 test set | SSIM: 0.8826 -> 0.9181 | Helped |
| Use cross-expression training with a 3D morphable model | Morphable Diffusion · FaceScape held-out jaw right expression | LPIPS: 0.2534 -> 0.1693 | Helped |
| Use cross-expression training with a 3D morphable model | Morphable Diffusion · FaceScape held-out jaw right expression | Re-ID: 97.67 -> 99.89 | Helped |
Surprising findings
- Zero-1-to-3 achieves a strong FID (10.97) on faces yet has very low PCK (4.69), indicating good image quality but poor alignment with target views.
Limitations
- Poor generalization to out-of-distribution ethnicities and hair types due to FaceScape bias toward Asian subjects wearing red caps.
- Does not generalize well to out-of-distribution camera parameters, requiring an external NeRF-based method for full 3D reconstruction.
- Output resolution limited to 256x256, giving limited detail for full bodies.
Experimental setup
Datasets: FaceScape THuman 2.0 Objaverse
Benchmarks: FaceScape THuman 2.0
Metrics: LPIPS SSIM FID PCK@0.2 Re-ID accuracy
Baselines: pixelNeRF Zero-1-to-3 SSD-NeRF SyncDreamer DiffusionRig DECA MoFaNeRF
Builds on: SyncDreamer Zero-1-to-3 DiffusionRig SparseConvNet
Closest papers by meaning
- Human-3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Models (NeurIPS 2024 · similarity 0.92)
- CAP4D: Creating Animatable 4D Portrait Avatars with Morphable Multi-View Diffusion Models (CVPR 2025 · similarity 0.91)
- DiffusionAvatars: Deferred Diffusion for High-fidelity 3D Head Avatars (CVPR 2024 · similarity 0.91)
- Zero-1-to-A: Zero-Shot One Image to Animatable Head Avatars Using Video Diffusion (CVPR 2025 · similarity 0.90)
- AvatarVerse: High-Quality & Stable 3D Avatar Creation from Text and Pose (AAAI 2024 · similarity 0.89)
- VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.