Research map / 3D vision / Face and shape
En3D: An Enhanced Generative Model for Sculpting 3D Humans from 2D Synthetic Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
En3D trains a triplane-based 3D human generator on 1.4M ControlNet-synthesized multi-view human images with known camera poses, then sculpts geometry with multi-view normal constraints on DMTET and textures a UV map with semantic partitioning and a differentiable rasterizer.
Problem
Learning high-quality, geometrically accurate, and diverse 3D human avatars from 2D data without relying on scarce 3D scans or imbalanced 2D human image collections with imprecise SMPL pose priors.
Approach
Synthesize balanced multi-view human images with pose-conditioned ControlNet from projected 3D skeletons (giving exact camera parameters), train an EG3D-style triplane generator with patch-composed rendering on them, and add inference-time geometry sculpting (DMTET with multi-view normal loss) and UV texture optimization.
Key results
- FID of 2.73 versus 7.48 for EG3D, 10.93 for AG3D and 15.91 for EVA3D on 360-degree image quality.
- Normal-map FID of 5.62, better than AG3D-syn (8.64) and EVA3D-syn (9.57).
- IS-360 of 3.43, above all compared methods (best baseline 3.28 for AG3D).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use En3D instead of the strongest listed baseline, EG3D | 360° human image generation · 360° human images | FID: 7.48 -> 2.73 | Helped |
| Use En3D instead of the strongest listed baseline, AG3D | 360° human image generation · 360° human images | IS-360: 3.28 -> 3.43 | Helped |
| Use En3D instead of the strongest listed baseline, AG3D-syn | 3D human shape generation · Rendered normals | Normal: 8.64 -> 5.62 | Helped |
| Use En3D instead of the strongest listed baseline, EVA3D or EVA3D-syn | 3D human generation · Multi-view faces | ID: 0.72 -> 0.74 | Helped |
| Use accurate physical parameters instead of estimated camera and pose parameters (ablation) | En3D 3D generative modeling · 360° human images | FID: 4.28 -> 2.73 | Helped |
| Use accurate physical parameters instead of estimated camera and pose parameters (ablation) | En3D 3D generative modeling · 360° human images | IS-360: 3.31 -> 3.43 | Helped |
| Add patch-composed rendering (ablation) | En3D 3D generative modeling · 360° human images | FID: 3.26 -> 2.73 | Helped |
| Add patch-composed rendering (ablation) | En3D 3D generative modeling · 360° human images | IS-360: 3.35 -> 3.43 | Helped |
| Add multi-view normal-guided geometric sculpting (ablation) | En3D geometry · 3D human shape refinement | shape quality | Helped |
| Add explicit UV texturing (ablation) | En3D guided synthesis · Image-guided 3D human synthesis | local texture fidelity | Helped |
Surprising findings
- Ablation shows that removing the patch-composed rendering raises FID only modestly (2.73 to 3.26), while estimated physical parameters instead of synthesized ones raise FID to 4.28.
Limitations
- Relies on a 2D diffusion model and ControlNet for synthetic data, so quality is bounded by the synthetic image distribution.
- Hands can be noisy and are optionally replaced with SMPL-X hands.
- Text-guided synthesis requires PTI inversion of a generated image.
Experimental setup
Datasets: synthetic multi-view human images generated with ControlNet
Benchmarks: FID (360-degree views) IS-360 normal FID ID consistency
Metrics: FID IS-360 Normal ID
Baselines: EVA3D AG3D EG3D
Builds on: EG3D ControlNet DMTET AG3D EVA3D
Closest papers by meaning
- Human-3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Models (NeurIPS 2024 · similarity 0.89)
- HumanNorm: Learning Normal Diffusion Model for High-quality and Realistic 3D Human Generation (CVPR 2024 · similarity 0.89)
- Arc2Avatar: Generating Expressive 3D Avatars from a Single Image via ID Guidance (CVPR 2025 · similarity 0.88)
- PEGASUS: Personalized Generative 3D Avatars with Composable Attributes (CVPR 2024 · similarity 0.88)
- DreamAvatar: Text-and-Shape Guided 3D Human Avatar Generation via Diffusion Models (CVPR 2024 · similarity 0.88)
- Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.