Research map / 3D vision / 3D generation
DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiffPortrait3D, a zero-shot conditional latent diffusion method that synthesizes 3D-consistent novel views of a single in-the-wild portrait using appearance reference attention, a camera-pose control module, and cross-view attention with 3D-aware noise.
Problem
Synthesizing photo-realistic, identity- and expression-preserving novel views of a portrait from a single unposed in-the-wild image, without per-subject optimization or fine-tuning, where 3D GANs struggle with out-of-domain portraits.
Approach
Decouples appearance and camera control on a frozen Stable Diffusion UNet: a trainable reference branch injects appearance via concatenated self-attention, a ControlNet-style view module takes a cross-identity EG3D render as pose condition, and cross-view attention plus 3D-aware initial noise from a coarse 3D NVS proxy enforce multi-view consistency.
Key results
- On NeRSemble novel view synthesis, the method reports LPIPS 0.21 and FID 33.13 versus EG3D-PTI at 0.66 ... (table columns ambiguous); reconstruction ID 0.95 vs 0.70 for best baseline on NeRSemble.
- Ablation shows fine-tuning the appearance reference module reduces LPIPS from 0.55 to 0.28 and FID from 99.09 to 25.77.
- Pose error is slightly worse than the EG3D-PTI baseline, which the authors attribute to camera misalignment between ground truth and EG3D renders.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune the appearance reference module on multi-view images (ablation) | DiffPortrait3D · NeRSemble | LPIPS: 0.55 -> 0.28 | Helped |
| use unaligned rather than aligned reference images (ablation) | DiffPortrait3D · NeRSemble | SSIM: 0.68 -> 0.68 | No clear effect |
| add a view-consistency module (ablation) | DiffPortrait3D · NeRSemble multi-view synthesis | view consistency | Helped |
| use 3D-aware noise instead of random Gaussian noise (ablation) | DiffPortrait3D · NeRSemble view animation | structural coherence | Helped |
Surprising findings
- Pose control accuracy is slightly worse than the 3D GAN baseline, attributed to camera misalignment in evaluation.
- Fine-tuning the appearance reference module is necessary, since removing text cross-attention degrades the frozen SD UNet's reference transfer.
Limitations
- Some identity differences remain from a single reference image.
- Test novel views are generated with EG3D renders, so pose evaluation depends on camera estimation accuracy.
- Relies on a 3D-convolutional NVS proxy for 3D-aware noise generation.
Experimental setup
Datasets: NeRSemble PanoHead synthetic renders EG3D renders in-the-wild portraits
Benchmarks: NeRSemble in-the-wild portraits
Metrics: LPIPS DISTS SSIM ID similarity FID POSE error
Baselines: GOAE TriPlaneNet EG3D-PTI Zero-1-to-3
Builds on: Stable Diffusion ControlNet AnimateDiff Zero-1-to-3
Closest papers by meaning
- RealPortrait: Realistic Portrait Animation with Diffusion Transformers (AAAI 2025 · similarity 0.91)
- DiffPortrait360: Consistent Portrait Diffusion for 360 View Synthesis (CVPR 2025 · similarity 0.89)
- DiffHuman: Probabilistic Photorealistic 3D Reconstruction of Humans (CVPR 2024 · similarity 0.89)
- Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation (CVPR 2024 · similarity 0.88)
- MultiDiff: Consistent Novel View Synthesis from a Single Image (CVPR 2024 · similarity 0.88)
- Coherent 3D Scene Diffusion From a Single RGB Image (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.