Research map / 3D vision / NeRF and radiance fields
DiffPortrait360: Consistent Portrait Diffusion for 360 View Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds on DiffPortrait3D with a dual appearance module, a ControlNet back-view reference generator, and sequential-view training to produce consistent 360-degree head views from a single portrait.
Problem
Generating view-consistent 360-degree head views from a single stylized or realistic portrait; existing diffusion methods only produce frontal views with inconsistencies, and 3D GAN methods only handle realistic heads.
Approach
A dual appearance module that conditions the diffusion UNet on both the front reference and a generated back-view reference, a ControlNet trained on stylized front/back pairs to hallucinate the back of the head, and training the temporal view-consistency module on continuous view sequences sampled from a 3D GAN instead of random sparse views.
Key results
- On RenderMe360, back-view FID 39.40 vs 91.37 for DiffPortrait3D* and 106.21 for SphereHead+PTI; back-view PSNR 30.92 vs 28.47 for DiffPortrait3D*.
- Frontal-view PSNR 29.44 and CSIM 0.746, outperforming the listed baselines on PSNR, SSIM, LPIPS and FID (frontal CSIM is highest among methods in Table 1).
- Generates a novel view in 5.6 seconds offline, with resulting NeRFs enabling real-time free-viewpoint rendering.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DiffPortrait360 instead of DiffPortrait3D* | DiffPortrait3D* · RenderMe360 frontal view | PSNR: 28.96 -> 29.44 | Helped |
| Use DiffPortrait360 instead of DiffPortrait3D* | DiffPortrait3D* · RenderMe360 back view | FID: 91.37 -> 39.40 | Helped |
| Add dual appearance control with a generated back-view reference (ablation) | DiffPortrait360 · 360-degree view synthesis | appearance consistency | Helped |
| Augment back-view generator training with stylized front-and-back pairs (ablation) | Back-view generation network · Stylized portrait back-view generation | back-view style consistency | Helped |
| Remove sequential view training (ablation) | DiffPortrait360 view-consistency module · 360-degree view synthesis | multi-view consistency | Hurt |
| Remove 3D-aware noise (ablation) | DiffPortrait360 view-consistency module · Novel view synthesis | visual alignment | Hurt |
Surprising findings
- Fine-tuning DiffPortrait3D on full-head data still fails to produce consistent back views, indicating the frontal-only pipeline is the bottleneck.
Limitations
- Relies on synthetic PanoHead/SphereHead sequences for sequential training because RenderMe360 lacks sparse views
- Back-view generator trained on a limited set of 1,000 stylized pairs with Unique3D-generated labels
Experimental setup
Datasets: RenderMe360 PanoHead synthetic data SphereHead synthetic data stylized front-back pairs generated with Unique3D
Benchmarks: RenderMe360
Metrics: PSNR SSIM LPIPS CSIM FID
Baselines: PanoHead + PTI SphereHead + PTI Zero-1-to-3 Unique3D DiffPortrait3D* Rome Itseez3D
Builds on: DiffPortrait3D ControlNet PanoHead Stable Diffusion
Closest papers by meaning
- DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis (CVPR 2024 · similarity 0.89)
- High-Fidelity 3D Head Avatars Reconstruction through Spatially-Varying Expression Conditioned Neural Radiance Field (AAAI 2024 · similarity 0.88)
- DiffusionAvatars: Deferred Diffusion for High-fidelity 3D Head Avatars (CVPR 2024 · similarity 0.87)
- RealPortrait: Realistic Portrait Animation with Diffusion Transformers (AAAI 2025 · similarity 0.87)
- Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation (CVPR 2024 · similarity 0.87)
- CAP4D: Creating Animatable 4D Portrait Avatars with Morphable Multi-View Diffusion Models (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.