Research map / Video / Video generation
Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Real3D-Portrait builds a one-shot 3D talking portrait system that pre-trains a large image-to-plane model on EG3D multi-view data, adds a PNCC-conditioned motion adapter, a head-torso-background super-resolution model, and a VAE audio-to-motion model for video- and audio-driven generation.
Problem
One-shot 3D talking portrait generation from a single unseen image needs both accurate 3D avatar reconstruction and stable motion-driven animation, plus natural torso and background rendering for realistic video.
Approach
Pre-train a feed-forward image-to-tri-plane model by distilling multi-view supervision from EG3D, then animate it with a lightweight motion adapter that predicts a residual diff-plane from PNCC, and render head, torso, and background separately with alpha-blending fusion, plus a generic VAE-based audio-to-motion model.
Key results
- Video-driven same-identity reenactment on CelebV-HQ: Real3D-Portrait reaches PSNR 18.95 and FID 37.50 versus 16.47 and 42.96 for Face-vid2vid, the best one-shot baseline in that table.
- Audio-driven: CSIM 0.763, FID 43.02, AED 0.146 and Sync 6.565, outperforming one-shot MakeItTalk and PC-AVS and approaching person-specific RAD-NeRF (CSIM 0.784, FID 39.45).
- MOS user study (20 participants): Real3D-Portrait (VD) scores 4.08 identity, 4.16 visual quality and 4.13 lip sync, above the one-shot video baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add multi-view I2P pre-training (ablation) | Real3D-Portrait · Video-driven cross-identity reenactment | FID: 65.32 -> 42.37 | Helped |
| Fine-tune the pre-trained I2P model on video data (ablation) | Real3D-Portrait · Video-driven cross-identity reenactment | AED: 0.233 -> 0.138 | Helped |
| Increase I2P model size from 40M to 87M parameters (ablation) | Real3D-Portrait · Video-driven cross-identity reenactment | FID: 45.48 -> 42.37 | Helped |
| Increase I2P model size from 87M to 200M parameters (ablation) | Real3D-Portrait · Video-driven cross-identity reenactment | FID: 43.15 -> 42.37 | No clear effect |
| Add Laplacian loss to the motion adapter (ablation) | Real3D-Portrait · Video-driven cross-identity reenactment | AED: 0.158 -> 0.138 | Helped |
| Use predefined keypoints instead of unsupervised keypoints (ablation) | Real3D-Portrait HTB-SR · Video-driven cross-identity reenactment | FID: 44.86 -> 42.37 | Helped |
| Use alpha-blending fusion instead of channel-wise concatenation (ablation) | Real3D-Portrait HTB-SR · Video-driven cross-identity reenactment | FID: 46.38 -> 42.37 | Helped |
| Add KNN background inpainting (ablation) | Real3D-Portrait HTB-SR · Video-driven cross-identity reenactment | FID: 43.95 -> 42.37 | Helped |
| Use Real3D-Portrait instead of Face-vid2vid | Video-driven reenactment · Same-identity reenactment | PSNR: 16.47 -> 18.95 | Helped |
| Use Real3D-Portrait instead of RAD-NeRF | Audio-driven generation · Audio-driven generation | CSIM: 0.784 -> 0.763 | Hurt |
| Use Real3D-Portrait instead of RAD-NeRF | Audio-driven generation · Audio-driven generation | Sync: 3.779 -> 6.565 | Helped |
Surprising findings
- Naive channel-wise concatenation of head, torso and background features produces hollow hair artifacts and blurry boundaries, which alpha-blending fusion fixes.
- Removing the Laplacian loss slightly improves FID (42.66 vs 42.37 for full model is close) and LPIPS-type metrics are mixed, suggesting limited gains on some metrics from the regularizer.
Limitations
- Relies on KNN-based background inpainting, which the authors note is simple and not state of the art.
- Comparison with person-specific RAD-NeRF is not a fair one-shot comparison, and the method only approaches its quality.
- Torso branch assumes mainly translational torso motion, so rotation of torso is not modeled.
Experimental setup
Datasets: CelebV-HQ VoxCeleb2 EG3D synthetic multi-view data
Benchmarks: CelebV-HQ validation split out-of-domain internet images with audio clips
Metrics: L1 PSNR SSIM LPIPS FID CSIM AED APD AKD Sync
Baselines: Face-vid2vid OTAvatar HiDe-NeRF MakeItTalk PC-AVS RAD-NeRF DaGAN TPS DPE LIA MCNet
Builds on: EG3D Face-vid2vid AD-NeRF HuBERT PNCC
Closest papers by meaning
- VASA-3D: Lifelike Audio-Driven Gaussian Head Avatars from a Single Image (NeurIPS 2025 · similarity 0.90)
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.89)
- DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation (ICLR 2025 · similarity 0.88)
- Teller: Real-Time Streaming Audio-Driven Portrait Animation with Autoregressive Motion Generation (CVPR 2025 · similarity 0.88)
- CAP4D: Creating Animatable 4D Portrait Avatars with Morphable Multi-View Diffusion Models (CVPR 2025 · similarity 0.88)
- IM-Portrait: Learning 3D-aware Video Diffusion for Photorealistic Talking Heads from Monocular VideosC (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.