Research map / 3D vision / Face and shape
Coherent 3D Portrait Video Reconstruction via Triplane Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a triplane fusion encoder that fuses a canonical near-frontal triplane from a reference image with per-frame lifted triplanes (via an undistorter and fuser) to produce temporally coherent, faithful 3D portrait videos, trained only on synthetic data from Next3D.
Problem
Single-view 3D portrait video reconstruction for telepresence is temporally inconsistent when lifting each frame independently, while self-reenactment methods lose the user's per-frame appearance such as expressions and lighting.
Approach
Fuse a canonical triplane lifted from a near-frontal reference image with the raw per-frame triplane, using a triplane undistorter (SPyNet-predicted 2D warping supervised by pseudo-ground-truth triplanes) and a visibility-aware fuser (RVRT backbone) to recover occluded regions while keeping dynamic appearance.
Key results
- On NeRSemble, the method reaches 22.770 PSNR and 0.219 LPIPS in synthesis quality, beating LP3D (22.331 / 0.223).
- Novel-view PSNR drop vs. synthesis quality is 0.330 dB for the method versus 0.806 dB for LP3D, indicating better generalization to novel viewpoints.
- Identity distance 0.187 and expression error 0.158 are best among compared methods (LP3D 0.215 and 0.168).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fuse the Undistorted input triplane with the canonical triplane | LP3D · NeRSemble | Synthesis Quality PSNR: 22.331 -> 22.770 | Helped |
| Add the Undistorter only (ablation) | LP3D · NeRSemble | IVV (PSNR): 1.025 -> 0.907 | Helped |
| Add the Undistorter only (ablation) | LP3D · NeRSemble | Synthesis Quality PSNR: 22.331 -> 22.196 | Hurt |
| Add the Fuser only (ablation) | LP3D · NeRSemble | IVV (PSNR): 1.025 -> 0.559 | Helped |
| Add the Fuser only (ablation) | LP3D · NeRSemble | Synthesis Quality PSNR: 22.331 -> 22.265 | Hurt |
| Replace the Undistorter with optical flow warping (ablation) | Ours · NeRSemble | Synthesis Quality PSNR: 22.770 -> 22.085 | Hurt |
| Remove shoulder augmentation during training (ablation) | Ours · NeRSemble | expression error: 0.187 -> 0.244 | Hurt |
| Add an adversarial GAN loss (ablation) | Ours · NeRSemble | LPIPS: 0.219 -> 0.207 | Helped |
| Add an adversarial GAN loss (ablation) | Ours · NeRSemble | Synthesis Quality PSNR: 22.770 -> 22.074 | Hurt |
| Use the authors' modified LP3D instead of original LP3D (ablation) | LP3D · NeRSemble | Synthesis Quality PSNR: 18.721 -> 22.331 | Helped |
Surprising findings
- Undistorter-only and Fuser-only variants improve view robustness but lower PSNR, and only the combination gives the best overall score.
- Replacing the undistorter with optical flow alignment worsens reconstruction, since the goal is identity correction rather than making triplanes identical.
Limitations
- Single reference image only; multiple references may help but are not explored.
- Fusing an extreme side view with very different expression can produce blurry reconstructions due to triplane alignment ambiguity.
- Current runtime is slower than real time.
Experimental setup
Datasets: FFHQ NeRSemble Next3D synthetic data
Benchmarks: NeRSemble
Metrics: PSNR LPIPS ArcFace identity distance expression L2 distance IVV NVV
Baselines: LP3D GPAvatar Li et al. (2024) VIVE3D
Builds on: LP3D Next3D RVRT SPyNet EG3D
Closest papers by meaning
- CAP4D: Creating Animatable 4D Portrait Avatars with Morphable Multi-View Diffusion Models (CVPR 2025 · similarity 0.87)
- VOODOO 3D: Volumetric Portrait Disentanglement For One-Shot 3D Head Reenactment (CVPR 2024 · similarity 0.87)
- Real-time 3D-aware Portrait Video Relighting (CVPR 2024 · similarity 0.86)
- MVD-Fusion: Single-view 3D via Depth-consistent Multi-view Generation (CVPR 2024 · similarity 0.86)
- 3DToonify: Creating Your High-Fidelity 3D Stylized Avatar Easily from 2D Portrait Images (CVPR 2024 · similarity 0.86)
- Synthetic Prior for Few-Shot Drivable Head Avatar Inversion (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.