Research map / 3D vision / Face and shape
VOODOO 3D: Volumetric Portrait Disentanglement For One-Shot 3D Head Reenactment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VOODOO 3D, a real-time, 3D-aware one-shot head reenactment method that disentangles source appearance and driver expression fully in a shared tri-plane volumetric space, without linear 3DMM models.
Problem
One-shot 3D-aware head reenactment suffers from identity leakage and unnatural expressions because existing methods disentangle expression using linear 3DMM/FLAME face models, limiting fidelity and view consistency for holographic teleconferencing.
Approach
Lift source and driver images into a canonical tri-plane space via a fine-tuned Lp3D 3D-lifting module, then predict an expression tri-plane residual from RGB features of frontalized source and driver (no 3DMM), trained self-supervised on in-the-wild video with a cross-identity GAN regularizer and iterative module fine-tuning.
Key results
- On HDTF self-reenactment, the method reaches PSNR 22.83 and LPIPS 0.168, versus 21.10 and 0.291 for MegaPortraits.
- On HDTF cross-reenactment, FID is 36.4 versus 61.7 for MegaPortraits, and ECMD is 0.754 versus 0.771 for MegaPortraits (wait—see table: ECMD 0.754 vs 0.771).
- Runs at 31.9 fps per view on an RTX 4090, with up to 45 views rendered in parallel for holographic displays.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Lp3D on real data (ablation) | VOODOO 3D lifting module · CelebA-HQ cross-reenactment | CSIM: 0.548 -> 0.670 | Helped |
| Fine-tune Lp3D on real data (ablation) | VOODOO 3D lifting module · CelebA-HQ cross-reenactment | ECMD: 0.82 -> 0.76 | Helped |
| Add source and driver frontalization (ablation) | VOODOO 3D expression module · CelebA-HQ cross-reenactment | ECMD: 1.01 -> 0.79 | Helped |
| Add source and driver frontalization (ablation) | VOODOO 3D expression module · CelebA-HQ cross-reenactment | CSIM: 0.668 -> 0.608 | Hurt |
| Add cross-identity regularization (ablation) | VOODOO 3D expression module · CelebA-HQ cross-reenactment | CSIM: 0.570 -> 0.608 | Helped |
| Add cross-identity regularization (ablation) | VOODOO 3D expression module · CelebA-HQ cross-reenactment | ECMD: 0.97 -> 0.79 | Helped |
| Use VOODOO 3D instead of MegaPortraits | Head reenactment · HDTF self-reenactment | FID: 52.0 -> 40.5 | Helped |
| Use VOODOO 3D instead of MegaPortraits | Head reenactment · HDTF cross-reenactment | FID: 61.7 -> 36.4 | Helped |
| Use VOODOO 3D instead of MegaPortraits | Head reenactment · CelebA-HQ cross-reenactment | CSIM: 0.647 -> 0.608 | Hurt |
Surprising findings
- Without frontalization of source and driver, the model ignores the driver expression and keeps the source expression.
- End-to-end training of the full model failed even with a lower learning rate and pretrained Lp3D weights, requiring iterative module fine-tuning instead.
Limitations
- Side-view source images beyond about 90 degrees yield plausible frontal faces but likeness cannot be guaranteed.
- Highly stylized portraits (cartoons) produce photorealistic elements such as teeth inconsistent in style.
- Accessories like glasses or braces may disappear or look different due to training data limits.
Experimental setup
Datasets: CelebV-HQ EG3D synthetic data HDTF CelebA-HQ
Benchmarks: HDTF CelebA-HQ
Metrics: PSNR SSIM LPIPS NAKD ECMD FID CSIM
Baselines: ROME StyleHeat OTAvatar MegaPortraits HeadNeRF LPR
Builds on: EG3D Lp3D MegaPortraits LPR
Closest papers by meaning
- PerformRecast: Expression and Head Pose Disentanglement for Portrait Video Editing (CVPR 2026 · similarity 0.88)
- Generalizable One-shot 3D Neural Head Avatar (NeurIPS 2023 · similarity 0.88)
- Coherent 3D Portrait Video Reconstruction via Triplane Fusion (CVPR 2025 · similarity 0.87)
- Learning Dense Correspondence for NeRF-Based Face Reenactment (AAAI 2024 · similarity 0.86)
- High-Fidelity 3D Head Avatars Reconstruction through Spatially-Varying Expression Conditioned Neural Radiance Field (AAAI 2024 · similarity 0.86)
- DiffPortrait360: Consistent Portrait Diffusion for 360 View Synthesis (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.