Research map / 3D vision / Face and shape
Portrait4D: Learning One-Shot 4D Head Avatar Synthesis using Synthetic Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns a part-wise 4D generative head model (GenHead) from monocular images, uses it to synthesize large-scale multi-view 4D data, and trains a transformer-based triplane reconstructor (Portrait4D) on that data for one-shot 4D head avatar synthesis.
Problem
One-shot 4D head avatar synthesis from a single portrait is hard because monocular 3DMM reconstructions are inaccurate and monocular video training lacks multi-view supervision and background separation.
Approach
Train a part-wise 3D-aware GAN with FLAME-derived deformation fields to generate multi-view, multi-motion synthetic 4D head data, then train a feed-forward transformer triplane reconstructor on that data with a disentangled learning strategy that randomly switches off motion cross-attention so the model learns static reconstruction too, improving generalization to real images.
Key results
- Self-reenactment on VFHQ at 512^2: LPIPS_h 0.181 and FID 43.0, versus 0.187/52.6 for the best 2D baseline Face-vid2vid (per the table, ours is best on FID and head LPIPS).
- Cross-reenactment on VFHQ: FID 54.8 vs. 78.0 for PIRenderer, with best APD of 1.020 among compared methods.
- Ablation on FFHQ shows removing the disentangled learning strategy and static data hurts identity similarity (0.239 baseline vs 0.703 final), and real-data training degrades geometry (APD 7.537).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add disentangled learning (ablation) | Portrait4D baseline · FFHQ reconstruction | LPIPSh: 0.244 -> 0.238 | Helped |
| add the de-expression module Φde (ablation) | Portrait4D baseline · FFHQ reconstruction | LPIPSh: 0.244 -> 0.247 | Hurt |
| combine disentangled learning and Φde (ablation) | Portrait4D baseline · FFHQ reconstruction | LPIPSh: 0.244 -> 0.212 | Helped |
| add static data for 3D reconstruction during disentangled learning (ablation) | Portrait4D with disentangled learning and Φde · FFHQ reconstruction | LPIPSh: 0.212 -> 0.188 | Helped |
| replace the motion features from [62] with FLAME codes [β, γeye, γjaw] (ablation) | Portrait4D · FFHQ reconstruction | LPIPSh: 0.188 -> 0.213 | Hurt |
| use detected opacity images instead of GenHead-rendered opacity images (ablation) | Portrait4D · FFHQ reconstruction | LPIPSh: 0.188 -> 0.198 | Hurt |
| train with monocular real data instead of multi-view synthetic data (ablation) | Portrait4D · FFHQ reconstruction | APD: 1.022 -> 7.537 | Hurt |
Surprising findings
- Training on real monocular images instead of multi-view synthetic data dramatically degrades learned geometry (APD rises to 7.537), supporting the value of synthetic multi-view data.
- Using FLAME expression codes as motion features improves AED/APD but substantially reduces reconstruction fidelity due to identity leakage.
Limitations
- Relies on synthetic data whose photorealism still shows domain gaps with real images.
- 3DMM-based deformation and neck pose derived from FLAME may be imprecise; limitations discussed only in supplementary material.
Experimental setup
Datasets: FFHQ VFHQ
Benchmarks: VFHQ FFHQ
Metrics: LPIPS FID ID similarity AED APD
Baselines: PIRenderer Face-vid2vid StyleHEAT ROME OTAvatar HideNeRF
Builds on: EG3D FLAME Face-vid2vid OTAvatar LivePortrait
Closest papers by meaning
- CAP4D: Creating Animatable 4D Portrait Avatars with Morphable Multi-View Diffusion Models (CVPR 2025 · similarity 0.89)
- AvatarArtist: Open-Domain 4D Avatarization (CVPR 2025 · similarity 0.88)
- Synthetic Prior for Few-Shot Drivable Head Avatar Inversion (CVPR 2025 · similarity 0.88)
- FA-LAM: Focus-Aware Large Avatar Model for One-Shot 4D Animatable Gaussian Head (NeurIPS 2026 · similarity 0.86)
- Zero-1-to-A: Zero-Shot One Image to Animatable Head Avatars Using Video Diffusion (CVPR 2025 · similarity 0.86)
- CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.