Research map / 3D vision / 3D generation
CAP4D: Creating Animatable 4D Portrait Avatars with Morphable Multi-View Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
CAP4D trains a morphable multi-view diffusion model (initialized from Stable Diffusion 2.1) conditioned on FLAME-based 3DMM maps and uses stochastic input/output conditioning to generate hundreds of consistent portrait views, then distills them into a real-time 3D Gaussian splatting avatar.
Problem
Reconstructing photorealistic, animatable 4D portrait avatars from an arbitrary number of reference images (one to hundreds) with consistent visual fidelity, where single-image methods lag multi-view methods in quality and multi-view methods need many diverse images.
Approach
A morphable multi-view diffusion model conditioned on 3DMM pose, expression deformation, and view-ray maps, sampled with stochastic input/output conditioning over random reference and generated image subsets so that any number of references can be used; the generated views then supervise a GaussianAvatars-style real-time avatar with an expression-dependent UV U-Net.
Key results
- Single-image self-reenactment: CAP4D PSNR 21.69, LPIPS 0.311, CSIM 0.633, JOD 5.672, beating GAGAvatar (20.78 PSNR) and Voodoo3D.
- 100-image self-reenactment: CAP4D PSNR 23.30, LPIPS 0.257, CSIM 0.792, JOD 6.15, versus FlashAvatar at 22.87 PSNR and 0.731 CSIM.
- Cross-reenactment user study (23 participants): CAP4D preferred overall over Voodoo3D 97%, Real3D 96%, Portrait4D-v2 85%, GAGAvatar 74%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add expression deformation maps (ablation) | CAP4D MMDM · Nersemble self-reenactment, 10 reference images | PSNR: 21.64 -> 22.54 | Helped |
| add expression deformation maps (ablation) | CAP4D MMDM · Nersemble self-reenactment, 10 reference images | LPIPS: 0.320 -> 0.308 | Helped |
| add view direction conditioning (ablation) | CAP4D MMDM · Nersemble self-reenactment, 10 reference images | PSNR: 22.66 -> 22.54 | No clear effect |
| add stochastic I/O conditioning (ablation) | CAP4D MMDM-only sampling · Nersemble self-reenactment, 10 reference images | PSNR: 23.43 -> 23.82 | Helped |
| add stochastic I/O conditioning (ablation) | CAP4D MMDM-only sampling · Nersemble self-reenactment, 10 reference images | CSIM: 0.755 -> 0.779 | Helped |
| add stochastic I/O conditioning (ablation) | CAP4D MMDM-only sampling · Nersemble self-reenactment, 10 reference images | JOD: 5.92 -> 6.06 | Helped |
| add deformation U-Net (ablation) | CAP4D 4D representation · Nersemble self-reenactment, 10 reference images | PSNR: 21.25 -> 21.69 | Helped |
| add LPIPS reconstruction loss (ablation) | CAP4D 4D representation · Nersemble self-reenactment, 10 reference images | LPIPS: 0.400 -> 0.311 | Helped |
| add LPIPS reconstruction loss (ablation) | CAP4D 4D representation · Nersemble self-reenactment, 10 reference images | PSNR: 21.75 -> 21.69 | Hurt |
| use CAP4D instead of GAGAvatar | self-reenactment from one reference image · Nersemble self-reenactment, single reference image | JOD: 5.034 -> 5.672 | Helped |
| use CAP4D instead of GaussianAvatars | self-reenactment from 10 reference images · Nersemble self-reenactment, 10 reference images | LPIPS: 0.448 -> 0.265 | Helped |
| use CAP4D instead of FlashAvatar | self-reenactment from 100 reference images · Nersemble self-reenactment, 100 reference images | CSIM: 0.731 -> 0.792 | Helped |
Surprising findings
- Predicting target frames directly with the MMDM gives higher PSNR than the full CAP4D pipeline in self-reenactment, at the cost of temporal consistency (JOD).
- Real3D-Portrait achieves slightly higher CSIM than CAP4D in cross-reenactment, yet human raters strongly preferred CAP4D.
Limitations
- Generation is time-consuming, up to 8 hours per avatar.
- The 3DMM does not model tongue or hair motion, so artifacts appear in those regions.
- Back-of-head quality is limited because no large 360-degree multi-identity dataset was available.
Experimental setup
Datasets: VFHQ MEAD Ava-256 Nersemble FFHQ
Benchmarks: Nersemble VFHQ FFHQ
Metrics: PSNR LPIPS CSIM JOD human preference
Baselines: GAGAvatar Portrait4D-v2 Real3D-Portrait Voodoo3D DiffusionRig FlashAvatar GaussianAvatars
Builds on: Stable Diffusion 2.1 CAT3D GaussianAvatars 3D Gaussian splatting DDIM
Closest papers by meaning
- AvatarArtist: Open-Domain 4D Avatarization (CVPR 2025 · similarity 0.91)
- Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation (CVPR 2024 · similarity 0.91)
- Zero-1-to-A: Zero-Shot One Image to Animatable Head Avatars Using Video Diffusion (CVPR 2025 · similarity 0.90)
- Portrait4D: Learning One-Shot 4D Head Avatar Synthesis using Synthetic Data (CVPR 2024 · similarity 0.89)
- Human-3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Models (NeurIPS 2024 · similarity 0.89)
- GAF: Gaussian Avatar Reconstruction from Monocular Videos via Multi-view Diffusion (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.