Research map / Image generation / GANs
AvatarArtist: Open-Domain 4D Avatarization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds AvatarArtist, a diffusion-transformer plus GAN pipeline that creates a 4D (parametric triplane) avatar from a single portrait in arbitrary styles, using a 2D-diffusion-assisted multi-domain 4D GAN dataset and a motion-aware cross-domain renderer.
Problem
Single-portrait 4D avatarization works only for realistic human domains; existing 4D methods lack multi-domain 4D training data and generalize poorly to stylized or non-human portraits.
Approach
Use SDEdit and landmark-guided ControlNet from a 2D diffusion model to transfer FFHQ portraits to 28 domains so existing 3DMM meshes can be reused; fine-tune a Next3D 4D GAN per domain to synthesize a large image-parametric-triplane dataset; train a DiT on triplane latents and a ViT-based motion-aware renderer that conditions on the source image and implicit motion embedding.
Key results
- In self-reenactment on VFHQ, the method reaches LPIPS 0.26 and FID 52.62, the best among compared methods on those metrics, but is slightly behind 2D methods on ID and AED.
- In cross-reenactment on VFHQ, the method achieves AKD 2.58 (best) and APD 52.3 (best) versus 4.92 for LivePortrait and 7.13 for Portrait4Dv2, while FID 89.3 trails XPortrait's 78.6.
- Ablation on FFHQ: replacing the renderer with the Next3D CNN raises FID from 68.69 to 130.72 and AKD from 2.56 to 5.89.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the Next3D CNN renderer with the motion-aware cross-domain renderer (ablation) | AvatarArtist · FFHQ cross-reenactment | FID: 130.72 -> 68.69 | Helped |
| replace the Next3D CNN renderer with the motion-aware cross-domain renderer (ablation) | AvatarArtist · FFHQ cross-reenactment | CLIP: 0.73 -> 0.86 | Helped |
| replace the Next3D CNN renderer with the motion-aware cross-domain renderer (ablation) | AvatarArtist · FFHQ cross-reenactment | AKD: 5.89 -> 2.56 | Helped |
| replace the Next3D CNN renderer with the motion-aware cross-domain renderer (ablation) | AvatarArtist · FFHQ cross-reenactment | APD: 42.93 -> 40.89 | Helped |
| add ControlNet landmark guidance to SDEdit for domain transfer (ablation) | AvatarArtist data generation · multi-domain portrait data | pose and expression consistency | Helped |
| add SDEdit to ControlNet-guided domain transfer (ablation) | AvatarArtist data generation · multi-domain portrait data | pose and expression consistency | Helped |
| use the motion-aware cross-domain renderer instead of a Next3D-style CNN renderer (ablation) | AvatarArtist · cross-domain portrait rendering | identity preservation | Helped |
| add implicit motion embedding to correct mesh motion inaccuracies (ablation) | AvatarArtist motion-aware renderer · cross-domain portrait rendering | expression accuracy | Helped |
Surprising findings
- The 2D diffusion-based methods (LivePortrait, XPortrait) remain stronger than the proposed 4D method on self-reenactment identity and expression metrics, and XPortrait beats it on cross-reenactment FID.
Limitations
- Self-reenactment performance is slightly worse than 2D methods on identity and expression metrics.
- Relies on 3DMM mesh extraction (FaceVerse), which is inaccurate for non-realistic portraits, requiring an implicit motion embedding to compensate.
- Data is limited to 28 domains generated from FFHQ with 6,000 images each, and GAN mode collapse remains a concern.
Experimental setup
Datasets: FFHQ VFHQ FaceVerse meshes Stable Diffusion-generated multi-domain images
Benchmarks: VFHQ FFHQ
Metrics: LPIPS FID ID AED APD CLIP score AKD
Baselines: LivePortrait XPortrait InvertAvatar Portrait4Dv2
Builds on: Next3D Direct3D PixArt-alpha Portrait4D SDEdit ControlNet DATID-3D
Closest papers by meaning
- CAP4D: Creating Animatable 4D Portrait Avatars with Morphable Multi-View Diffusion Models (CVPR 2025 · similarity 0.91)
- DiffusionGAN3D: Boosting Text-guided 3D Generation and Domain Adaptation by Combining 3D GANs and Diffusion Priors (CVPR 2024 · similarity 0.88)
- Portrait4D: Learning One-Shot 4D Head Avatar Synthesis using Synthetic Data (CVPR 2024 · similarity 0.88)
- Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation (CVPR 2024 · similarity 0.88)
- AvatarGO: Zero-shot 4D Human-Object Interaction Generation and Animation (ICLR 2025 · similarity 0.86)
- Zero-1-to-A: Zero-Shot One Image to Animatable Head Avatars Using Video Diffusion (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.