Research map / Video / Video generation
Make-Your-Anchor: A Diffusion-based 2D Avatar Generation Framework
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Make-Your-Anchor fine-tunes a structure-guided Stable Diffusion model on a one-minute video of a person, conditioned on SMPL-X 3D mesh renderings and CLIP appearance features, to generate full-body anchor videos with batch-overlapped temporal denoising and identity-specific face enhancement.
Problem
Generating anchor-style 2D human videos with precise torso and hand movements, beyond talking-face methods, while preserving identity and temporal consistency from a short single-person video.
Approach
A structure-guided diffusion model conditioned on 3D mesh renderings and image-CLIP appearance features, trained in two stages (multi-identity pretraining then single-identity finetuning), with a training-free 3D U-Net extension via all-frame cross-attention and overlapped-window noise averaging for arbitrary-length video, plus an inpainting-based face refinement model.
Key results
- Achieves FID 40.33 and FVD 139.82 versus DisCo's 60.95 and 390.77 and Pose2Img's 51.92 and 328.18.
- Face landmark error LMD (Face) of 1.44 versus 4.15 for DisCo.
- Hand LMD of 5.41 versus 11.21 for DisCo; removing temporal denoising raises FVD to 344.85.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add batch-overlapped temporal denoising (ablation) | Make-Your-Anchor · Test clips | FVD: 344.85 -> 139.82 | Helped |
| Add identity-specific face enhancement (ablation) | Make-Your-Anchor · Test clips | LMD (Face): 1.56 -> 1.44 | Helped |
| Add identity-specific face enhancement (ablation) | Make-Your-Anchor · Test clips | FVD: 124.10 -> 139.82 | Mixed |
| Use two-stage training instead of direct target-subject training (ablation) | Make-Your-Anchor · Test clips | FID: 55.87 -> 40.33 | Helped |
| Use two-stage training instead of training the target with all other subject data (ablation) | Make-Your-Anchor · Test clips | FID: 53.99 -> 40.33 | Helped |
| Perturb SMPL parameters during training (ablation) | Make-Your-Anchor · Test clips | FID: 40.33 -> 39.56 | Helped |
| Increase temporal denoising window size and overlap from ws=8, os=2 to ws=32, os=8 (ablation) | Make-Your-Anchor · Test clips | FVD: 140.29 -> 138.48 | Helped |
| Increase temporal denoising window size and overlap from ws=8, os=2 to ws=32, os=8 (ablation) | Make-Your-Anchor · Test clips | FID: 39.64 -> 41.01 | Hurt |
Surprising findings
- Without the batch-overlapped temporal denoising, FVD rises from 139.82 to 344.85, indicating it is the main driver of temporal consistency.
- Perturbing SMPL parameters during training slightly improved all metrics (FID 39.56, FVD 136.21).
Limitations
- Performance degrades when inference body orientation differs significantly from fine-tuning videos.
- Does not model occluded foreground elements, causing ghosting.
- Hand landmark-based LMD is unreliable for complex gestures.
Experimental setup
Datasets: Anchor video dataset from TalkSHOW (27 hours, SMPL-X annotated) Ten identity videos (1-5 min each), web celebrity videos, green-screen footage
Benchmarks: Ten-identity in-house test set (30 clips)
Metrics: FID FVD LMD (face) LMD (body) LMD (hand)
Baselines: Pose2Img TPS DreamPose DisCo
Builds on: Stable Diffusion ControlNet DreamPose TalkSHOW
Closest papers by meaning
- VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis (CVPR 2025 · similarity 0.89)
- DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation (ICLR 2025 · similarity 0.88)
- Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation (CVPR 2024 · similarity 0.88)
- ShowMaker: Creating High-Fidelity 2D Human Video via Fine-Grained Diffusion Modeling (NeurIPS 2024 · similarity 0.88)
- DiffusionAvatars: Deferred Diffusion for High-fidelity 3D Head Avatars (CVPR 2024 · similarity 0.87)
- Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.