Research map / Video / Video generation
MVPortrait: Text-Guided Motion and Emotion Control for Multi-view Vivid Portrait Animation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MVPortrait, a two-stage text-guided framework that generates FLAME motion/emotion sequences from text via two diffusion models and then renders them into multi-view portrait videos with a view attention module.
Problem
Portrait animation lacks explicit control over head motion and facial expression, cannot produce multi-view videos, and text-guided portrait animation is underexplored.
Approach
Uses FLAME as an intermediate representation: separate MotionDM and EmotionDM (transformer-based MDM-style diffusion) map text to FLAME pose and expression sequences, and a diffusion video model with a FLAME encoder and a view attention module (inserted after the motion module) renders consistent multi-view animations.
Key results
- On CelebV-Text text-guided animation, MVPortrait achieves CLIPSIM 0.183 and VideoCLIP 0.595 versus 0.171/0.564 for AnimateAnything, and higher user-study MC (2.57) and EC (2.29).
- Multi-view synthesis ID score of 0.8409 vs 0.8006 for Portrait4D-v2, and SSIM 0.6224 vs 0.5278; LPIPS 0.2101 vs 0.4468 for Portrait4D-v2.
- Audio-driven FID 56.20 and FVD 333.18 on HDTF/CelebV-Text, with FVD better than AniPortrait (387.84) and SadTalker (644.22) but FID worse than AniPortrait (51.49).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MVPortrait instead of AnimateAnything | Text-guided portrait animation · CelebV-Text test set | FID: 34.9 -> 28.6 | Helped |
| Use MVPortrait instead of MMVID-interp | Text-guided portrait animation · CelebV-Text test set | CLIPSIM: 0.175 -> 0.183 | Helped |
| Use MVPortrait instead of AnimateAnything | Text-guided portrait animation · CelebV-Text test set | Variability: 0.095 -> 0.110 | Helped |
| Use MVPortrait instead of LivePortrait | Video-driven portrait animation · CelebV-Text test set | FID: 49.6557 -> 48.4632 | Helped |
| Use MVPortrait instead of FollowYourEmoji | Video-driven portrait animation · CelebV-Text test set | FLAME-L1: 0.213 -> 0.196 | Helped |
| Use MVPortrait instead of AniPortrait | Audio-driven portrait animation · HDTF and CelebV-Text | FVD: 387.8381 -> 333.1806 | Helped |
| Use MVPortrait instead of AniPortrait | Audio-driven portrait animation · HDTF and CelebV-Text | FID: 51.4862 -> 56.1997 | Hurt |
| Use MVPortrait instead of Portrait4D-v2 | Multi-view portrait synthesis · Multi-view portrait synthesis | SSIM: 0.5278 -> 0.6224 | Helped |
| Use MVPortrait instead of Portrait4D-v2 | Multi-view portrait synthesis · Multi-view portrait synthesis | ID: 0.8006 -> 0.8409 | Helped |
| Add sliding average smoothing during training (ablation) | Text2FLAME · CelebV-Text test set | FID: 108.4 -> 28.6 | Helped |
| Use the smaller network instead of 4 layers with latent dimension 128 (ablation) | Text2FLAME · CelebV-Text test set | FID: 102.5 -> 28.6 | Helped |
| Train separate motion and emotion models instead of one joint model (ablation) | Text2FLAME · CelebV-Text test set | FID: 76.8 -> 28.6 | Helped |
Surprising findings
- AnimateAnything achieves a lower FVD (283.0) than MVPortrait (570.0) on text-guided animation, which the authors attribute to larger head motion amplitude diverging from the dataset distribution.
- LPIPS favors images with black backgrounds, so Triplanenet achieves a low LPIPS (0.0936) despite poor visual quality.
Limitations
- Limited by accuracy of text annotations in the video dataset and uneven distribution of facial movements and emotions.
- FLAME struggles to capture micro-expressions.
- Higher FVD on text-guided animation than AnimateAnything due to larger motion amplitude.
Experimental setup
Datasets: CelebV-Text CelebV-TF RenderMe-360 HDTF
Benchmarks: CelebV-Text test set HDTF RenderMe-360
Metrics: FID FVD CLIPSIM VideoCLIP LIQE Variability user study MC/EC/VS L1 SSIM ID
Baselines: AnimateAnything MMVID-interp Triplanenet Portrait4D-v2 FollowYourEmoji LivePortrait SadTalker EchoMimic AniPortrait
Builds on: MDM AnimateDiff AniPortrait DECA FLAME
Closest papers by meaning
- RealPortrait: Realistic Portrait Animation with Diffusion Transformers (AAAI 2025 · similarity 0.90)
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.88)
- KeyFace: Expressive Audio-Driven Facial Animation for Long Sequences via KeyFrame Interpolation (CVPR 2025 · similarity 0.88)
- ExpCLIP: Bridging Text and Facial Expressions via Semantic Alignment (AAAI 2024 · similarity 0.88)
- InstructAvatar: Text-Guided Emotion and Motion Control for Avatar Generation (AAAI 2025 · similarity 0.87)
- EMOPortraits: Emotion-enhanced Multimodal One-shot Head Avatars (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.