Research map / Video / Video generation
HunyuanPortrait: Implicit Condition Control for Enhanced Portrait Animation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HunyuanPortrait, a stable-video-diffusion-based portrait animation framework that drives a reference portrait with implicit motion embeddings from a pre-trained motion encoder, using an intensity-aware motion encoder, motion memory bank, and IMAdapter for identity.
Problem
Portrait animation from a single reference image and driving video suffers from identity distortion, weak control of subtle facial dynamics, and temporal inconsistency when relying on explicit landmarks or image-only diffusion models, especially in cross-reenactment.
Approach
Encodes driving motion implicitly via a pre-trained motion encoder (cropped face region) refined by an intensity-aware motion encoder and learnable motion memory bank, and injects identity features from ArcFace plus DINOv2 through a multi-scale ID-aware adapter (IMAdapter) into stable video diffusion via attention, avoiding explicit keypoints.
Key results
- Self-reenactment LMD of 2.02e-3 vs. 5.63e-3 for the best baseline FollowYourEmoji; FVD 333.48 vs. 416.41 for X-Portrait.
- Cross-reenactment identity similarity 0.887 vs. 0.820 for FollowYourEmoji, with user-study facial movement score 4.55 vs. 4.11 for the best baseline.
- Ablation: removing IAME raises FVD from 333.48 to 385.43 and LMD from 2.02 to 4.01.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add motion memory bank (ablation) | HunyuanPortrait · Cross Reenactment | FVD: 361.94 -> 333.48 | Helped |
| add intensity-aware motion encoder (ablation) | HunyuanPortrait · Cross Reenactment | FVD: 385.43 -> 333.48 | Helped |
| add ID features to IMAdapter (ablation) | HunyuanPortrait · Cross Reenactment | Identity Similarity: 8.21 -> 8.87 | Helped |
| add IMAdapter (ablation) | HunyuanPortrait · Cross Reenactment | Identity Similarity: 8.09 -> 8.87 | Helped |
Limitations
- Relies on a pre-trained motion encoder that does not fully decouple identity from motion, requiring extra training strategies.
- Cross-reenactment evaluation relies partly on user studies because no ground truth exists and metrics poorly match human perception.
- Baseline comparisons for FollowYourEmoji and X-Portrait are limited by lack of open-source training code.
Experimental setup
Datasets: HDTF VFHQ VoxCeleb2 CelebV-HQ self-collected 20h video dataset AnimeGANv3-stylized portrait data
Benchmarks: VFHQ VoxCeleb2
Metrics: LMD FID-VID FVD PSNR SSIM LPIPS AED APD ArcFace identity similarity user study facial movements
Baselines: LivePortrait AniPortrait FollowYourEmoji X-Portrait
Builds on: Stable Video Diffusion LivePortrait motion encoder ArcFace DINOv2 AnimateDiff
Closest papers by meaning
- RealPortrait: Realistic Portrait Animation with Diffusion Transformers (AAAI 2025 · similarity 0.91)
- High-Fidelity Relightable Monocular Portrait Animation with Lighting-Controllable Video Diffusion Model (CVPR 2025 · similarity 0.88)
- Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer (CVPR 2025 · similarity 0.87)
- AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers (CVPR 2025 · similarity 0.86)
- DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis (CVPR 2024 · similarity 0.86)
- MVPortrait: Text-Guided Motion and Emotion Control for Multi-view Vivid Portrait Animation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.