Research map / Video / Video generation
RealPortrait: Realistic Portrait Animation with Diffusion Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RealPortrait, a DiT-based portrait animation framework that uses dense motion position maps via a parallel ControlNet, shared-weight reference attention, and image-video mixed training.
Problem
Portrait animation that transfers facial expressions and head pose from a driving video to a static portrait with high realism and temporal coherence, avoiding the stiffness of sparse landmarks and the low quality of GAN decoders and SD-based pipelines.
Approach
Replaces the separate SD reference network with a reference attention that shares the DiT backbone weights, and drives the DiT with a dense 2D motion position map (warped features from Face vid2vid, masked to the face and placed back in full-image coordinates) through a parallel ControlNet trained jointly with the backbone, plus random-crop image data to avoid shortcut learning.
Key results
- Self-reenactment: PSNR 32.213, SSIM 0.831, LPIPS 0.064, FID 55.346, best among compared methods (LivePortrait FID 57.896, X-Portrait 56.547).
- Cross-reenactment: ID similarity 0.673, expression accuracy 0.664 and pose accuracy 0.031 (lower is better), slightly ahead of LivePortrait and X-Portrait.
- Ablation: 2D motion position maps improve pose accuracy (0.031) over 1D implicit keypoints (0.053) with similar ID similarity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 2D motion position maps instead of 1D implicit keypoints (ablation) | RealPortrait · Cross-reenactment | Pose Acc: 0.053 -> 0.031 | Helped |
| Use 2D motion position maps instead of 1D implicit keypoints (ablation) | RealPortrait · Cross-reenactment | ID Sim: 0.664 -> 0.673 | Helped |
| Use 2D motion position maps instead of 1D implicit keypoints (ablation) | RealPortrait · Cross-reenactment | Exp Acc: 0.693 -> 0.664 | Helped |
| Add image training data to video training data (ablation) | RealPortrait · Diverse appearances, including non-realistic styles | appearance preservation | Helped |
Limitations
- Inference is slow due to the large model; not real-time.
- Relies on Face vid2vid motion encoder, which does not decouple pose and expression, causing morphology inconsistencies under large head motion.
- Training data is private-leaning mixture of public datasets; evaluation relies on a small test split and reference-based metrics.
Experimental setup
Datasets: VGGFace2 CelebA FFHQ CelebV-HQ TalkingHead-1KH HDTF MEAD VFHQ
Benchmarks: self-reenactment test set from VFHQ and CelebV-HQ (100 clips) cross-reenactment with 200 reference portraits
Metrics: PSNR SSIM LPIPS FID ID Sim Expression accuracy L1 Pose accuracy L1
Baselines: Face vid2vid MCNet TPSM LivePortrait FADM MagicDance AniPortrait X-Portrait
Builds on: Open-Sora PixArt-δ Face vid2vid ControlNet DiT
Closest papers by meaning
- Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer (CVPR 2025 · similarity 0.91)
- DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis (CVPR 2024 · similarity 0.91)
- HunyuanPortrait: Implicit Condition Control for Enhanced Portrait Animation (CVPR 2025 · similarity 0.91)
- MVPortrait: Text-Guided Motion and Emotion Control for Multi-view Vivid Portrait Animation (CVPR 2025 · similarity 0.90)
- X-NeMo: Expressive Neural Motion Reenactment via Disentangled Latent Attention (ICLR 2025 · similarity 0.89)
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.