Research map / Video / Video generation
MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
MagicAnimate is a diffusion-based human image animation framework that adds temporal attention, a dense appearance encoder, and overlapping-segment video fusion to produce temporally consistent long videos from a reference image and DensePose sequence.
Problem
Human image animation from a reference image and motion sequence suffers from temporal flickering in frame-by-frame diffusion methods and poor identity/background preservation when using CLIP-based or warping-based approaches.
Approach
Inflate a Stable Diffusion UNet with temporal attention, replace CLIP reference conditioning with a trainable appearance encoder whose dense self-attention features are concatenated into the UNet's spatial self-attention, use DensePose via a pose ControlNet, and fuse overlapping video segments by averaging predictions with shared initial noise.
Key results
- On TikTok, MagicAnimate reaches FVD 179.07 and FID-VID 21.75 versus DisCo's 292.80 and 59.90, a 38.8% FVD improvement over the strongest baseline.
- On TED-talks, MagicAnimate achieves FVD 131.51 and FID 22.78, the best among compared methods, with a 28.1% FVD improvement over MRAA.
- Ablation shows removing temporal attention raises FVD from 179.07 to 247.30 on TikTok.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MagicAnimate instead of DisCo | human image animation · TikTok | FVD: 292.80 -> 179.07 | Helped |
| use MagicAnimate instead of DisCo | human image animation · TED-talks | FVD: 195.00 -> 131.51 | Helped |
| remove temporal attention layers (ablation) | MagicAnimate · TikTok | FVD: 179.07 -> 247.30 | Hurt |
| add the dense appearance encoder instead of CLIP (ablation) | MagicAnimate · TikTok | FVD: 724.96 -> 179.07 | Helped |
| add the dense appearance encoder instead of IP-Adapter (ablation) | MagicAnimate · TikTok | FVD: 590.99 -> 179.07 | Helped |
| remove image-video joint training at both the appearance encoding and temporal modeling stages (ablation) | MagicAnimate · TikTok | FVD: 158.16 -> 179.07 | No clear effect |
| remove inference-stage temporal video fusion (ablation) | MagicAnimate · TikTok | L1: 3.13 -> 3.21 | Hurt |
| use different initial noises for different video segments (ablation) | MagicAnimate · TikTok | FID-FVD: 21.75 -> 22.50 | Hurt |
Surprising findings
- MagicAnimate has higher L1 error than baselines on TED-talks, attributed to DensePose lacking background information.
- Adding temporal attention alone degrades per-frame quality (FID rises from 27.54 to 32.09 on TikTok when temporal attention is added), which motivated joint image-video training.
Limitations
- Higher L1 error on TED-talks because DensePose control signals lack background information, so dynamic backgrounds are not learned consistently.
- Segment-by-segment processing can leave minor discontinuities between segments, mitigated but not eliminated by fusion.
- Single-frame quality degrades when temporal attention is introduced without joint image training.
Experimental setup
Datasets: TikTok TED-talks large-scale human image dataset
Benchmarks: TikTok TED-talks
Metrics: L1 PSNR SSIM LPIPS FID FID-VID FVD AKD MKR AED
Baselines: MRAA TPS DreamPose DisCo IP-Adapter+ControlNet IP-Adapter+ControlNet-V
Builds on: Stable Diffusion ControlNet DisCo DreamPose IP-Adapter
Closest papers by meaning
- EverAnimate: Minute-Scale Human Animation via Latent Flow Restoration (NeurIPS 2026 · similarity 0.89)
- Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (CVPR 2024 · similarity 0.89)
- Bidirectional Temporal Diffusion Model for Temporally Consistent Human Animation (ICLR 2024 · similarity 0.89)
- AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning (ICLR 2024 · similarity 0.88)
- StableAnimator: High-Quality Identity-Preserving Human Image Animation (CVPR 2025 · similarity 0.88)
- TPC: Test-time Procrustes Calibration for Diffusion-based Human Image Animation (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.