Research map / Video / Video generation
Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Animate Anyone animates a character image into a pose-driven video by adding a ReferenceNet (spatial-attention feature injection), a lightweight pose guider, and temporal layers on top of Stable Diffusion.
Problem
Pose-guided character image-to-video synthesis that keeps fine appearance details consistent over time and generalizes across arbitrary characters, which prior GAN and diffusion methods fail to achieve.
Approach
A symmetric ReferenceNet (a copy of the SD UNet) extracts detailed reference features that are injected into the denoising UNet via spatial-attention concatenation, combined with a lightweight zero-initialized pose guider and AnimateDiff-style temporal layers trained in two stages.
Key results
- On UBC fashion, FVD 81.6 and LPIPS 0.044 versus 148.3 and 0.048 for the next best (BDMM).
- On TikTok dance generation, FVD 171.9 versus 225.5 for SD-I2V and 292.8 for DisCo.
- On Ted-talk, FVD 80.5, slightly ahead of TPSMM (80.7) and MRAA (82.8) which use ground-truth driving images.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ReferenceNet instead of CLIP-only conditioning (ablation) | Animate Anyone · UBC fashion video dataset | SSIM: 0.897 -> 0.931 | Helped |
| use ReferenceNet instead of ControlNet for reference-image conditioning (ablation) | Animate Anyone · UBC fashion video dataset | SSIM: 0.892 -> 0.931 | Helped |
| combine CLIP image features and ControlNet instead of ReferenceNet (ablation) | Animate Anyone · UBC fashion video dataset | SSIM: 0.931 -> 0.898 | Hurt |
| use Stable Diffusion weights rather than ImageNet weights in ReferenceNet (ablation) | Animate Anyone · UBC fashion video dataset | SSIM: 0.901 -> 0.931 | Helped |
| use spatial attention rather than feature concatenation (ablation) | Animate Anyone · UBC fashion video dataset | SSIM: 0.909 -> 0.931 | Helped |
| add a temporal layer (ablation) | Animate Anyone · UBC fashion video dataset | FVD: 176.7 -> 81.6 | Helped |
| use two-stage training rather than training the entire network directly (ablation) | Animate Anyone · UBC fashion video dataset | SSIM: 0.917 -> 0.931 | Helped |
| use two-stage training rather than training the entire network directly (ablation) | Animate Anyone · UBC fashion video dataset | FVD: 89.3 -> 81.6 | Helped |
Surprising findings
- ControlNet-style control features fail to improve results because reference and target images are spatially unaligned, so spatial control is inapplicable to reference-image conditioning.
- Training the temporal layer jointly with the rest of the network (no two-stage training) lowers per-frame quality metrics despite comparable temporal behavior.
Limitations
- Struggles to generate stable hand movements, sometimes causing distortions and motion blur.
- Unseen parts of the character from a single view make generation ill-posed and potentially unstable.
- Diffusion sampling (DDPM/DDIM) makes it less efficient than non-diffusion methods.
Experimental setup
Datasets: 5K internal character video clips collected from internet UBC fashion video dataset TikTok dataset Ted-talk dataset
Benchmarks: UBC fashion video dataset TikTok dataset Ted-talk dataset
Metrics: SSIM PSNR LPIPS FVD
Baselines: DreamPose DisCo BDMM MRAA TPSMM FOMM SD-I2V AnimateDiff Gen-2
Builds on: Stable Diffusion ControlNet AnimateDiff IP-Adapter DWPose
Closest papers by meaning
- Animate-X: Universal Character Image Animation with Enhanced Motion Representation (ICLR 2025 · similarity 0.91)
- AnyMoLe: Any Character Motion In-betweening Leveraging Video Diffusion Models (CVPR 2025 · similarity 0.90)
- MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model (CVPR 2024 · similarity 0.89)
- Free-viewpoint Human Animation with Pose-correlated Reference Selection (CVPR 2025 · similarity 0.88)
- ReMask-Animate: Refined Character Image Animation Using Mask-Guided Adapters (AAAI 2025 · similarity 0.88)
- MIMO: Controllable Character Video Synthesis with Spatial Decomposed Modeling (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.