Research map / Video / Video generation
AnimateAnything: Consistent and Controllable Animation for Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AnimateAnything, a two-stage I2V method that unifies camera trajectories, arrow drags and reference videos into frame-by-frame optical flow, then uses the flow to guide a CogVideoX-based video generator with a frequency-domain stabilization module.
Problem
Controllable image-to-video generation that jointly handles heterogeneous control signals (camera trajectories, drag/arrow motion annotations, reference videos) without conflicts or flicker from large-scale motion.
Approach
Convert all control signals into a unified dense optical flow via a flow generation model guided by a camera reference model (Plücker embeddings), then condition video generation on the flow latents with a single ViT block and apply FFT-based spectral weight modulation to suppress flicker.
Key results
- On OpenVid, the method reaches FVD 322 vs 453 for Pyramid-Flow and 512 for Open-Sora, and SSIM 0.836 vs 0.789 for Pyramid-Flow.
- On camera trajectory estimation (basic trajectory, DUSt3R), translation error 0.041 and rotation error 0.159, better than MotionCtrl (0.057/0.233) and CameraCtrl (0.090/0.300).
- Ablation: removing frequency stabilization raises FVD from 168 to 311 and LPIPS from 0.142 to 0.241.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use unified optical flow guidance instead of camera embedding as the DiT-block input (ablation) | AnimateAnything · Real10K | FVD: 346 -> 168 | Helped |
| Use unified optical flow guidance instead of a ControlNet-like reference network (ablation) | AnimateAnything · Real10K | FVD: 356 -> 168 | Helped |
| Add Frequency Stabilization (ablation) | AnimateAnything · Real10K | FVD: 311 -> 168 | Helped |
| Add noise to optical flow before feeding it into the Flow Encoder (ablation) | AnimateAnything · Real10K | FVD: 474 -> 168 | Helped |
Limitations
- Camera-trajectory training data is scarce; available pose datasets (RealEstate10K, DL3DV) are mostly indoor or static, requiring a workaround with fixed-camera dynamic videos.
- Stage-2 relies on the alignment between reference image and flow; large misalignment may degrade results, though the paper reports tolerance.
Experimental setup
Datasets: RealEstate10K DL3DV-10K WebVid10M OpenVid-1M Unimatch-extracted optical flow
Benchmarks: WebVid10M OpenVid Real10K
Metrics: LPIPS PSNR SSIM FID FVD Subject Consistency Motion Smoothness Aesthetic Quality translation error rotation error
Baselines: Motion-I2V MOFA-Video DynamiCrafter CogVideoX Pyramid-Flow Open-Sora MotionCtrl CameraCtrl DragAnything MotionClone
Builds on: CogVideoX AnimateAnyone Motion-I2V MOFA-Video CameraCtrl
Closest papers by meaning
- IM-Zero: Instance-level Motion Controllable Video Generation in a Zero-shot Manner (CVPR 2025 · similarity 0.88)
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.88)
- Follow-Your-Click: Open-domain Regional Image Animation via Motion Prompts (AAAI 2025 · similarity 0.87)
- ByTheWay: Boost Your Text-to-Video Generation Model to Higher Quality in a Training-free Way (CVPR 2025 · similarity 0.86)
- Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (CVPR 2024 · similarity 0.86)
- Frame In-N-Out: Unbounded Controllable Image-to-Video Generation (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.