Research map / Video / Video generation
Consistent and Controllable Image Animation with Motion Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Cinemo animates a static image by learning the distribution of motion residuals in a video latent diffusion model, adds DCT-based initial noise refinement (DCTInit) to suppress abrupt motion, and uses an MS-SSIM-based dynamics degree bucket for motion magnitude control.
Problem
Image animation with diffusion models struggles to keep appearance consistent with the static input image and to avoid abrupt motion changes, while suppressing motion to achieve these tends to reduce dynamics.
Approach
Predict motion residuals (frame differences from the first frame) instead of raw frames, refine inference noise with low-frequency 3D DCT coefficients of the input image's latent (DCTInit), and condition on a dynamics degree bucket computed from MS-SSIM between frames to control motion magnitude.
Key results
- On UCF-101 Cinemo reports FVD 168.16 and IS 58.71 and FID 13.17, the best among compared methods including ConsistI2V (177.66 FVD).
- On MSR-VTT Cinemo reports FVD 93.51 and CLIPSIM 0.2858, outperforming ConsistI2V (104.58 FVD, 0.2674 CLIPSIM).
- On VBench I2V Cinemo achieves 71.30 imaging quality, 61.85 aesthetic quality and 98.21 motion smoothness, though its dynamic degree (30.68) is below DynamiCrafter and SVD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| learn motion residuals instead of predicting frames directly (ablation) | Cinemo · VBench I2V | Subject Consistency: 95.49 -> 96.86 | Helped |
| learn motion residuals instead of predicting frames directly (ablation) | Cinemo · VBench I2V | Background Consistency: 97.97 -> 98.13 | Helped |
| learn motion residuals instead of predicting frames directly (ablation) | Cinemo · VBench I2V | Motion Smoothness: 97.38 -> 97.98 | Helped |
| add DCTInit noise refinement instead of using FFT noise refinement (ablation) | Cinemo with motion residual learning · VBench I2V | Subject Consistency: 96.93 -> 97.23 | Helped |
| add DCTInit noise refinement instead of using FFT noise refinement (ablation) | Cinemo with motion residual learning · VBench I2V | Background Consistency: 98.33 -> 98.70 | Helped |
| add DCTInit noise refinement instead of using FFT noise refinement (ablation) | Cinemo with motion residual learning · VBench I2V | Motion Smoothness: 98.15 -> 98.21 | Helped |
Surprising findings
- The FFT-based low-frequency noise initialization introduced abrupt color changes in some examples, which DCT initialization avoids.
- Cinemo's dynamic degree is lower than DynamiCrafter and SVD, indicating a trade-off between smoothness and motion magnitude.
Limitations
- Dynamic degree is lower than some baselines such as DynamiCrafter and SVD, so motion magnitude is reduced by default.
- Comparisons with commercial tools rely on snapshot visual examples, which the authors note are not conclusive.
- Training is costly (8 A100 GPUs for one week).
Experimental setup
Datasets: WebVid-10M Vimeo25M MSR-VTT UCF-101
Benchmarks: UCF-101 MSR-VTT VBench I2V
Metrics: FVD IS FID CLIPSIM imaging quality aesthetic quality motion smoothness dynamic degree user study ratings
Baselines: SVD I2VGen-XL DynamiCrafter SEINE ConsistI2V PIA Follow-Your-Click Gen-2 Genmo Pika Labs
Builds on: LaVie Stable Diffusion classifier-free guidance DDIM
Closest papers by meaning
- MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model (CVPR 2024 · similarity 0.87)
- FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generation (NeurIPS 2025 · similarity 0.87)
- Motion Consistency Model: Accelerating Video Diffusion with Disentangled Motion-Appearance Distillation (NeurIPS 2024 · similarity 0.87)
- Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noise (CVPR 2025 · similarity 0.87)
- MotionPro: A Precise Motion Controller for Image-to-Video Generation (CVPR 2025 · similarity 0.86)
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.