Research map / Video / Video generation
Motion Consistency Model: Accelerating Video Diffusion with Disentangled Motion-Appearance Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes motion consistency model (MCM), a single-stage video diffusion distillation method that distills motion via a learnable motion-representation consistency loss while an image discriminator improves frame appearance, plus mixed real/generated trajectory distillation.
Problem
Distilling video diffusion models into few-step samplers yields poor frame quality because public video datasets have low frame quality (low resolution, motion blur, watermarks), and naively combining consistency distillation with an image adversarial loss creates conflicting objectives and training
Approach
Disentangle motion from appearance: apply the consistency distillation loss only to a learnable motion representation (a two-layer MLP head on the LCM output, EMA-updated and discarded at inference) while an image discriminator (on sparsely sampled frames) handles appearance; and mix training on real-video ODE trajectories with trajectories simulated from the student's own one-step high-quality ge
Key results
- On WebVid mini with ModelScopeT2V teacher at 1 step, MCM reaches FVD 526 and CLIPSIM 29.81 versus LCM's 854 and 27.50 (ours-impl. LCM).
- On MSRVTT zero-shot with AnimateDiff teacher at 4 steps, MCM achieves FVD 801 versus 956 for LCM and 892 for AnimateDiff-Lightning.
- For frame quality with LAION-aes images, MCM achieves FID 69 at 2 steps versus 119 for the two-stage baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add image discriminator (adversarial loss on sparsely sampled frames) to video LCM (ablation) | ModelScopeT2V LCM baseline · WebVid mini | FVD@1 step: 854 -> 703 | Helped |
| add motion consistency distillation (learnable motion representation) on top of adversarial loss (ablation) | ModelScopeT2V LCM + adversarial · WebVid mini | FVD@2 steps: 650 -> 526 | Helped |
| add mixed trajectory distillation (real + generated trajectories, lambda_real=0.5) (ablation) | MCM w/ adv + MCD · WebVid mini | FVD@2 steps and CLIPSIM@2 steps: 526 -> 450 FVD; 30.40 -> 30.63 CLIPSIM | Helped |
| use only generated (simulated high-quality) trajectories (lambda_real=0) instead of mixing (ablation) | MCM mixed trajectory training · WebVid mini | FVD and CLIPSIM across steps: FVD 483-556 vs 450-526 with lambda_real=0.5; CLIPSIM 28.77 at 1 step vs 29.81; leads to near all-black videos in ~5% sam | Hurt |
| use only real trajectories (lambda_real=1) instead of mixing (ablation) | MCM mixed trajectory training · WebVid mini | FVD@2 steps: 526 vs 450 at 2 steps | Hurt |
| learnable MLP motion representation vs raw latent for motion consistency (ablation) | MCM motion distillation · WebVid mini | FVD@2 steps: 650 -> 526 | Helped |
| latent difference as motion representation (ablation) | MCM motion distillation · WebVid mini | FVD@1 and FVD@4 steps: FVD@1 703 -> 628; FVD@8 760 -> 779 (worse) | Mixed |
| latent low-frequency component as motion representation (ablation) | MCM motion distillation · WebVid mini | FVD@8 steps: 919 vs 760 raw latent | Hurt |
| MCM (adv + MCD + mixed) vs AnimateDiff-Lightning, AnimateLCM, LCM on AnimateDiff teacher | AnimateDiff teacher · WebVid mini and MSRVTT | FVD and CLIPSIM at 1,2,4 steps: WebVid 1-step FVD 1025 vs 1242-1575 baselines; MCM outperforms competing distillation at steps 1,2,4 | Helped |
| MCM with image dataset (LAION-aes or anime) vs two-stage image-then-video training | ModelScopeT2V frame quality · LAION-aes and Anime FID | FID at 1-8 steps: LAION-aes FID 82 vs 110 at 1 step; Anime FID 67 vs 149 at 4 steps | Helped |
| MCM at 8 steps on ModelScopeT2V vs strongest baseline DPM++ | ModelScopeT2V teacher · WebVid mini | CLIPSIM@8 steps: MCM 29.58 vs DPM++ 31.00 | Hurt |
Surprising findings
- Training only on generated (self-simulated) trajectories causes near all-black videos in roughly 5% of generations, mode collapse that motivates mixing real samples.
- Handcrafted motion representations (latent low-frequency components) did not help and degraded FVD at 8 steps, while learnable MLP motion representation performed best.
Limitations
- Sensitive to distribution and diversity of training data, potentially limiting generalization.
- Potential misuse risks such as deepfake creation.
- Hyperparameters (lambda_real, lambda_adv) selected by grid search; mixed trajectory relies on the student's one-step outputs.
Experimental setup
Datasets: WebVid 2M WebVid mini MSRVTT LAION-Aesthetic 6+ anime/realistic/3D cartoon style image datasets
Benchmarks: WebVid mini MSRVTT LAION-aes FID Anime FID
Metrics: FVD CLIPSIM FID
Baselines: DDIM DPM++ LCM AnimateLCM AnimateDiff-Lightning two-stage training
Builds on: latent consistency model consistency models AnimateDiff ModelScopeT2V ADD
Closest papers by meaning
- OSV: One Step is Enough for High-Quality Image to Video Generation (CVPR 2025 · similarity 0.91)
- TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models (CVPR 2024 · similarity 0.89)
- Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition (ICLR 2024 · similarity 0.89)
- Emergent Temporal Correspondences from Video Diffusion Transformers (NeurIPS 2025 · similarity 0.89)
- VideoGuide: Improving Video Diffusion Models without Training Through a Teacher's Guide (CVPR 2025 · similarity 0.89)
- Dancing with Still Images: Video Distillation via Static-Dynamic Disentanglement (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.