Research map / Video / Video generation
Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CMD, a latent video diffusion model that encodes a video as a learned weighted-sum content frame plus a low-dimensional triplane-style motion latent, so a pretrained image diffusion model (Stable Diffusion) can be fine-tuned for content generation and a lightweight DiT generates motion.
Problem
Video diffusion models extended from image models are memory- and compute-heavy because they process high-dimensional video cubes directly, while latent video diffusion models do not exploit strong pretrained image models.
Approach
Encode each video into an image-like content frame (softmax-weighted sum of frames) plus factorized x/y motion latents; fine-tune a pretrained image DM on content frames without new parameters and train a small DiT for motion conditioned on the content frame and text.
Key results
- On WebVid-10M, CMD achieves FVD 238.3, 18.5% better than the prior state-of-the-art 292.4.
- Sampling a 16-frame 512x1024 video takes 3.13 s with 50 DDIM steps and 5.56 GB memory, versus 35.1 s and 8.50 GB for ModelScope (about 7.7x faster per the abstract).
- On UCF-101, the content frame diffusion model is the bottleneck for generation with FVD 73.1 versus reconstruction FVD 7.72.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CMD instead of the previous state-of-the-art approach | Text-to-video generation · WebVid-10M | FVD: 292.4 -> 238.3 | Helped |
| Use content-frame and motion-latent decomposition for sampling | CMD versus ModelScope · 512×1024, 16-frame sampling, 50 steps | FLOPs: 939.0 -> 46.8 | Helped |
| Use content-frame and motion-latent decomposition for sampling | CMD versus ModelScope · 512×1024, 16-frame sampling, 50 steps | time: 35.1 -> 3.13 | Helped |
| Use content-frame and motion-latent decomposition for sampling | CMD versus ModelScope · 512×1024, 16-frame sampling, batch 1 | memory (GB): 8.50 -> 5.56 | Helped |
| Use a weighted-sum content frame rather than no weighted sum (ablation) | CMD autoencoder with D=8 and L=16 · WebVid-10M | R-FVD: 76.1 -> 69.5 | Helped |
| Increase motion diffusion content-frame patch size from 16 to 8 (ablation) | DiT-L/2 motion diffusion · UCF-101 | FVD: 40.4 -> 32.9 | Helped |
| Decrease motion diffusion content-frame patch size from 8 to 4 (ablation) | DiT-L/2 motion diffusion · UCF-101 | FVD: 32.9 -> 19.5 | Helped |
| Increase autoencoder motion-latent channel size from D=8 to D=16 (ablation) | CMD autoencoder with L=16 and weighted-sum content frame · WebVid-10M | R-FVD: 69.5 -> 56.8 | Helped |
| Increase video length from L=16 to L=24 (ablation) | CMD autoencoder with D=8 and weighted-sum content frame · WebVid-10M | R-FVD: 69.5 -> 81.3 | Hurt |
| Set classifier-free guidance scale to 7.0 instead of 9.0 (ablation) | CMD · WebVid-10M | FVD: 238.3 -> 235.5 | Helped |
| Increase classifier-free guidance scale from 7.0 to 11.0 (ablation) | CMD · WebVid-10M | FVD: 235.5 -> 246.9 | Hurt |
| Increase classifier-free guidance scale from 7.0 to 11.0 (ablation) | CMD · WebVid-10M | CLIPSIM: 0.3001 -> 0.3034 | Helped |
Surprising findings
- Content-frame design is limited for highly dynamic motion, giving frame-wise quality drop, as acknowledged by the authors.
Limitations
- Focused on fixed-length 16-frame videos; long-video generation not addressed.
- Frame-wise quality drop for extremely dynamic motion due to content-frame and latent design.
- Relatively small autoencoder and diffusion models (1.6B) due to limited resources.
Experimental setup
Datasets: UCF-101 WebVid-10M MSR-VTT
Benchmarks: UCF-101 WebVid-10M MSR-VTT
Metrics: FVD CLIPSIM R-FVD FLOPs sec/step memory (GB)
Baselines: DIGAN TATS CogVideo Make-A-Video MAGVIT LVDM ModelScope VideoLDM PYoCo VideoFactory
Builds on: latent diffusion models DiT PVDM VideoFusion Stable Diffusion
Closest papers by meaning
- MeDM: Mediating Image Diffusion Models for Video-to-Video Translation with Temporal Correspondence Guidance (AAAI 2024 · similarity 0.90)
- OSV: One Step is Enough for High-Quality Image to Video Generation (CVPR 2025 · similarity 0.90)
- From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CVPR 2025 · similarity 0.90)
- TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models (CVPR 2024 · similarity 0.90)
- LLM-grounded Video Diffusion Models (ICLR 2024 · similarity 0.90)
- Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.