Research map / Video / Video generation
Decouple Content and Motion for Conditional Image-to-Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Decouples RGB video into first-frame content and H.264-style motion vectors and residuals, and trains a 3D-UNet diffusion model (D-VDM, and compressed ED-VDM) to generate conditional image-to-video on BAIR, MHAD and NATOPS.
Problem
Conditional image-to-video generation in RGB pixel space has poor motion consistency, redundancy, and high computational cost.
Approach
Generate temporal motion (motion vectors and residuals from video codec P-frames) with a diffusion model conditioned on the first frame and warp them onto the starting image, instead of generating RGB pixels; ED-VDM further compresses residuals with a VAE to a 16x spatial downsample.
Key results
- D-VDM achieves FVD 65.5 on BAIR (64x64), improving over prior best VDM at 66.9, and FVD 145.41 on MHAD and 152.19 on NATOPS at 64x64.
- ED-VDM reaches FVD 204.17 on MHAD and 179.65 on NATOPS at 128x128 with claimed 110x training/inference speedup over VDM.
- ED-VDM on BAIR gets FVD 92.4, slightly worse than the best baseline, which the authors attribute to the small 4x4 compressed latent at low resolution.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace RGB-pixel prediction with frame-residual prediction | D-VDM on BAIR · BAIR, 64×64 | FVD: 66.9 -> 65.5 | Helped |
| Use motion vectors and compressed residuals instead of frame residuals | ED-VDM on BAIR · BAIR, 64×64 | FVD: 65.5 -> 92.4 | Hurt |
| Use motion vectors and compressed residuals | ED-VDM on MHAD and NATOPS · MHAD and NATOPS, 128×128 | FVD: 214.39 -> 204.17; 195.17 -> 179.65 | Helped |
| Use motion vectors and compressed residuals | ED-VDM · Training on 16×128×128×3 videos | FLOPs (×10^9): 8814 -> 78 | Helped |
| Replace DCT residual compression with autoencoder compression (ablation) | ED-VDM residual autoencoder · 128-MHAD-101 | PSNR: 17.08 -> 31.80 | Helped |
| Replace DCT residual compression with autoencoder compression (ablation) | ED-VDM residual autoencoder · 128-MHAD-101 | SSIM: 0.85 -> 0.96 | Helped |
Surprising findings
- ED-VDM underperforms the prior best on BAIR at 64x64 resolution in FVD (92.4 vs 66.9 for VDM), with the authors attributing this to the tiny latent size.
Limitations
- Image quality on BAIR at 64x64 is slightly lower for ED-VDM than SOTA.
- Relies on H.264-style motion vectors that are block-based (16x16 macroblocks), which may limit fine-grained motion modeling.
Experimental setup
Datasets: BAIR MHAD NATOPS
Benchmarks: BAIR MHAD NATOPS
Metrics: FVD cFVD sFVD PSNR SSIM LPIPS R-FVD FLOPs memory
Baselines: VDM LDM LFDM MCVD CCVS ImaGINator
Builds on: DDPM Latent Diffusion LFDM H.264
Closest papers by meaning
- Extrapolating and Decoupling Image-to-Video Generation Models: Motion Modeling is Easier Than You Think (CVPR 2025 · similarity 0.90)
- Enhancing Motion in Text-to-Video Generation with Decomposed Encoding and Conditioning (NeurIPS 2024 · similarity 0.89)
- Hierarchical Spatio-temporal Decoupling for Text-to-Video Generation (CVPR 2024 · similarity 0.89)
- Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model (NeurIPS 2024 · similarity 0.89)
- ConditionVideo: Training-Free Condition-Guided Video Generation (AAAI 2024 · similarity 0.88)
- StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.