Research map / Video / Video generation
Make Pixels Dance: High-Dynamic Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PixelDance, a latent video diffusion model conditioned on text plus first- and last-frame image instructions, trained mainly on WebVid, to generate high-dynamic videos.
Problem
Text-to-video diffusion models tend to produce clips with minimal motion and weak fidelity on out-of-domain content, and text alone is insufficient for controlling high-dynamic motion.
Approach
Condition the video diffusion model on image instructions for both the first frame (strictly enforced) and an optional, perturbed last frame (randomly chosen from last three frames, dropped with probability 25%), concatenated channel-wise with noisy latents, plus text via cross-attention; at inference, apply the last-frame guidance only for the first tau denoising steps.
Key results
- Zero-shot on MSR-VTT, PixelDance reaches FVD 381 and CLIPSIM 0.3125, versus ModelScope's FVD 550.
- On UCF-101 zero-shot, PixelDance achieves IS 42.10, FID 49.36, FVD 242.82, outperforming Dysen-VDM (FVD 325.42).
- Ablation on UCF-101 shows removing the last-frame instruction raises FVD from 242.82 to 339.08 and removing text instruction raises it to 375.79 (w/o c_text).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PixelDance instead of the strongest listed baseline, Make-A-Video | Zero-shot video generation · MSR-VTT | CLIPSIM: 0.3049 -> 0.3125 | Helped |
| Use PixelDance instead of ModelScope | Zero-shot video generation · MSR-VTT | FVD: 550 -> 381 | Helped |
| Use PixelDance instead of VidRD | Zero-shot video generation · UCF-101 | IS: 39.37 -> 42.10 | Helped |
| Use PixelDance instead of InternVid | Zero-shot video generation · UCF-101 | FID: 60.25 -> 49.36 | Helped |
| Use PixelDance instead of Dysen-VDM | Zero-shot video generation · UCF-101 | FVD: 325.42 -> 242.82 | Helped |
| Use WebVid and self-collected video data rather than WebVid alone (ablation) | T2V baseline · UCF-101 | FID: 56.16 -> 59.35 | No clear effect |
| Use WebVid and self-collected video data rather than WebVid alone (ablation) | T2V baseline · UCF-101 | FVD: 454.29 -> 450.58 | No clear effect |
| Add the text instruction (ablation) | PixelDance · UCF-101 | FID: 51.26 -> 49.36 | Helped |
| Add the text instruction (ablation) | PixelDance · UCF-101 | FVD: 375.79 -> 242.82 | Helped |
| Add the last-frame instruction (ablation) | PixelDance · UCF-101 | FID: 49.45 -> 49.36 | Helped |
| Add the last-frame instruction (ablation) | PixelDance · UCF-101 | FVD: 339.08 -> 242.82 | Helped |
Surprising findings
- Training with the last-frame instruction improves results even when it is not supplied at inference (WebVid-only model with last frame outperforms the variant without it).
Limitations
- Relies on watermark-free first-frame images from text-to-image models or users for inference.
- Human evaluation compared only against Gen2 and PiKa on 50 prompts.
Experimental setup
Datasets: WebVid LAION-400M self-collected watermark-free video clips (500K) MSR-VTT UCF-101
Benchmarks: MSR-VTT UCF-101
Metrics: FVD CLIPSIM IS FID human preference (text faithfulness, video quality)
Baselines: CogVideo MagicVideo LVDM Video-LDM InternVid ModelScope Make-A-Video Latent-Shift VideoFactory VidRD Dysen-VDM Gen2
Builds on: latent diffusion models Stable Diffusion Video LDM / temporal UNet inflation classifier-free guidance WebVid
Closest papers by meaning
- Motion Prompting: Controlling Video Generation with Motion Trajectories (CVPR 2025 · similarity 0.89)
- Move-in-2D: 2D-Conditioned Human Motion Generation (CVPR 2025 · similarity 0.89)
- Optical-Flow Guided Prompt Optimization for Coherent Video Generation (CVPR 2025 · similarity 0.88)
- Enhancing Motion in Text-to-Video Generation with Decomposed Encoding and Conditioning (NeurIPS 2024 · similarity 0.88)
- Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis (CVPR 2024 · similarity 0.88)
- LAMP: Learn A Motion Pattern for Few-Shot Video Generation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.