Research map / Video / Video generation
MotionCraft: Physics-Based Zero-Shot Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
MotionCraft zero-shot generates physics-driven videos by warping the noise latent of Stable Diffusion with optical flow from physics simulators, using multi-frame cross-attention and spatial-eta sampling.
Problem
Video generation with realistic, finely controlled, physically plausible motion without training large video diffusion models; existing zero-shot methods (Text2Video-Zero) produce inconsistent motion and content.
Approach
Warp the DDIM-inverted noise latent (not pixels) of a pretrained image diffusion model with an optical flow from a physics simulation, then regenerate each frame with multi-frame cross-frame attention (to first and previous frame) and a pixel-wise spatial-eta mix of DDIM/DDPM sampling, so new content can be synthesized while following the prescribed motion.
Key results
- Average Motion Consistency 0.8655 for MotionCraft vs 0.6740 for Text2Video-Zero across six scenes.
- Average Frame Consistency 0.9819 for MotionCraft vs 0.9658 for Text2Video-Zero.
- Correlation between RGB optical flow and noise-latent optical flow averages 0.727 on MSU benchmark frames.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MotionCraft instead of Text2Video-Zero | Zero-shot video generation · Dragons | Frame Consistency: 0.9664 -> 0.9991 | Helped |
| Use MotionCraft instead of Text2Video-Zero | Zero-shot video generation · Dragons | Motion Consistency: 0.6846 -> 0.9637 | Helped |
| Use MotionCraft instead of Text2Video-Zero | Zero-shot video generation · Melting Man | Frame Consistency: 0.9463 -> 0.9566 | Helped |
| Use MotionCraft instead of Text2Video-Zero | Zero-shot video generation · Melting Man | Motion Consistency: 0.7817 -> 0.8252 | Helped |
| Use MotionCraft instead of Text2Video-Zero | Zero-shot video generation · Satellite Scan | Frame Consistency: 0.9588 -> 0.9875 | Helped |
| Use MotionCraft instead of Text2Video-Zero | Zero-shot video generation · Satellite Scan | Motion Consistency: 0.2852 -> 0.9219 | Helped |
| Use MotionCraft instead of Text2Video-Zero | Zero-shot video generation · Revolving Earth | Frame Consistency: 0.9812 -> 0.9696 | Hurt |
| Use MotionCraft instead of Text2Video-Zero | Zero-shot video generation · Revolving Earth | Motion Consistency: 0.7213 -> 0.6783 | Hurt |
| Use MotionCraft instead of Text2Video-Zero | Zero-shot video generation · Birds | Frame Consistency: 0.9765 -> 0.9968 | Helped |
| Use MotionCraft instead of Text2Video-Zero | Zero-shot video generation · Birds | Motion Consistency: 0.8973 -> 0.9385 | Helped |
| Attend to both the initial frame and previous frame instead of using no MCFA or only one attended frame (ablation) | MotionCraft · Video generation scenes | Visual temporal consistency | Helped |
| Add Spatial-η sampling in selected image regions (ablation) | MotionCraft · Video generation scenes | Plausibility of novel content | Helped |
Surprising findings
- Text2Video-Zero Revolving Earth scene has higher Motion Consistency (0.7213) than MotionCraft (0.6783), and higher Frame Consistency on that scene too (0.9812 vs 0.9696).
- CLIP-based Frame Consistency can be high (0.97) even when object identities change across frames, motivating the new metric.
Limitations
- Relies on the pretrained Stable Diffusion and inherits its limitations, including inexact DDIM inversion.
- Global color shift grows over later frames; MCFA only partially mitigates it.
- Motion must come from a physics simulation, so dynamics hard to simulate (e.g., dancers) are not covered.
Experimental setup
Datasets: MSU Video Frame Interpolation Benchmark
Benchmarks: MotionCraft qualitative scenes (fluid, rigid body, agents)
Metrics: Frame Consistency (CLIP) Motion Consistency (SSIM after RAFT flow registration)
Baselines: Text2Video-Zero
Builds on: Text2Video-Zero Stable Diffusion DDIM classifier-free guidance RAFT
Closest papers by meaning
- Motion-Zero: A Zero-Shot Trajectory Control Framework of Moving Object for Diffusion-Based Video Generation (AAAI 2025 · similarity 0.88)
- PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation (NeurIPS 2025 · similarity 0.88)
- Scene Splatter: Momentum 3D Scene Generation from Single Image with Video Diffusion Model (CVPR 2025 · similarity 0.87)
- How Far Is Video Generation from World Model: A Physical Law Perspective (ICML 2025 · similarity 0.87)
- StochSync: Stochastic Diffusion Synchronization for Image Generation in Arbitrary Spaces (ICLR 2025 · similarity 0.87)
- Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noise (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.