Research map / Video / Video generation
Motion-Zero: A Zero-Shot Trajectory Control Framework of Moving Object for Diffusion-Based Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Motion-Zero is a training-free, plug-and-play inference-time framework that controls the bounding-box trajectory of a single moving object in text-to-video diffusion models via initial-noise priors, attention-based spatial losses, and shifted temporal attention.
Problem
Controlling the motion trajectory of objects in videos generated by text-to-video diffusion models, without the large-scale motion-trajectory training that existing methods need and without being tied to a single base model.
Approach
Combines an Initial Noise Prior Module (DDIM inversion of a meta video with box-local latent mixing), spatial constraints that apply attention-map losses (box, outside-box, center, and appearance similarity) to update latents at each denoising step, and a Shift Temporal Attention Mechanism that aligns box contents across frames, all applied at inference only.
Key results
- On ZeroScope, Motion-Zero raised Text Align from 20.31 to 21.96 and mIoU to 0.54 with AP50 0.64, versus TrailBlazer mIoU 0.52 and Peekaboo 0.43.
- On ModelScope, Motion-Zero improved Text Align from 20.55 to 23.54 and reached AP50 0.64 while keeping control metrics high.
- User study (60 participants) rated Motion-Zero highest on Appearance (4.75), Consistency (4.57), and Control (4.67) out of 5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Motion-Zero to ZeroScope | ZeroScope · 33-prompt evaluation | mIoU: 0.54 vs 0.52 | Helped |
| Apply Motion-Zero to ZeroScope | ZeroScope · 33-prompt evaluation | Text Align: 20.31 -> 21.96 | Helped |
| Add Initial Noise Prior Module (ablation) | Motion-Zero · 33-prompt evaluation | Cov.: 0.86 -> 0.96 | Helped |
| Add Spatial Constraints (ablation) | Motion-Zero · 33-prompt evaluation | mIoU: 0.18 -> 0.54 | Helped |
| Add Spatial Constraints (ablation) | Motion-Zero · 33-prompt evaluation | AP50: 0.17 -> 0.64 | Helped |
| Add Shift Temporal Attention Mechanism (ablation) | Motion-Zero · 33-prompt evaluation | Cov.: 0.83 -> 0.96 | Helped |
Surprising findings
- Removing the spatial constraints hurt location control most (mIoU fell from 0.54 to 0.18), while removing INPM or STAM mainly hurt text alignment and generation quality.
Limitations
- Controls only a single object trajectory per video.
- Spatial-constraint optimization of latents at each step adds per-step gradient computation.
- Evaluation uses a small set of 33 prompts and relies on a detector (OWL-ViT) for control metrics.
Experimental setup
Datasets: 33 prompts with moving objects and motion patterns (following TrailBlazer)
Benchmarks: 33-prompt trajectory evaluation set (8 simple, 17 complex trajectories)
Metrics: Text Align (CLIP score) Consistency PickScore mIoU AP50 Cov. CD user study scores
Baselines: TrailBlazer Peekaboo ZeroScope +Prompt ModelScope +Prompt VideoComposer DragNUWA
Builds on: DDIM ZeroScope ModelScope TrailBlazer
Closest papers by meaning
- IM-Zero: Instance-level Motion Controllable Video Generation in a Zero-shot Manner (CVPR 2025 · similarity 0.89)
- SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generation (ICLR 2025 · similarity 0.89)
- Training-free Camera Control for Video Generation (ICLR 2025 · similarity 0.89)
- Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer (CVPR 2024 · similarity 0.89)
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.88)
- Motion Guidance: Diffusion-Based Image Editing with Differentiable Motion Estimators (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.