Research map / Video / Video generation
Motion Prompting: Controlling Video Generation with Motion Trajectories
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a track-conditioned ControlNet adapter on Lumiere using dense point tracks from 2.2M videos, enabling object, camera, and motion control via "motion prompts" and expansion of user inputs.
Problem
Text prompts are insufficient for controlling nuanced motion in video generation; sparse and dense motion specification needs a flexible, unified representation and practical user interface.
Approach
Represent motion as spatio-temporally sparse or dense point trajectories encoded as a space-time volume of random track embeddings, train a single-stage ControlNet on dense tracks from unfiltered videos, and convert high-level user inputs (mouse drags, camera paths, primitives) into detailed tracks via "motion prompt expansion".
Key results
- With 2048 tracks on DAVIS, the method reaches PSNR 19.327, SSIM 0.608, LPIPS 0.227, FVD 655.9, EPE 3.887, outperforming Image Conductor and DragAnything.
- In a 2AFC human study, the method is preferred over Image Conductor and DragAnything with win rates of 73.7-80.5% across motion adherence, motion quality, and visual quality.
- Ablation shows training on dense tracks outperforms sparse-only and dense+sparse training, even on sparse-track inference (N=4 FVD/EPE improved).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use motion-prompted video generation rather than Image Conductor | Lumiere-based motion-prompted model · DAVIS validation, N=2048 | PSNR: 11.609 -> 19.327 | Helped |
| use motion-prompted video generation rather than DragAnything | Lumiere-based motion-prompted model · DAVIS validation, N=2048 | EPE: 12.485 -> 3.887 | Helped |
| use motion-prompted video generation rather than DragAnything | Lumiere-based motion-prompted model · DAVIS validation, N=1 | EPE: 9.135 -> 14.619 | Hurt |
| train with dense rather than sparse tracks (ablation) | Motion Prompting · DAVIS ablation, N=4 | PSNR: 15.075 -> 15.638 | Helped |
| train with dense rather than sparse tracks (ablation) | Motion Prompting · DAVIS ablation, N=4 | EPE: 30.712 -> 24.553 | Helped |
| train with dense rather than a dense-and-sparse mixture (ablation) | Motion Prompting · DAVIS ablation, N=4 | PSNR: 15.162 -> 15.638 | Helped |
| train with dense rather than a dense-and-sparse mixture (ablation) | Motion Prompting · DAVIS ablation, N=4 | EPE: 29.466 -> 24.553 | Helped |
| train with dense rather than sparse tracks (ablation) | Motion Prompting · DAVIS ablation, N=2048 | PSNR: 15.697 -> 19.197 | Helped |
| train with dense rather than sparse tracks (ablation) | Motion Prompting · DAVIS ablation, N=2048 | EPE: 26.724 -> 4.806 | Helped |
| train with dense rather than a dense-and-sparse mixture (ablation) | Motion Prompting · DAVIS ablation, N=2048 | PSNR: 15.294 -> 19.197 | Helped |
| train with dense rather than a dense-and-sparse mixture (ablation) | Motion Prompting · DAVIS ablation, N=2048 | EPE: 27.931 -> 4.806 | Helped |
Surprising findings
- The loss is not correlated with the model's track-following performance.
- Dense-only training works better than sparse training even when evaluated on sparse tracks.
Limitations
- Generation is not real-time or causal (about 12 minutes per video).
- Failures from motion conditioning (e.g., locked horns) and from the underlying video model (e.g., spontaneous chess piece creation).
- Composing object and camera motion is an approximation in 2D and fails for extreme camera motion.
Experimental setup
Datasets: internal video dataset of 2.2M videos DAVIS validation set
Benchmarks: DAVIS
Metrics: PSNR SSIM LPIPS FVD EPE
Baselines: Image Conductor DragAnything
Builds on: Lumiere ControlNet BootsTAP
Closest papers by meaning
- Move-in-2D: 2D-Conditioned Human Motion Generation (CVPR 2025 · similarity 0.90)
- Optical-Flow Guided Prompt Optimization for Coherent Video Generation (CVPR 2025 · similarity 0.90)
- Enhancing Motion in Text-to-Video Generation with Decomposed Encoding and Conditioning (NeurIPS 2024 · similarity 0.89)
- OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllers (CVPR 2024 · similarity 0.89)
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.89)
- Make Pixels Dance: High-Dynamic Video Generation (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.