Research map / Video / Video generation
Motion Modes: What Could Happen Next?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Motion Modes is a training-free method that guides the denoising of a pretrained flow-based image-to-video generator (Motion-I2V) with energy functions to disentangle object motion from camera motion and iteratively sample diverse, plausible object motions from a single image.
Problem
Predicting diverse, plausible object motions from a single static image while disentangling them from camera motion and other scene changes, without relying on synthetic training data or user-specified motion arrows.
Approach
Use inference-time guidance energies (static camera, object motion, diversity repulsion, smoothness) on a pretrained motion-field diffusion generator, iteratively sampling motions with a stopping criterion, instead of training or fine-tuning any model.
Key results
- Motion Modes achieves the best diversity energy (1.04 vs 1.21-1.77 for baselines?) with focus energies near zero, e.g., E_f 0.07 vs 1.11-2.20 for other methods in Table 1 (diversity energy 1.04 vs Prompt Gen 1.28 and random noise 1.27 with lower-is-better convention for focus).
- In a user study with 32 participants, motions from Motion Modes were judged more plausible, diverse, and expected than baselines across 320 comparisons per baseline.
- In a second study, 96% of motions were judged plausible, 92% of participant expectations were produced, and 19% were plausible but outside expectation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add static camera guidance Ec (ablation) | Motion Modes · 28 input images | camera guidance energy: 1.29 -> 0.09 | Helped |
| add object motion guidance Eo (ablation) | Motion Modes · 28 input images | object motion guidance energy: 1.75 -> 0.05 | Helped |
| add diversity guidance Ed (ablation) | Motion Modes · 28 input images | diversity energy: 1.36 -> 1.04 | Helped |
| use diversity guidance Ed instead of farthest point sampling (ablation) | Motion Modes · 28 input images | diversity energy: 1.49 -> 1.04 | Helped |
| use static camera and object motion guidance instead of ControlNet (ablation) | Motion Modes · 28 input images | focus energy: 0.80 -> 0.07 | Helped |
| add smoothness guidance Es | Motion Modes · input images | plausibility | Helped |
| use Motion Modes guidance rather than baseline sampling methods | Motion-I2V object motion generation · 28 input images | diversity energy: Motion Modes: 1.04; baselines: 1.21, 1.27, 1.28, 1.75, 1.77 | Helped |
| use Motion Modes guidance rather than baseline sampling methods | Motion-I2V object motion generation · 28 input images | focus energy: Motion Modes: 0.07; baselines: 1.14, 1.17, 1.71, 1.98, 2.20 | Helped |
Surprising findings
- Prompt-based GPT-4o baseline achieves relatively high diversity, close to Motion Modes, though it is less focused.
- ControlNet and random-arrow baselines look focused mainly because they produce little overall camera or object motion.
Limitations
- Inherits biases and quality limits of the underlying video generator (e.g., bent handles, extra cat tails).
- Produces discrete motion samples, covering only a continuous space of plausible motions partially.
- Requires one forward pass per extracted motion, which is slow; cannot handle moving-camera motion.
Experimental setup
Datasets: 28-image evaluation set from state-of-the-art text-to-image generator outputs and photographs
Benchmarks: Motion Modes evaluation set of 28 images
Metrics: diversity guidance energy focus energy (object motion and static camera energies) user study plausibility user study diversity user study expectedness inspirational rate
Baselines: Prompt Generation (GPT-4o + Motion-I2V) ControlNet (MotionBrush) Random Arrows (MotionDrag) Random Noise FPS Noise
Builds on: Motion-I2V particle guidance classifier-free guidance ControlNet
Closest papers by meaning
- PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation (NeurIPS 2025 · similarity 0.87)
- How Far Is Video Generation from World Model: A Physical Law Perspective (ICML 2025 · similarity 0.87)
- VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models (ICML 2025 · similarity 0.86)
- ObjectMover: Generative Object Movement with Video Prior (CVPR 2025 · similarity 0.86)
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.86)
- MotionCraft: Physics-Based Zero-Shot Video Generation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.