Research map / Video / Video generation
Video Diffusion Models are Training-free Motion Interpreter and Controller
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that video diffusion features already encode motion; proposes MOFT (content-correlation removal plus motion-channel filtering) as a training-free feature for motion control and point-drag manipulation across AnimateDiff, ModelScope, and SVD-style models.
Problem
Video diffusion models encode cross-frame motion in their features, but this is not understood or exploited; existing motion control methods need training per architecture and are black-box.
Approach
PCA on intermediate U-Net features shows motion structure emerges after subtracting the across-frame mean, and only a few channels carry motion. Filtering these channels gives a motion feature (MOFT) that guides latent optimization for training-free motion control, combined with DIFT for point-drag.
Key results
- Top-4% motion channel filtering with MOFT reaches Motion Fidelity 84.0 vs 31.1 for vanilla features and 70.2 for SMM feature, with imaging quality 0.693 vs 0.697 unguided.
- In a 26-person user study, the method scores 3.21 faithfulness and 3.49 naturalness, versus Gen-2 MB 3.37/2.90 and DragNUWA 2.50/2.08.
- For point-drag, DIFT+MOFT reduces mean distance to 0.175 from 0.437 for DIFT alone, while DragNUWA achieves 0.075.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add content correlation removal to vanilla video features (ablation) | Motion feature guidance · Motion control | Motion Fidelity: 31.1 -> 67.1 | Helped |
| Add motion channel filtering after content correlation removal (ablation) | Motion feature guidance · Motion control | Motion Fidelity: 67.1 -> 82.5 | Helped |
| Use synthesized MOFT guidance instead of reference MOFT guidance | MOFT motion control · Motion control | Motion Fidelity: 82.5 -> 84.0 | Helped |
| Add MOFT guidance to DIFT point-drag manipulation (ablation) | DIFT point-drag manipulation · Point-drag manipulation | Mean Distance: 0.437 -> 0.175 | Helped |
| Use top 4% of motion channels instead of top 100% (ablation) | MOFT · Motion control | Motion Fidelity: 67.5 -> 84.0 | Helped |
| Reduce selected motion channels from top 4% to top 1% (ablation) | MOFT · Motion control | Motion Fidelity: 84.0 -> 76.3 | Hurt |
| Reduce selected motion channels from top 4% to top 1% (ablation) | MOFT · Motion control | Naturalness: 0.693 -> 0.677 | Hurt |
| Use MOFT for initial coarse control and DIFT for precise point-drag control instead of DIFT alone (ablation) | Point-drag manipulation · Point-drag manipulation | Point-control precision | Helped |
| Add shared K&V from the reference branch (ablation) | MOFT motion-control pipeline · Video consistency | Content consistency | Helped |
| Apply gradient clipping outside the masked region (ablation) | MOFT motion-control pipeline · Video consistency | Background consistency | Mixed |
Surprising findings
- Vanilla diffusion features show weak motion correspondence (fidelity 31.1), while mean-subtracting across frames raises it sharply.
- Keeping only the top 4% of motion channels outperforms using all channels, suggesting most channels add interfering content information.
Limitations
- Does not support motion control on real videos because video DDIM inversion significantly alters content.
- Cannot precisely control motion scale; only rough control via control weights or gradient clipping.
Experimental setup
Datasets: synthetic panning videos created by moving the whole picture
Benchmarks: 270 prompt-motion direction pairs (in-house) 40 image-motion direction pairs (in-house) user study with 26 people on 56 clips
Metrics: Motion Fidelity Imaging Quality (SPAQ-trained predictor) Mean Distance (drag precision) user faithfulness and naturalness ratings
Baselines: DragNUWA Gen-2 Motion Brush SMM feature DIFT
Builds on: AnimateDiff DIFT SparseCtrl FreeControl VideoFusion
Closest papers by meaning
- Motion Consistency Model: Accelerating Video Diffusion with Disentangled Motion-Appearance Distillation (NeurIPS 2024 · similarity 0.89)
- EDEN: Enhanced Diffusion for High-quality Large-motion Video Frame Interpolation (CVPR 2025 · similarity 0.89)
- Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition (ICLR 2024 · similarity 0.89)
- VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models (CVPR 2024 · similarity 0.88)
- Spectral Motion Alignment for Video Motion Transfer Using Diffusion Models (AAAI 2025 · similarity 0.88)
- Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.