Research map / Video / Video generation
MotionClone: Training-Free Motion Cloning for Controllable Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MotionClone, a training-free method that clones motion from a reference video by guiding video diffusion sampling with sparse, top-k temporal-attention maps extracted via a single denoising step.
Problem
Motion-controllable video generation usually needs model training or fine-tuning to encode motion cues, which limits flexibility and generalization; dense depth/sketch cues also entangle motion with reference structure.
Approach
Uses sparse (top-k, k=1) temporal-attention weights from a reference video as motion representation, guiding sampling via an energy-based loss, and extracts this representation from a single noise-then-denoise step at t=400 instead of costly DDIM inversion.
Key results
- Textual alignment 0.3187 (CLIP) vs 0.3134 for next best VMC; temporal consistency 0.9621 vs 0.9614 for VMC.
- User study motion preservation 3.69 vs 3.50 (Gen-1) and appearance diversity 4.31 vs 3.51 (VMC).
- Motion representation extracted with a single denoising step at t=400, avoiding DDIM inversion costs, with no clear quality loss versus inversion variants.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sparse primary temporal-attention control instead of uniform plain control | MotionClone · Motion cloning in novel scenarios | motion alignment | Helped |
| Use MotionClone instead of VMC | MotionClone · 40 real videos | Textual Alignment (CLIP cosine similarity): 0.3134 -> 0.3187 | Helped |
| Use MotionClone instead of VMC | MotionClone · 40 real videos | Temporal Consistency (CLIP similarity): 0.9614 -> 0.9621 | Helped |
| Use MotionClone instead of Gen-1 | MotionClone · 40 real videos | Motion Preservation (user study): 3.50 -> 3.69 | Helped |
| Use MotionClone instead of VMC | MotionClone · 40 real videos | Appearance Diversity (user study): 3.51 -> 4.31 | Helped |
| Use MotionClone instead of VMC | MotionClone · 40 real videos | Textual Alignment (user study): 3.79 -> 4.15 | Helped |
| Use MotionClone instead of Gen-1 | MotionClone · 40 real videos | Temporal Consistency (user study): 3.34 -> 4.28 | Helped |
| Lower the top-k mask value (ablation) | MotionClone · Motion cloning | primary motion alignment | Helped |
| Set extraction timestep to 800 instead of 200, 400, or 600 (ablation) | MotionClone · Motion cloning | motion alignment: 200,400, 600; 800 | Hurt |
| Apply motion guidance in up_block.1 rather than other blocks (ablation) | MotionClone · Motion cloning | motion manipulation while safeguarding visual quality | Helped |
| Use a tailored prompt rather than null-text when preparing the motion representation (ablation) | MotionClone · Motion representation preparation | generation quality | No clear effect |
| Apply DDIM inversion (ablation) | MotionClone · Motion cloning | generation quality | No clear effect |
Surprising findings
- Using the full (plain) temporal attention map for guidance gives limited motion transfer; sparse top-1 control is substantially better.
- Tailored prompts when preparing motion representations make little difference, since motion information is preserved in diffusion features at t=400.
Limitations
- Struggles with local subtle motions (e.g., winking) due to low spatial resolution of temporal attention features in latent space.
- Quality drops when multiple moving objects overlap.
Experimental setup
Datasets: DAVIS online-sourced videos
Benchmarks: 40 real reference videos (15 camera motion, 25 object motion)
Metrics: CLIP textual alignment CLIP temporal consistency user study: motion preservation user study: appearance diversity user study: textual alignment user study: temporal consistency
Baselines: VideoComposer Tune-A-Video Control-A-Video VMC Gen-1
Builds on: AnimateDiff SparseCtrl classifier-free guidance DDIM inversion
Closest papers by meaning
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.89)
- Motion Prompting: Controlling Video Generation with Motion Trajectories (CVPR 2025 · similarity 0.88)
- MotionPro: A Precise Motion Controller for Image-to-Video Generation (CVPR 2025 · similarity 0.88)
- Enhancing Motion in Text-to-Video Generation with Decomposed Encoding and Conditioning (NeurIPS 2024 · similarity 0.88)
- IM-Zero: Instance-level Motion Controllable Video Generation in a Zero-shot Manner (CVPR 2025 · similarity 0.87)
- ConMo: Controllable Motion Disentanglement and Recomposition for Zero-Shot Motion Transfer (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.