Research map / Video / Video generation
Video Motion Transfer with Diffusion Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiTFlow, a training-free, optimization-based video motion transfer method for diffusion transformers that guides latents (or positional embeddings) with a cross-frame Attention Motion Flow loss extracted from a reference video.
Problem
Transferring motion from a reference video to newly generated video with diffusion transformers, where existing motion transfer methods were designed for UNet architectures with separate temporal attention.
Approach
Extract a patch-wise Attention Motion Flow (displacement maps from cross-frame attention argmax) in a DiT and optimize latents or positional embeddings with a soft-argmax AMF loss during denoising, enabling motion transfer and zero-shot reuse of optimized positional embeddings.
Key results
- DiTFlow reaches MF 0.785 on CogVideoX-5B and 0.726 on CogVideoX-2B across Caption/Subject/Scene prompts, versus 0.766 and 0.688 for the best baseline SMM.
- Human study (66 preferences from 30 people) shows DiTFlow significantly outperforms MOFT and SMM in motion and prompt adherence.
- Optimizing positional embeddings enables zero-shot prompt changes with smaller prompt-adherence loss (-0.3%) than latent optimization (-4.4%), though motion preservation drops more (-5.2% vs -0.2%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AMF-guided latent optimization instead of SMM | CogVideoX-5B · DAVIS, all prompts | MF: 0.766 -> 0.785 | Helped |
| use AMF-guided latent optimization instead of SMM | CogVideoX-2B · DAVIS, all prompts | MF: 0.688 -> 0.726 | Helped |
| use AMF-guided latent optimization instead of SMM | CogVideoX-5B · DAVIS, all prompts | IQ: 0.315 -> 0.319 | Helped |
| use AMF-guided latent optimization instead of SMM | CogVideoX-2B · DAVIS, all prompts | IQ: 0.312 -> 0.317 | Helped |
| optimize at block 20 rather than block 0, 10, or 30 (ablation) | DiTFlow on CogVideoX-5B · DAVIS | MF: 0.620, 0.670, 0.797, 0.558 | Helped |
| increase guidance from 0% to 20% of denoising steps (ablation) | DiTFlow on CogVideoX-5B · DAVIS | MF: 0.623 -> 0.797 | Helped |
| increase guidance from 20% to 40% of denoising steps (ablation) | DiTFlow on CogVideoX-5B · DAVIS | MF: 0.797 -> 0.813 | Helped |
| increase guidance from 20% to 80% of denoising steps (ablation) | DiTFlow on CogVideoX-5B · DAVIS | MF: 0.797 -> 0.803 | Helped |
| increase guidance from 20% to 100% of denoising steps (ablation) | DiTFlow on CogVideoX-5B · DAVIS | MF: 0.797 -> 0.799 | Helped |
| increase optimization steps from 1 to 5 (ablation) | DiTFlow on CogVideoX-5B · DAVIS | MF: 0.769 -> 0.797 | Helped |
| increase optimization steps from 5 to 10 (ablation) | DiTFlow on CogVideoX-5B · DAVIS | MF: 0.797 -> 0.803 | Helped |
| optimize positional embeddings rather than latents for zero-shot generation (ablation) | DiTFlow on CogVideoX-5B · DAVIS, new prompts without further optimization | IQ: -4.4% vs -0.3% | Helped |
Surprising findings
- Optimizing positional embeddings preserves prompt adherence in zero-shot settings, whereas optimizing latents leaks content from the optimization prompt into new generations.
Limitations
- Per-video optimization roughly adds computation time (about 1.6x for CogVideoX-5B).
- Zero-shot positional-embedding variant has lower motion fidelity than per-video latent optimization.
- Evaluation uses a single dataset (50 DAVIS videos, 150 motion-prompt pairs).
Experimental setup
Datasets: DAVIS
Benchmarks: DAVIS
Metrics: Motion Fidelity (MF) Image Quality (CLIPScore-based IQ) Likert-5 human preference
Baselines: Backbone Injection MotionClone SMM MOFT
Builds on: MOFT SMM MotionClone DDIM inversion CogVideoX
Closest papers by meaning
- Emergent Temporal Correspondences from Video Diffusion Transformers (NeurIPS 2025 · similarity 0.90)
- RoPECraft: Training-Free Motion Transfer with Trajectory-Guided RoPE Optimization on Diffusion Transformers (NeurIPS 2025 · similarity 0.90)
- EDEN: Enhanced Diffusion for High-quality Large-motion Video Frame Interpolation (CVPR 2025 · similarity 0.89)
- RIFLEx: A Free Lunch for Length Extrapolation in Video Diffusion Transformers (ICML 2025 · similarity 0.88)
- Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer (CVPR 2024 · similarity 0.87)
- Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.