Research map / Video / Video generation
RoPECraft: Training-Free Motion Transfer with Trajectory-Guided RoPE Optimization on Diffusion Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RoPECraft, a training-free video motion transfer method that warps RoPE tensors with optical flow and optimizes them during early denoising with a flow-matching loss plus Fourier phase regularization, evaluated on Wan2.1-1.3B against recent baselines.
Problem
Motion transfer from a reference video to a newly generated video using diffusion transformers, without costly fine-tuning or inversion, while preserving prompt faithfulness and avoiding duplicated subjects.
Approach
Modify only the rotary positional embeddings of a frozen video DiT: warp their complex-exponential tensors with optical-flow displacements per row and column, refine them during early denoising steps via a flow-matching trajectory-alignment objective, and regularize with a phase-based Fourier loss.
Key results
- Achieves MF 0.5816 and CD-FVD 1284.58 on DAVIS, the best among compared methods, versus GWTF at MF 0.5713 and CD-FVD 1485.23.
- Reports FTD of 0.2644 (FG) and 0.2584 (FG+BG), second best behind GWTF's 0.2457 and 0.2308.
- Runtime of about 109 s per video, comparable to SMM, DitFlow, and MOFT and faster than ConMo (150.7 s).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RoPECraft instead of GWTF | Wan2.1-1.3B motion transfer on DAVIS · DAVIS | MF: 0.5713$\pm$0.22 -> 0.5816$\pm$0.19 | Helped |
| Use RoPECraft instead of GWTF | Wan2.1-1.3B motion transfer on DAVIS · DAVIS | CD-FVD: 1485.23 -> 1284.58 | Helped |
| Use RoPECraft instead of GWTF | Wan2.1-1.3B motion transfer on DAVIS · DAVIS | CLIP similarity: 0.2378$\pm$0.04 -> 0.2350$\pm$0.04 | Hurt |
| Use RoPECraft instead of GWTF | Wan2.1-1.3B motion transfer on DAVIS · DAVIS | FTD (FG): 0.2457$\pm$0.14 -> 0.2644$\pm$0.14 | Hurt |
| Use RoPECraft instead of GWTF | Wan2.1-1.3B motion transfer on DAVIS · DAVIS | FTD (FG+BG): 0.2308$\pm$0.10 -> 0.2584$\pm$0.13 | Hurt |
| Add phase constraint to motion-augmented RoPE and optimization (ablation) | RoPECraft · DAVIS ablation | MF: 0.7092 -> 0.7210 | Helped |
| Add phase constraint to motion-augmented RoPE and optimization (ablation) | RoPECraft · DAVIS ablation | FTD: 0.2105 -> 0.2060 | Helped |
| Use (t,s)=(10,10) instead of (10,5) (ablation) | RoPECraft · DAVIS hyperparameter ablation | MF: 0.5675 -> 0.6160 | Helped |
| Use (t,s)=(10,10) instead of (10,5) (ablation) | RoPECraft · DAVIS hyperparameter ablation | CLIP: 0.1664 -> 0.1572 | Hurt |
| Use (t,s)=(10,10) instead of (10,5) (ablation) | RoPECraft · DAVIS hyperparameter ablation | CD-FVD: 1364.25 -> 1492.86 | Hurt |
| Use RoPECraft instead of ConMo | Wan2.1-1.3B motion transfer on RealCamVid · RealCamVid | MF: 0.5973 -> 0.7019 | Helped |
| Use RoPECraft instead of GWTF | Wan2.1-1.3B motion transfer on RealCamVid · RealCamVid | FTD: 0.2072 -> 0.1697 | Helped |
Surprising findings
- GWTF achieves the best FTD scores on DAVIS, outperforming the proposed method on that metric.
- Motion Fidelity can report near-perfect similarity for trajectories that diverge geometrically, motivating the proposed FTD metric.
Limitations
- Relies on CoTracker3 for FTD, which struggles with zoom camera motion and stationary tracking points.
- Optical flow extractor struggles with occluded or missing subjects, causing distorted figures.
- Backbone limitations cause intermittent object appearance, such as a boat disappearing.
Experimental setup
Datasets: DAVIS RealCamVid
Benchmarks: DAVIS RealCamVid
Metrics: Motion Fidelity CD-FVD CLIP similarity Fréchet Trajectory Distance user preference
Baselines: GWTF SMM MOFT DitFlow (latents) DitFlow (RoPE) ConMo
Builds on: DiTFlow Go With The Flow RoPE flow matching
Closest papers by meaning
- Video Motion Transfer with Diffusion Transformers (CVPR 2025 · similarity 0.90)
- FasterDiT: Towards Faster Diffusion Transformers Training without Architecture Modification (NeurIPS 2024 · similarity 0.87)
- BlockDance: Reuse Structurally Similar Spatio-Temporal Features to Accelerate Diffusion Transformers (CVPR 2025 · similarity 0.87)
- RIFLEx: A Free Lunch for Length Extrapolation in Video Diffusion Transformers (ICML 2025 · similarity 0.86)
- TinyFusion: Diffusion Transformers Learned Shallow (CVPR 2025 · similarity 0.85)
- FlexiDiT: Your Diffusion Transformer Can Easily Generate High-Quality Samples with Less Compute (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.