Research map / Video / Video segmentation and tracking
Spectral Motion Alignment for Video Motion Transfer Using Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Spectral Motion Alignment (SMA), which adds wavelet-domain global and FFT-domain local loss terms to pixel/feature motion-vector distillation so video diffusion models transfer motion more faithfully.
Problem
Motion transfer with diffusion models typically estimates motion as pixel- or feature-space frame residuals, which miss global motion context and carry motion-independent spatial artifacts.
Approach
Align predicted and ground-truth motion vectors in the spectral domain: a Haar-wavelet multi-scale loss over temporal motion sequences captures global dynamics, and a low-frequency-weighted FFT amplitude/phase loss suppresses high-frequency artifacts. The losses are plug-in additions to existing motion-customization objectives.
Key results
- MotionDirector w/ SMA raises text-align from 0.7550 to 0.8081 and user motion accuracy from 2.67 to 3.88.
- VMC (Zeroscope) w/ SMA raises text-align from 0.8223 to 0.8425 and user motion accuracy from 2.47 to 4.07.
- Tune-A-Video w/ SMA raises text-align from 0.8289 to 0.8633 and frame consistency from 0.8951 to 0.9568.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SMA to MotionDirector | MotionDirector · Text-video pairs | Motion-Acc: 2.67 -> 3.88 | Helped |
| Add SMA to VMC | VMC (Show-1) and VMC (Zeroscope) · Text-video pairs | Motion-Acc: 2.72 -> 3.92; 2.47 -> 4.07 | Helped |
| Add SMA to Tune-A-Video | Tune-A-Video · Text-video pairs | Motion-Acc: 2.27 -> 3.25 | Helped |
| Add SMA to ControlVideo | ControlVideo · Text-video pairs | Motion-Acc: 2.69 -> 3.40 | Helped |
| Add local alignment loss (ablation) | VMC baseline · VMC ablation | Text-Align: 0.8163 -> 0.8382 | Helped |
| Add global alignment loss (ablation) | VMC baseline · VMC ablation | Text-Align: 0.8163 -> 0.8344 | Helped |
| Add both local and global alignment losses (ablation) | VMC baseline · VMC ablation | Text-Align: 0.8163 -> 0.8427 | Helped |
| Add local alignment loss (ablation) | VMC baseline · VMC ablation | Frame-Con: 0.9497 -> 0.9613 | Helped |
| Add global alignment loss (ablation) | VMC baseline · VMC ablation | Frame-Con: 0.9497 -> 0.9585 | Helped |
| Add both local and global alignment losses (ablation) | VMC baseline · VMC ablation | Frame-Con: 0.9497 -> 0.9659 | Helped |
Surprising findings
- DWT-based local refinement was also effective, but the authors found DFT outperformed DWT and chose DFT.
Limitations
- Evaluation uses a small curated set of 30 text-video pairs with short 8-16 frame clips.
- Hyperparameters (lambda_g, lambda_l, wavelet level) must be tuned per case.
Experimental setup
Datasets: DAVIS WebVid-10M
Benchmarks: curated 30 text-video pair set from DAVIS and WebVid-10M
Metrics: text-alignment CLIP cosine similarity frame consistency CLIP cosine similarity user study edit accuracy user study temporal consistency user study motion accuracy
Baselines: MotionDirector VMC DMT Tune-A-Video ControlVideo
Builds on: VMC MotionDirector Tune-A-Video DMT ControlVideo
Closest papers by meaning
- Video Diffusion Models are Training-free Motion Interpreter and Controller (NeurIPS 2024 · similarity 0.88)
- EDEN: Enhanced Diffusion for High-quality Large-motion Video Frame Interpolation (CVPR 2025 · similarity 0.87)
- Motion Consistency Model: Accelerating Video Diffusion with Disentangled Motion-Appearance Distillation (NeurIPS 2024 · similarity 0.86)
- MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion (ICLR 2025 · similarity 0.86)
- LaMo: Self-Supervised Latent Motion Priors for Physical Realism in Video Generation (NeurIPS 2026 · similarity 0.86)
- Video Motion Transfer with Diffusion Transformers (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.