Research map / Video / Video generation
Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Zero-shot text-driven motion transfer that guides a frozen ZeroScope text-to-video diffusion model with a pairwise spatial-marginal-mean space-time feature loss to keep an input video's motion while following a new text prompt.
Problem
Text-driven motion transfer: synthesizing a video that follows a target text prompt (new object and scene) while preserving the motion and scene layout of an input video, especially when source and target differ strongly in shape and fine-grained motion.
Approach
Analyzes space-time diffusion features of a pretrained T2V model and introduces Spatial Marginal Mean (SMM) descriptors per frame; guidance at each denoising step preserves pairwise SMM differences (not exact values) between the source and generated videos, with low-frequency filtered initialization from DDIM-inverted noise.
Key results
- On 54 video-edit pairs from 21 videos, the method achieves better balance of CLIP text similarity and Motion-Fidelity than TokenFlow, Control-A-Video, Tune-A-Video and SDEdit.
- User study with 7000 judgments from 150 users: our results preferred over TokenFlow 72.57%, Control-A-Video 84.50%, Tune-A-Video 77.80%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use pairwise SMM differences instead of full space-time feature reconstruction (ablation) | Ours · Text-driven motion transfer | edit flexibility | Helped |
| Use pairwise SMM differences instead of direct SMM feature reconstruction (ablation) | Ours · Text-driven motion transfer | edit fidelity | Helped |
| Remove feature-guidance optimization (ablation) | Ours · Text-driven motion transfer | motion fidelity | Hurt |
| Replace low-frequency initialization with randomly sampled noise (ablation) | Ours · Text-driven motion transfer | motion fidelity | Hurt |
| Use Ours instead of TokenFlow | Text-driven video editing · 54 video-edit text pairs | motion fidelity | Mixed |
| Use Ours instead of Tune-A-Video | Text-driven video editing · 54 video-edit text pairs | motion fidelity | Helped |
Surprising findings
- Spatial marginal mean features, which collapse spatial dimensions, still retain object pose and position while allowing larger structural changes than full space-time features.
Limitations
- Motion fidelity degrades when the target object and input motion are out-of-distribution for the T2V model.
- Public T2V models are limited in quality, resolution, and video length.
- Relies on tracking-based metric and off-the-shelf tracker for evaluation.
Experimental setup
Datasets: DAVIS Web videos
Benchmarks: DAVIS
Metrics: CLIP text similarity Motion-Fidelity score (tracklet correlation) 2AFC user preference
Baselines: SA-NLA TokenFlow Gen-1 Control-A-Video Tune-A-Video SDEdit
Builds on: DDIM inversion Stable Diffusion ZeroScope DIFT features SDEdit
Closest papers by meaning
- Motion-Zero: A Zero-Shot Trajectory Control Framework of Moving Object for Diffusion-Based Video Generation (AAAI 2025 · similarity 0.89)
- Enhancing Motion in Text-to-Video Generation with Decomposed Encoding and Conditioning (NeurIPS 2024 · similarity 0.88)
- Enhanced Fine-Grained Motion Diffusion for Text-Driven Human Motion Synthesis (AAAI 2024 · similarity 0.88)
- EIDT-V: Exploiting Intersections in Diffusion Trajectories for Model-Agnostic, Zero-Shot, Training-Free Text-to-Video Generation (CVPR 2025 · similarity 0.88)
- Towards Detailed Text-to-Motion Synthesis via Basic-to-Advanced Hierarchical Diffusion Model (AAAI 2024 · similarity 0.88)
- TIV-Diffusion: Towards Object-Centric Movement for Text-driven Image to Video Generation (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.