Research map / Video / Video generation
VideoDirector: Precise Video Editing via Text-to-Video Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VideoDirector, a training-free method that performs precise text-driven video editing directly on a text-to-video model (AnimateDiff) using spatial-temporal decoupled guidance, multi-frame null-text optimization, and self/cross-attention control.
Problem
Video editing via text prompts with T2I-based methods yields poor temporal coherence and realism; directly applying the inversion-then-editing paradigm (DDIM inversion, null-text optimization, prompt-to-prompt) to T2V models causes color flickering and content distortion.
Approach
Makes pivotal inversion work on T2V models by adding spatial-temporal decoupled guidance (temporal and key-based gradients over SAM2 foreground/background masks) and per-frame multi-frame null-text embeddings, plus a self-attention control scheme (SA-I/SA-II with masked mutual attention) alongside cross-attention control.
Key results
- Achieves 97.68% motion smoothness vs. 96.69% for TokenFlow (best baseline) in Table 1.
- Masked LPIPS of 0.270 vs. 0.313 for TokenFlow, and masked PSNR of 21.37 vs. 17.94, indicating better preservation of unedited content.
- Ablation shows removing STDG drops motion smoothness from 97.68% to 89.23% and PickScore changes little (20.39 vs 21.64).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add spatial-temporal decoupled guidance (ablation) | VideoDirector · Video editing pairs | m.L: 0.369 -> 0.270 | Helped |
| use multi-frame rather than shared null-text embeddings (ablation) | VideoDirector · Video editing pairs | m.L: 0.346 -> 0.270 | Helped |
| add cross-attention control (ablation) | VideoDirector · Video editing pairs | m.L: 0.301 -> 0.270 | Helped |
| add self-attention control (ablation) | VideoDirector · Video editing pairs | m.L: 0.537 -> 0.270 | Helped |
| add self-attention control-I (ablation) | VideoDirector · Video editing pairs | m.L: 0.418 -> 0.270 | Helped |
| add self-attention control-II (ablation) | VideoDirector · Video editing pairs | m.L: 0.371 -> 0.270 | Helped |
| use VideoDirector instead of T2I-based video editing baselines | video editing methods · Video editing pairs | m.P: 21.37 vs 17.94 | Helped |
Surprising findings
- Directly applying prompt-to-prompt and null-text inversion to a T2V model yields severe color flickering and content distortion, motivating the method.
Limitations
- Fixed to 16 frames due to high memory consumption of AnimateDiff.
- Thresholds and guidance coefficients (tau_s, eta_b, zeta_b) manually tuned per input video.
- Evaluation on only 75 text-video pairs with a 9-participant user study.
Experimental setup
Datasets: DAVIS TokenFlow videos MotionClone videos online video platform clips
Benchmarks: VBench (motion smoothness)
Metrics: PickScore masked PSNR masked LPIPS VBench motion smoothness user study rank
Baselines: Video-P2P RAVE Flatten TokenFlow
Builds on: Prompt-to-Prompt Null-text inversion MotionClone AnimateDiff DDIM inversion
Closest papers by meaning
- Video-P2P: Video Editing with Cross-attention Control (CVPR 2024 · similarity 0.90)
- A Video is Worth 256 Bases: Spatial-Temporal Expectation-Maximization Inversion for Zero-Shot Video Editing (CVPR 2024 · similarity 0.89)
- SimDA: Simple Diffusion Adapter for Efficient Video Generation (CVPR 2024 · similarity 0.88)
- Re-Attentional Controllable Video Diffusion Editing (AAAI 2025 · similarity 0.88)
- Slicedit: Zero-Shot Video Editing With Text-to-Image Diffusion Models Using Spatio-Temporal Slices (ICML 2024 · similarity 0.88)
- COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.