Research map / Video / Video generation
Video-P2P: Video Editing with Cross-attention Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Video-P2P adapts a pre-trained image diffusion model (Stable Diffusion v1-5), inflated into a text-to-set model, to perform cross-attention-controlled text-driven video editing using shared unconditional embedding optimization and decoupled guidance.
Problem
Text-driven local video editing with attention control is unavailable because no large-scale pre-trained video generation models are publicly available, and frame-by-frame image editing lacks cross-frame semantic consistency.
Approach
Inflate a pre-trained image diffusion model into a text-to-set (T2S) model with frame-attention, fine-tune it on the input video for an approximate inversion, optimize a single shared unconditional embedding for accurate video inversion, and use decoupled guidance (optimized embedding for source prompt, initialized embedding for target prompt) with attention-map injection.
Key results
- Video-P2P achieves CLIP 0.3367, Masked PSNR 20.65, LPIPS 0.3213, OSV 47.38, Temp 0.9725 on 45-sample DAVIS test set, outperforming TAV+DDIM (Masked PSNR 17.37, LPIPS 0.4651) on preservation metrics.
- Shared unconditional embedding reconstructs video at 22.75 dB PSNR with 2.94M parameters versus 22.68M for per-frame multi-embedding (22.97 dB) and 15.43 dB for TAV+DDIM.
- In a user study, Video-P2P ranked first on structure preserving (1.39 average rank, 69.02% preference), text alignment (1.54, 62.50%), and realism and quality (1.43, 69.02%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| decoupled-guidance attention control (optimized embedding for source, initialized embedding for target) (ablation) | Video-P2P · DAVIS-based test set (45 samples) | Masked PSNR: 18.97 -> 20.65 | Helped |
| decoupled-guidance attention control (ablation) | Video-P2P · DAVIS-based test set (45 samples) | LPIPS: 0.3866 -> 0.3213 | Helped |
| decoupled-guidance attention control (ablation) | Video-P2P · DAVIS-based test set (45 samples) | OSV (lower is better): 68.75 -> 47.38 | Helped |
| shared unconditional embedding optimization (ablation) | Video-P2P · DAVIS-based test set (45 samples) | Temp (temporal consistency): 0.9665 -> 0.9725 | Helped |
| shared unconditional embedding optimization (ablation) | Video-P2P · DAVIS-based test set (45 samples) | OSV (lower is better): 48.78 -> 47.38 | Helped |
| shared unconditional embedding optimization (ablation) | Video-P2P · DAVIS-based test set (45 samples) | Masked PSNR: 20.64 -> 20.65 | No clear effect |
| model initialization (fine-tuning T2S model on the input video) (ablation) | Video-P2P · DAVIS-based test set (45 samples) | OSV (lower is better): 50.27 -> 47.38 | Helped |
| model initialization (fine-tuning T2S model on the input video) (ablation) | Video-P2P · DAVIS-based test set (45 samples) | Masked PSNR: 20.62 -> 20.65 | Helped |
| shared unconditional embedding vs per-frame multiple unconditional embeddings (ablation) | Video inversion reconstruction · Video reconstruction (PSNR) | PSNR (dB): 22.97 (multi) vs 22.75 (shared) | No clear effect |
| shared unconditional embedding optimization instead of TAV+DDIM inversion (ablation) | Video reconstruction · Video reconstruction (PSNR) | PSNR (dB): 15.43 -> 22.75 | Helped |
| Video-P2P framework vs Image-P2P/TAV+DDIM baselines | Video editing on DAVIS-based test set · DAVIS-based test set (45 samples) | OSV and Temp consistency: OSV 47.38 vs 55.67 (TAV+DDIM); Temp 0.9725 vs 0.9697 | Helped |
| Video-P2P vs TAV+DDIM on masked region preservation | Video editing on DAVIS-based test set · DAVIS-based test set (45 samples) | Masked PSNR and LPIPS: Masked PSNR 20.65 vs 17.37; LPIPS 0.3213 vs 0.4651 | Helped |
Surprising findings
- Using per-frame multiple unconditional embeddings barely improves PSNR (+0.2 dB over shared) while multiplying parameter usage by n, and about 10% of videos became unstable with multiple embeddings.
Limitations
- Editing longer videos (over 1 minute) remains challenging; most experiments use 8 or 24 frames.
- Cannot perform video motion editing due to lack of temporal priors.
- Inversion cost (null-text inversion) makes the pipeline slow; faster inversion methods are suggested but not used.
Experimental setup
Datasets: DAVIS
Benchmarks: DAVIS-based test set of 45 samples
Metrics: CLIP score Masked PSNR LPIPS Object Semantic Variance (OSV) Temp (CLIP frame similarity) user study ranking and preference rate PSNR reconstruction
Baselines: Tune-A-Video (TAV+DDIM) Image-P2P Dreamix Text2Live Gen-1
Builds on: Prompt-to-Prompt Null-Text Inversion Tune-A-Video Stable Diffusion DDIM inversion
Closest papers by meaning
- VideoDirector: Precise Video Editing via Text-to-Video Models (CVPR 2025 · similarity 0.90)
- CCEdit: Creative and Controllable Video Editing via Diffusion Models (CVPR 2024 · similarity 0.87)
- DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation (CVPR 2025 · similarity 0.86)
- FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing (ICLR 2024 · similarity 0.86)
- Re-Attentional Controllable Video Diffusion Editing (AAAI 2025 · similarity 0.86)
- VideoGrain: Modulating Space-Time Attention for Multi-Grained Video Editing (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.