Research map / Video / Video generation
COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COVE, a training-free video editing method that uses sliding-window diffusion-feature correspondence (top-K tokens per frame) to guide self-attention during DDIM inversion and denoising of a Stable Diffusion 2.1 model, with temporal token merging for efficiency.
Problem
Zero-shot text-driven video editing with pre-trained text-to-image diffusion models struggles to maintain temporal consistency (flickering, blurring), and prior correspondence methods rely on optical flow yielding only one-to-one matches.
Approach
Extract diffusion features per frame, compute one-to-many token correspondences across frames via cosine similarity restricted to a sliding window in the next frame, and perform self-attention only among corresponding (merged) tokens during inversion and denoising, replacing correspondence-agnostic spatial-temporal attention.
Key results
- COVE reaches Subject Consistency 0.9731 and Motion Smoothness 0.9892, above all five baselines (best baseline FateZero 0.9622 / TokenFlow 0.9803).
- In a 45-participant user study COVE is preferred 44.5% of the time versus 7.4-14.8% for baselines.
- Correspondence PCK of 0.92 for diffusion features vs 0.87 for optical flow; sliding-window with l=9 reduces correspondence cost from 241e9 to 4.03e9 operations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply COVE correspondence-guided video editing | Stable Diffusion 2.1 · 115 text-video pairs | Subject Consistency: 0.9622 vs 0.9731 | Helped |
| apply COVE correspondence-guided video editing | Stable Diffusion 2.1 · 115 text-video pairs | Motion Smoothness: 0.9803 vs 0.9892 | Helped |
| apply COVE correspondence-guided video editing | Stable Diffusion 2.1 · 115 text-video pairs | Aesthetic Quality: 0.6904 vs 0.7122 | Helped |
| apply COVE correspondence-guided video editing | Stable Diffusion 2.1 · 115 text-video pairs | Imaging Quality: 0.7355 vs 0.7441 | Helped |
| apply COVE correspondence-guided video editing | Stable Diffusion 2.1 · 115 text-video pairs | User Study: 14.8% vs 44.5% | Helped |
| add correspondence-guided attention (ablation) | COVE · 115 text-video pairs | Subject Consistency: 0.9431 -> 0.9731 | Helped |
| add correspondence-guided attention (ablation) | COVE · 115 text-video pairs | Motion Smoothness: 0.9049 -> 0.9892 | Helped |
| increase selected correspondence tokens from K=1 to K=3 (ablation) | COVE · 115 text-video pairs | Subject Consistency: 0.9637 -> 0.9731 | Helped |
| increase selected correspondence tokens from K=1 to K=3 (ablation) | COVE · 115 text-video pairs | Imaging Quality: 0.7148 -> 0.7441 | Helped |
| increase selected correspondence tokens from K=3 to K=5 (ablation) | COVE · 115 text-video pairs | Subject Consistency: 0.9731 -> 0.9745 | No clear effect |
| increase selected correspondence tokens from K=3 to K=5 (ablation) | COVE · 115 text-video pairs | Aesthetic Quality: 0.7122 -> 0.7167 | Helped |
| add temporal-dimensional token merging (ablation) | COVE · 10-frame source video | Editing time: 2.7 min -> 2.4 min | Helped |
Surprising findings
- Correspondence-guided attention raises memory from 9 GB to 14 GB and time from 2.2 to 2.7 min; token merging recovers to 11 GB and 2.4 min.
Limitations
- Correspondence-guided attention is not memory- or time-efficient enough, adding GPU memory and time overhead (xFormers not used there).
- Using self-attention to exploit correspondence may not be the most effective choice; alternatives are unexplored.
- Evaluation uses a small set of 23 videos and 115 text-video pairs, with no error bars reported.
Experimental setup
Datasets: 23 social media and public videos (3 with 10 frames, 15 with 20 frames, 5 with 32 frames)
Benchmarks: VBench metrics on 115 text-video pairs
Metrics: Subject Consistency Motion Smoothness Aesthetic Quality Imaging Quality User Study preference PCK (correspondence accuracy)
Baselines: FateZero TokenFlow FLATTEN FRESCO RAVE
Builds on: DDIM inversion latent diffusion Tune-A-Video spatial-temporal attention ToMe token merging
Closest papers by meaning
- TokenFlow: Consistent Diffusion Features for Consistent Video Editing (ICLR 2024 · similarity 0.90)
- VideoGuide: Improving Video Diffusion Models without Training Through a Teacher's Guide (CVPR 2025 · similarity 0.90)
- FADE: Frequency-Aware Diffusion Model Factorization for Video Editing (CVPR 2025 · similarity 0.89)
- MeDM: Mediating Image Diffusion Models for Video-to-Video Translation with Temporal Correspondence Guidance (AAAI 2024 · similarity 0.89)
- Slicedit: Zero-Shot Video Editing With Text-to-Image Diffusion Models Using Spatio-Temporal Slices (ICML 2024 · similarity 0.89)
- Emergent Temporal Correspondences from Video Diffusion Transformers (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.