Research map / Video / Video generation
VidToMe: Video Token Merging for Zero-Shot Video Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VidToMe, a training-free zero-shot video editing method that merges self-attention tokens across frames (local per-chunk and global cross-chunk) in a pre-trained Stable Diffusion to improve temporal consistency and reduce attention memory.
Problem
Zero-shot text-driven video editing with image diffusion models struggles to keep strict temporal consistency and suffers quadratic memory growth when extending self-attention across frames.
Approach
Merge temporally redundant self-attention tokens across frames via bipartite soft matching (ToMe-style), with local merging within chunks and global token merging across chunks, then unmerge to restore per-frame tokens; it is a parameter-free drop-in that works with existing image editing controls.
Key results
- Ours + PnP achieves interpolation error 0.111 and PSNR 25.0 versus 0.139 and 22.5 for Pix2Video on temporal consistency.
- Ours + ControlNet-Depth receives a 54.4% user preference rate versus 17.0% for the next-best baseline Pix2Video.
- VidToMe keeps peak GPU memory under 7 GB on an A100, while extended self-attention on four frames needs roughly 16x memory of one frame.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace per-frame self-attention with VidToMe token merging (ablation) | Video editing · Video editing | Interpolation Error: 0.253 -> 0.105 | Helped |
| replace extended multi-frame self-attention with VidToMe token merging (ablation) | Video editing · Video editing | Interpolation Error: 0.140 -> 0.105 | Helped |
| replace token averaging with destination-token replacement when merging (ablation) | VidToMe · Video editing | edit fidelity | Helped |
| remove global token merging (ablation) | VidToMe · Video editing | long-term temporal consistency | Hurt |
| merge tokens across all frames with vanilla ToMe after concatenation (ablation) | VidToMe local token merging · Video editing | frame quality | Hurt |
| apply ToMe within each frame before concatenating frame tokens (ablation) | VidToMe local token merging · Video editing | frame quality | Hurt |
Surprising findings
- Averaging merged tokens (original ToMe) reduces diversity and randomness; replacing merged token values with destination tokens works better.
Limitations
- Editing quality depends on the selected image editing technique; a per-frame failure propagates to the whole video.
- Similarity-based matching can incorrectly merge objects with similar features, mixing them in outputs.
Experimental setup
Datasets: DAVIS
Benchmarks: DAVIS
Metrics: Interpolation Error PSNR Warp Error Frame CLIP Score Directional CLIP Score Text CLIP Score user preference rate
Baselines: Text2Video-Zero Tune-A-Video vid2vid-zero Pix2Video
Builds on: Token Merging (ToMe) Plug-and-Play ControlNet DDIM inversion Stable Diffusion
Closest papers by meaning
- Video Token Merging for Long Video Understanding (NeurIPS 2024 · similarity 0.88)
- COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing (NeurIPS 2024 · similarity 0.88)
- TokenFlow: Consistent Diffusion Features for Consistent Video Editing (ICLR 2024 · similarity 0.87)
- FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing (ICLR 2024 · similarity 0.87)
- vid-TLDR: Training Free Token Merging for Light-weight Video Transformer (CVPR 2024 · similarity 0.87)
- HoliTom: Holistic Token Merging for Fast Video Large Language Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.