Research map / Video / Video generation
TokenFlow: Consistent Diffusion Features for Consistent Video Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TokenFlow, a training-free method that propagates edited Stable Diffusion self-attention tokens across video frames using nearest-neighbor correspondences from the original video, enabling temporally consistent text-driven video editing.
Problem
Text-driven video editing with image diffusion models suffers from temporal inconsistency across frames, since per-frame editing breaks feature consistency and extended self-attention only implicitly preserves motion.
Approach
Enforce consistency in diffusion feature space by sampling random keyframes for joint extended-attention editing, then explicitly propagating their edited self-attention tokens to all frames via nearest-neighbor correspondences computed on the original video's diffusion features, at every denoising step and layer.
Key results
- Achieves the highest CLIP score among compared methods while maintaining low warping error; Rerender-a-Video has lower warping error but users significantly preferred TokenFlow for temporal consistency.
- Per-frame sampling with TokenFlow reduces per-frame editing time by about 20% (sampling 187 s vs PnP 208 s on a 40-frame video, total 237 s including preprocessing).
- TokenFlow video reconstruction slightly improves over vanilla DDIM reconstruction in PSNR and LPIPS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| propagate edited diffusion features using original-video correspondences instead of relying on extended attention alone (ablation) | TokenFlow · video editing | temporal consistency | Helped |
| use fixed, evenly spaced keyframes instead of randomizing keyframe selection at each generation step (ablation) | TokenFlow · video editing | temporal consistency | Hurt |
| sample video frames rather than edit every frame independently | TokenFlow · 40-frame video | per-frame editing time: 20% | Helped |
Surprising findings
- Rerender-a-Video obtains the lowest warping error yet users still strongly preferred TokenFlow, indicating warping error misses long-range artifacts from optical flow.
Limitations
- Cannot handle edits requiring structural changes because it preserves the original video's motion and correspondences.
- Relies on the underlying image editing method preserving structure; failures produce meaningless correspondences and artifacts.
- Decoder of latent diffusion introduces high-frequency flickering; requires post-processing deflickering.
Experimental setup
Datasets: DAVIS Internet videos
Benchmarks: DAVIS Internet video evaluation set (61 text-video pairs)
Metrics: CLIP similarity warping error 2AFC user study PSNR LPIPS runtime
Baselines: Tune-A-Video Text2Video-Zero Rerender-a-Video Fate-Zero Gen-1 PnP-Diffusion Text2LIVE
Builds on: Plug-and-Play Diffusion Stable Diffusion DDIM inversion Tune-A-Video Rerender-a-Video
Closest papers by meaning
- COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing (NeurIPS 2024 · similarity 0.90)
- FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing (ICLR 2024 · similarity 0.89)
- MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers (CVPR 2024 · similarity 0.89)
- VideoTetris: Towards Compositional Text-to-Video Generation (NeurIPS 2024 · similarity 0.89)
- SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction (ICLR 2024 · similarity 0.89)
- TokenCompose: Text-to-Image Diffusion with Token-level Supervision (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.