Research map / Video / Video generation
Slicedit: Zero-Shot Video Editing With Text-to-Image Diffusion Models Using Spatio-Temporal Slices
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Zero-shot text-based video editing that inflates a pretrained Stable Diffusion v2.1 denoiser with extended attention and applies it also to spatiotemporal (y-t) slices, combined with DDPM inversion and attention injection.
Problem
Zero-shot text-guided video editing with pretrained text-to-image diffusion models struggles with temporal consistency, especially under strong nonrigid motion, occlusions, and long videos, while preserving unedited regions.
Approach
Spatiotemporal y-t slices of natural videos look like natural images, so the same pretrained T2I denoiser can be applied to them (unconditionally) to enforce temporal smoothness; this is combined with extended-attention denoising of frames inside a DDPM-inversion editing pipeline with attention injection.
Key results
- Ours achieves Flow Err. 0.252 and LPIPS 0.159 versus TokenFlow's 0.4 and 0.35 and Pix2Video's 0.55 and 0.45, with CLIP-Text 0.329.
- Removing the y-t spatiotemporal slice denoiser raises Flow Err. from 0.252 to 0.279 and LPIPS from 0.159 to 0.207.
- In a 50-worker MTurk study, users preferred Slicedit over TokenFlow and Rerender A Video for preserving the original video's essence.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add y-t slice denoising (ablation) | Slicedit · 60 text-video pairs | Flow Err.: 0.279 -> 0.252 | Helped |
| add x-t slice denoising alongside y-t slices (ablation) | Slicedit · 60 text-video pairs | Flow Err.: 0.252 -> 0.248 | Helped |
| use text₁ instead of an empty slice-denoiser prompt (ablation) | Slicedit · 60 text-video pairs | Flow Err.: 0.252 -> 0.252 | No clear effect |
| use text₂ instead of an empty slice-denoiser prompt (ablation) | Slicedit · 60 text-video pairs | Flow Err.: 0.252 -> 0.252 | No clear effect |
| replace DDPM inversion and sampling with DDIM (ablation) | Slicedit · 60 text-video pairs | CLIP-Text: 0.329 -> 0.29 | Hurt |
| replace DDPM inversion and sampling with DDIM (ablation) | Slicedit · 60 text-video pairs | Flow Err.: 0.252 -> 0.55 | Hurt |
| add extended-attention injection from the source video (ablation) | Slicedit · 60 text-video pairs | Flow Err.: 0.452 -> 0.252 | Helped |
| use the source prompt rather than an empty prompt during inversion (ablation) | Slicedit · 60 text-video pairs | LPIPS: 0.182 -> 0.159 | Helped |
| use hyperparameters (8,14,85) rather than (8,10,85) (ablation) | Slicedit · 60 text-video pairs | CLIP-Text: 0.329 -> 0.332 | Helped |
| use hyperparameters (8,14,85) rather than (8,10,85) (ablation) | Slicedit · 60 text-video pairs | Flow Err.: 0.252 -> 0.275 | Hurt |
| use hyperparameters (8,14,85) rather than (8,10,85) (ablation) | Slicedit · 60 text-video pairs | LPIPS: 0.159 -> 0.22 | Hurt |
| apply Slicedit rather than TokenFlow | video editing on 60 text-video pairs · 60 text-video pairs | Flow Err.: 0.4 -> 0.252 | Helped |
Surprising findings
- The pretrained image denoiser predicts noise on spatiotemporal slices better than on video frames at low noise levels, and comparably at high noise levels.
- Replacing DDPM inversion with DDIM inversion makes the method fail to follow the text prompt (CLIP-Text 0.29).
Limitations
- Limited to structure-preserving edits; cannot change e.g. a dog into an elephant.
- Struggles with global editing such as converting natural video into paintings.
- Requires 64-frame segments; shorter videos are frame-interpolated with RIFE, and runtime is about 33.7 minutes per 64-frame video.
Experimental setup
Datasets: DAVIS LOVEU-TGVE Pexels videos (denoiser analysis)
Benchmarks: Slicedit 60 text-video pair dataset
Metrics: CLIP-Text CLIP-Consistency Flow Error LPIPS user preference
Baselines: Pix2Video Rerender A Video TokenFlow ControlVideo Tune A Video Flatten
Builds on: DDPM inversion Tune-A-Video Pix2Video FateZero TokenFlow RIFE
Closest papers by meaning
- COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing (NeurIPS 2024 · similarity 0.89)
- VideoDirector: Precise Video Editing via Text-to-Video Models (CVPR 2025 · similarity 0.88)
- MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers (CVPR 2024 · similarity 0.87)
- FADE: Frequency-Aware Diffusion Model Factorization for Video Editing (CVPR 2025 · similarity 0.87)
- FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing (ICLR 2024 · similarity 0.87)
- VideoGuide: Improving Video Diffusion Models without Training Through a Teacher's Guide (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.