Research map / Video / Video segmentation and tracking
A Video is Worth 256 Bases: Spatial-Temporal Expectation-Maximization Inversion for Zero-Shot Video Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STEM inversion, which uses EM-style clustering to compress a video into 256 shared bases that replace per-frame key/value context in self-attention, improving DDIM inversion for zero-shot video editing with TokenFlow and FateZero.
Problem
Zero-shot text-driven video editing with diffusion models relies on DDIM inversion, which is inaccurate with classifier-free guidance and gives limited temporal context (1-frame or 2-frame) for reconstruction and editing.
Approach
Treat the video's dense features as data for an EM-style procedure (K-means-like soft clustering) that yields a compact set of 256 global bases; every frame's inversion uses these fixed bases as keys and values in self-attention, giving a global, time-invariant context at lower cost than all-frame attention.
Key results
- STEM-TokenFlow reaches warp error 3.5e-3 versus 4.9e-3 for TokenFlow, and STEM-FateZero 4.3e-3 versus 7.2e-3 for FateZero, with comparable CLIP scores.
- In a user study over 12 videos per user, STEM-FateZero was preferred for edit fidelity 83% vs 17% for FateZero, and STEM-TokenFlow 75% vs 25% for TokenFlow.
- With K=256 bases, STEM inversion costs about 1.50 min versus 1.12 min for 2-frame DDIM on a 48-frame 640x360 video, and K=256 matches K=512 in ablations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace TokenFlow's DDIM inversion with STEM inversion | TokenFlow · Video editing benchmark | Warp error (1e-3): 4.9 -> 3.5 | Helped |
| Replace TokenFlow's DDIM inversion with STEM inversion | TokenFlow · Video editing benchmark | CLIP score: 0.31 -> 0.31 | No clear effect |
| Replace FateZero's DDIM inversion with STEM inversion | FateZero · Video editing benchmark | Warp error (1e-3): 7.2 -> 4.3 | Helped |
| Replace FateZero's DDIM inversion with STEM inversion | FateZero · Video editing benchmark | CLIP score: 0.29 -> 0.30 | Helped |
| Use STEM inversion instead of DDIM inversion | Video reconstruction · Video reconstruction | PSNR | Helped |
| Use K = 128 bases instead of K = 256 (ablation) | STEM inversion · Video editing | Editing ability | Hurt |
| Use K = 256 bases instead of K = 512 (ablation) | STEM inversion · Video reconstruction and editing | Performance | No clear effect |
| Use DDIM inversion with 2-frame context instead of 1-frame context | Video inversion · 48-frame video, 640 × 360 | Time (min): 0.80 -> 1.12 | Hurt |
| Use DDIM inversion with all-frame context instead of 1-frame context | Video inversion · 48-frame video, 640 × 360 | Time (min): 0.80 -> 5.04 | Hurt |
| Replace FateZero's DDIM inversion with STEM inversion | FateZero · User study | Edit fidelity selection: 17% -> 83% | Helped |
| Replace TokenFlow's DDIM inversion with STEM inversion | TokenFlow · User study | Temporal consistency selection: 33% -> 67% | Helped |
Surprising findings
- Using all-frame context in DDIM inversion costs about 5.04 min, while STEM with K=128/256 is much cheaper despite exploring global context.
Limitations
- Fixed 256 bases may be insufficient for very long videos; adaptive basis count is left to future work.
- EM iterations are serial, making STEM slower than 1-frame DDIM inversion.
- Evaluation uses a small set of DAVIS and internet videos and a user study of 12 videos per user.
Experimental setup
Datasets: DAVIS
Benchmarks: DAVIS
Metrics: PSNR SSIM CLIP score warp error user study preference
Baselines: Tune-A-Video Pix2Video Text2Video-Zero FateZero TokenFlow DDIM inversion
Builds on: DDIM inversion TokenFlow FateZero Stable Diffusion Null-text inversion
Closest papers by meaning
- VideoDirector: Precise Video Editing via Text-to-Video Models (CVPR 2025 · similarity 0.89)
- Ground-A-Video: Zero-shot Grounded Video Editing using Text-to-image Diffusion Models (ICLR 2024 · similarity 0.87)
- Semantic Image Inversion and Editing using Rectified Stochastic Differential Equations (ICLR 2025 · similarity 0.87)
- Tuning-Free Inversion-Enhanced Control for Consistent Image Editing (AAAI 2024 · similarity 0.86)
- FreeInv: Free Lunch for Improving DDIM Inversion (NeurIPS 2025 · similarity 0.86)
- PostEdit: Posterior Sampling for Efficient Zero-Shot Image Editing (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.