Research map / Video / Video generation
Visual Prompting for One-shot Controllable Video Editing without Inversion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reframes one-shot controllable video editing as visual prompting with an image inpainting diffusion model, avoiding DDIM inversion, and adds consistency-style content sampling (CCS) and SVGD-based temporal sampling (TCS).
Problem
One-shot controllable video editing propagates user edits on the first frame to all frames while keeping content consistency; prior DDIM-inversion-based methods accumulate reconstruction errors that hurt consistency, and video-diffusion-based temporal priors are slow and of limited quality.
Approach
Treats the first source/edited frame pair as a visual prompt example and each source frame as a query, arranged in a 2x2 grid for a pre-trained Stable Diffusion inpainting model, so no DDIM inversion is needed; CCS modifies the sampler to multi-step consistency sampling anchored on the source frame, and TCS applies SVGD to align edited frames with source-frame temporal distribution.
Key results
- Ours achieves CLIP-TC 97.1 (x10^-2) and TIFA 69.1, the best among compared methods, versus Videoshop's 64.4 TIFA and 95.2 CLIP-TC.
- Ours runs in 19 s per video versus 32 s for Videoshop and 149 s for AnyV2V.
- Ablations show removing CCS lowers source-faithfulness metrics (e.g., CLIP-src 81.3 vs 91.0 w/o CCS in the table) and removing TCS drops CLIP-TC from 97.1 to 89.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use visual prompting with CCS and TCS instead of the compared OCVE approach | Ours · Generated MagicBrush video dataset | CLIPtar (×10−2): 88.8 -> 90.1 | Helped |
| Use visual prompting with CCS and TCS instead of the compared OCVE approach | Ours · Generated MagicBrush video dataset | time (s) (×100): 32 -> 19 | Helped |
| Add CCS (ablation) | Ours · Generated MagicBrush video dataset | CLIPsrc (×10−2): 81.3 -> 93.2 | Helped |
| Add CCS (ablation) | Ours · Generated MagicBrush video dataset | FVD (×102): 25.1 -> 15.2 | Helped |
| Add TCS (ablation) | Ours · Generated MagicBrush video dataset | CLIPTC (×10−2): 89.8 -> 97.1 | Helped |
Surprising findings
- Ours w/o TCS has higher edit-fidelity scores than full model on some metrics (e.g., CLIP-tgt 89.8 vs 90.1 is close, and TIFA 68.3), while the full model gives the best temporal consistency.
Limitations
- Relies on a CLIP-space difference vector as the editing prompt, which may not capture all edits.
- Evaluated only on a generated video dataset derived from MagicBrush, not real videos.
Experimental setup
Datasets: MagicBrush-derived generated video dataset (Videoshop protocol, 10388 tuples)
Benchmarks: Videoshop generated video dataset (MagicBrush-derived)
Metrics: CLIP-tgt CLIP+tgt TIFA CLIP-src CLIP+src Flow score FVD SSIM CLIP-TC runtime
Baselines: Videoshop AnyV2V BDIA Pix2Video Fatezero Spacetime RAVE
Builds on: Videoshop consistency models Stein Variational Gradient Descent DDIM inversion Stable Diffusion Inpainting
Closest papers by meaning
- VideoDirector: Precise Video Editing via Text-to-Video Models (CVPR 2025 · similarity 0.86)
- Re-Attentional Controllable Video Diffusion Editing (AAAI 2025 · similarity 0.86)
- Tuning-Free Inversion-Enhanced Control for Consistent Image Editing (AAAI 2024 · similarity 0.86)
- Align-A-Video: Deterministic Reward Tuning of Image Diffusion Models for Consistent Video Editing (CVPR 2025 · similarity 0.86)
- Señorita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialists (NeurIPS 2025 · similarity 0.85)
- COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.