Research map / Image generation / Editing and style transfer
Pathways on the Image Manifold: Image Editing via Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frame2Frame reformulates text-based image editing as image-to-video generation: a VLM writes a temporal editing caption, CogVideoX-I2V generates a video from the source image, and a VLM (GPT-4o) selects the frame that realizes the edit.
Problem
Text-guided image editing struggles to follow complex edit instructions accurately while preserving key elements of the source image; single-shot diffusion editing often compromises fidelity.
Approach
Treat image editing as a temporal process: use a pretrained image-to-video model to generate a smooth trajectory on the image manifold from the source to the edit, guided by a VLM-written temporal caption, and pick the earliest frame that achieves the edit.
Key results
- On TEdBench, F2F achieves LPIPS 0.22, CLIP-I 0.89 and CLIP 0.63, matching or beating the best baselines (FlowEdit/LEDITS++) across metrics.
- On PosEdit, F2F reaches LPIPS 0.14 and CLIP-I 0.82 to the source versus 0.26/0.65 for LEDITS++, and CLIP 0.64.
- In a 59-participant human survey, F2F was preferred for edit accuracy 54.1% overall vs 45.9% for LEDITS++, and for edit quality 65.6% vs 34.4%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Frame2Frame video-based editing instead of LEDITS++ | Image editing · TEdBench | CLIP: 0.63 -> 0.63 | No clear effect |
| Use Frame2Frame video-based editing instead of FlowEdit | Image editing · TEdBench | LPIPS: 0.22 -> 0.22 | No clear effect |
| Use Frame2Frame video-based editing instead of SDEdit | Image editing · PosEdit | Target CLIP-I: 0.64 -> 0.84 | Helped |
| Use Frame2Frame video-based editing instead of LEDITS++ | Image editing · PosEdit | Source LPIPS: 0.26 -> 0.14 | Helped |
| Use Frame2Frame video-based editing instead of LEDITS++ | Image editing · TEdBench human evaluation | Edit accuracy preference: 34.4% -> 65.6% | Helped |
| Use Frame2Frame video-based editing instead of LEDITS++ | Image editing · TEdBench human evaluation | Edit quality preference: 45.9% -> 54.1% | Helped |
Surprising findings
- Video models produce plausible imaginative edits, such as filling a vase with green water, despite being trained on real-world transformations.
- Human survey gap in source preservation appeared larger than the LPIPS numbers suggested.
Limitations
- Natural camera motion can introduce unintended perspective shifts in edits.
- Video models trained mostly on real-world transformations struggle with edits far from training distribution.
- Computationally heavy: video generation is slower than image editing methods.
Experimental setup
Datasets: TEdBench UTD-MHAD
Benchmarks: TEdBench PosEdit
Metrics: LPIPS CLIP-I CLIP score human preference
Baselines: SDEdit Pix2Pix-Zero Imagic LEDITS++ FlowEdit
Builds on: CogVideoX SDEdit Imagic LEDITS++ Pix2Pix-Zero FlowEdit
Closest papers by meaning
- VINCIE-NExT: Unlocking Video Editing from Images via In-Context Modeling (NeurIPS 2026 · similarity 0.89)
- EditAR: Unified Conditional Generation with Autoregressive Models (CVPR 2025 · similarity 0.88)
- Transforming Image Editors into Video Editors (NeurIPS 2026 · similarity 0.88)
- FADE: Frequency-Aware Diffusion Model Factorization for Video Editing (CVPR 2025 · similarity 0.88)
- TokenFlow: Consistent Diffusion Features for Consistent Video Editing (ICLR 2024 · similarity 0.88)
- PAIR Diffusion: A Comprehensive Multimodal Object-Level Image Editor (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.