Research map / Image generation / Editing and style transfer
Text-Vision Co-Instructed Image Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes text-vision co-instructed image editing, combining textual instructions with sparse drag/point visual prompts via a plug-and-play content-aware spatial controller on top of Qwen-Image-Edit and FLUX.1 Kontext, trained on a 23K video-derived dataset and evaluated on a new 120-sample TV-Edit-Bench.
Problem
Text-only instruction editing lacks precise spatial control, while drag/point-based visual prompt editing is ambiguous about semantic intent; existing methods do not jointly exploit both modalities.
Approach
Jointly condition a pretrained instruction-based editing backbone on text and sparse point trajectories through a lightweight MM-DiT-based content-aware controller (image, text, and point maps fused early) with time-modulated injection, trained with a Z0-prediction loss biased toward high-noise timesteps.
Key results
- TV-Edit-Qwen reduces MD_d to 0.0462 on TV-Edit-Bench, a 28.7% improvement over the best drag-based method (GoodDrag 0.0648).
- TV-Edit-Qwen raises PF from 0.86 (Qwen-Image-Edit) to 0.93, surpassing NanoBanana Pro (0.89).
- On DragBench, TV-Edit-Qwen achieves MD 17.31 vs. 23.77 for the best prior baseline (DragLoRA), with IF 0.86.
Surprising findings
- Drag-based methods with strong geometric accuracy get low PF scores, showing geometry alone does not capture semantic intent.
- Instruction-only editors like NanoBanana Pro achieve high semantic scores but MD_d above 0.10, meaning poor spatial control.
Limitations
- Built on large editing foundation models, so inference speed prevents real-time interactive editing.
- Limited to 2D operations; struggles with complex 3D manipulations such as out-of-plane rotations.
- TV-Edit-Bench is small (120 samples) and mixes generated pairs from Wan2.2 and NanoBanana Pro.
Experimental setup
Datasets: TV-Edit-23K real video datasets (unspecified) DragBench
Benchmarks: TV-Edit-Bench DragBench
Metrics: DS_global DS_local LPIPS MD_s MD_d PF CP MD IF
Baselines: DragDiffusion GoodDrag DragLoRA CLIP-Drag GeoDrag LightningDrag FLUX-Kontext Qwen-Image-Edit LongCat-Image-Edit MotionEdit NanoBananaPro
Builds on: ControlNet Rectified Flow DragDiffusion GoodDrag Qwen-Image-Edit
Closest papers by meaning
- InsightEdit: Towards Better Instruction Following for Image Editing (CVPR 2025 · similarity 0.90)
- VicEdit: Learning to Edit Videos from Visual In-Context Examples (NeurIPS 2026 · similarity 0.90)
- Balancing Preservation and Modification: A Region and Semantic Aware Metric for Instruction-Based Image Editing (ICML 2025 · similarity 0.89)
- FireEdit: Fine-grained Instruction-based Image Editing via Region-aware Vision Language Model (CVPR 2025 · similarity 0.89)
- Guiding Instruction-based Image Editing via Multimodal Large Language Models (ICLR 2024 · similarity 0.89)
- Reasoning to Edit: Hypothetical Instruction-Based Image Editing with Visual Reasoning (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.