Research map / Image generation / Editing and style transfer
ZONE: Zero-Shot Instruction-Guided Local Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ZONE, a zero-shot method that performs instruction-guided local image editing by locating edit regions via InstructPix2Pix cross-attention, refining masks with SAM and a Region-IoU scheme, and blending with an FFT-based edge smoother.
Problem
Text-guided image editing either requires carefully crafted prompts or masks, or instruction-guided models over-edit non-edited regions, leaving visible artifacts; local editing needs to be intuitive and preserve the rest of the image.
Approach
Instruction-guided diffusion (IP2P) cross-attention maps are edit-aware rather than token-aware, so the edited region can be located from the instruction alone; the region is then refined with SAM via a Region-IoU selection and composited back using an FFT low-pass difference edge smoother.
Key results
- ZONE achieves L1 0.0146 and LPIPS 0.0441 versus 0.0945 and 0.2816 for InstructPix2Pix, and the best CLIP-I of 0.9688.
- In human evaluation, ZONE reaches a 69.4% success rate and 26.6% user preference rate, the highest among methods (InstructPix2Pix 59.8% SR).
- Text2LIVE gets the highest CLIP-T (0.3062 for Text2LIVE vs 0.2969 for ZONE, ZONE second-best by CLIP-T among instruction-based pairs is not top), which the authors attribute to direct CLIP supervision.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ZONE for local editing instead of the strongest baseline | ZONE · Local editing test set | L1: 0.0426 -> 0.0146 | Helped |
| Use ZONE for local editing instead of the strongest baseline | ZONE · Local editing test set | L2: 0.0075 -> 0.0061 | Helped |
| Use ZONE for local editing instead of the strongest baseline | ZONE · Local editing test set | LPIPS: 0.1695 -> 0.0441 | Helped |
| Use ZONE for local editing instead of the strongest baseline | ZONE · Local editing test set | CLIP-I: 0.9089 -> 0.9688 | Helped |
| Use ZONE for local editing instead of Text2LIVE | ZONE · Local editing test set | CLIP-T: 0.3062 -> 0.2969 | Hurt |
| Use ZONE instead of the strongest baseline | ZONE · Human evaluation | SR: 59.8 ± 3.1 -> 69.4 ± 3.5 | Helped |
| Use ZONE instead of the strongest baseline | ZONE · Human evaluation | UPR: 18.9 -> 26.6 | Helped |
| Add the FFT-based edge smoother when compositing the edited layer (ablation) | ZONE · Local image editing | edge artifacts | Helped |
Surprising findings
- Text2LIVE scores highest on CLIP-T yet the authors judge its qualitative results do not support this metric advantage.
Limitations
- Evaluated on a small 100-image in-house test set.
- Relies on the edited region being identified via IP2P attention and a fixed threshold T=128 and beta hyperparameters.
- Edge smoother and mask refinement depend on SAM and IP2P quality, so over-edit issues propagate to the mask.
Experimental setup
Datasets: 100 annotated evaluation images (60 real, 40 synthetic)
Benchmarks: in-house 100-image local editing test set
Metrics: L1 L2 LPIPS CLIP-I CLIP-T success rate user preference rate
Baselines: DiffEdit Text2LIVE Pix2Pix-Zero InstructPix2Pix MagicBrush
Builds on: InstructPix2Pix Segment Anything Text2LIVE Prompt-to-Prompt
Closest papers by meaning
- Text-Driven Image Editing via Learnable Regions (CVPR 2024 · similarity 0.87)
- Text-Vision Co-Instructed Image Editing (NeurIPS 2026 · similarity 0.87)
- Prompt Augmentation for Self-supervised Text-guided Image Manipulation (CVPR 2024 · similarity 0.86)
- PixPerfect: Seamless Latent Diffusion Local Editing with Discriminative Pixel-Space Refinement (NeurIPS 2025 · similarity 0.86)
- PIXELS: Progressive Image Xemplar-based Editing with Latent Surgery (AAAI 2025 · similarity 0.86)
- Balancing Preservation and Modification: A Region and Semantic Aware Metric for Instruction-Based Image Editing (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.