Research map / Image generation / Editing and style transfer
Focus on Your Instruction: Fine-grained and Multi-instruction Image Editing by Attention Modulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free FoI method uses InstructPix2Pix's implicit cross-attention grounding to extract per-instruction masks, then modulates cross-attention and uses mask-guided disentangled sampling to make instruction-based editing precise and multi-instruction capable.
Problem
Instruction-based diffusion editors such as InstructPix2Pix over-edit unintended regions and struggle to execute multiple instructions at once without interference.
Approach
Exploit IP2P's implicit grounding visible in early-step cross-attention maps to extract a mask per sub-instruction, then confine each instruction's cross-attention to its mask (substituting null-instruction attention outside) and restrict classifier-free guidance of the instruction to the union mask during most sampling steps, all without training or test-time optimization.
Key results
- FoI achieves CLIP-I 0.9402 and DINO-I 0.9277 on single-instruction editing, vs. 0.9178 and 0.8702 for MagicBrush.
- On multi-instruction editing FoI reaches PickScore 0.5077 vs. 0.1807 CLIP-D/0.1591 PickScore for the best baseline MagicBrush.
- In a 60-participant human study, over 80% preferred FoI for multi-instruction instruction alignment (80.83%) and image alignment (81.33%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply FoI attention modulation and mask-guided disentangle sampling | InstructPix2Pix · Single-instruction | CLIP image similarity: 0.9178 -> 0.9402 | Helped |
| Apply FoI attention modulation and mask-guided disentangle sampling | InstructPix2Pix · Single-instruction | PickScore: 0.1780 -> 0.3901 | Helped |
| Apply FoI attention modulation and mask-guided disentangle sampling | InstructPix2Pix · Multi-instruction | CLIP image similarity: 0.8609 -> 0.9255 | Helped |
| Apply FoI attention modulation and mask-guided disentangle sampling | InstructPix2Pix · Multi-instruction | PickScore: 0.1591 -> 0.5077 | Helped |
| Apply FoI rather than MagicBrush | InstructPix2Pix · Single-instruction | CLIP direction similarity: 0.1934 -> 0.1699 | Hurt |
| Apply FoI rather than MagicBrush | InstructPix2Pix · Multi-instruction | CLIP direction similarity: 0.1807 -> 0.1685 | Hurt |
| Extract masks only at the first denoising step rather than at later steps (ablation) | FoI · Image editing | Editing quality | Helped |
| Apply cross-condition attention modulation throughout denoising rather than stopping early (ablation) | FoI · Multi-instruction image editing | Editing quality | Helped |
| Use disentangle sampling for the initial 75% of steps, then standard IP2P sampling (ablation) | FoI · Image editing | Editing quality | Helped |
| Use mask-guided disentangle sampling (ablation) | FoI · Image editing with broad adjectives | Editing precision | Helped |
| Increase text guidance sT while holding sI at 1.5 (ablation) | FoI · Image editing | Robustness to guidance selection | Helped |
Surprising findings
- CLIP direction similarity is lower for FoI than for over-editing baselines MagicBrush and InsDiff, which the authors interpret as evidence of less over-editing rather than weaker editing.
Limitations
- Relies on IP2P's implicit grounding, so quality is bounded by the base model's attention maps.
- Mask extraction uses a random threshold in [0.4, 0.7] and fixed hyperparameters, with disentangled sampling applied only to the first 75% of steps.
- Evaluated on a relatively small multi-instruction set of 100 images and a 20-image-per-category human study.
Experimental setup
Datasets: IP2P dataset (5,000 filtered localized edit images) 100 real images with GPT-4V-generated multi-instructions
Benchmarks: Custom single-instruction set (5,000 IP2P-derived images) Custom multi-instruction set (100 real images)
Metrics: CLIP image similarity DINOv2 image similarity CLIP direction similarity PickScore human preference
Baselines: DiffEdit NTI+P2P InstructPix2Pix MagicBrush InstructDiffusion
Builds on: InstructPix2Pix Prompt-to-Prompt Null-Text Inversion DiffEdit
Closest papers by meaning
- InsightEdit: Towards Better Instruction Following for Image Editing (CVPR 2025 · similarity 0.88)
- SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models (CVPR 2024 · similarity 0.86)
- Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformer (NeurIPS 2025 · similarity 0.86)
- Guiding Instruction-based Image Editing via Multimodal Large Language Models (ICLR 2024 · similarity 0.86)
- Instruct-CLIP: Improving Instruction-Guided Image Editing with Automated Data Refinement Using Contrastive Learning (CVPR 2025 · similarity 0.86)
- Balancing Preservation and Modification: A Region and Semantic Aware Metric for Instruction-Based Image Editing (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.