Research map / Image generation / Editing and style transfer
Instruct-CLIP: Improving Instruction-Guided Image Editing with Automated Data Refinement Using Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Instruct-CLIP learns contrastive alignment between image-edit changes and instructions to refine the InstructPix2Pix dataset (120K+ samples) and fine-tunes IP2P with an I-CLIP-based latent loss.
Problem
Instruction-guided image editing datasets built with text-to-image models often have edit instructions that do not match the actual visual changes, degrading trained editing models.
Approach
A CLIP-style contrastive model embeds the visual difference between original and edited images alongside the instruction; a DeCap-style decoder regenerates instructions from the visual-change embedding to refine dataset captions, and a latent-diffusion version (LD-DINOv2 + I-CLIP) provides an auxiliary alignment loss during diffusion fine-tuning.
Key results
- On MagicBrush single-turn, Ours reaches CLIP-I 0.911 and DINO-I 0.803 versus MagicBrush 0.929/0.836, with CLIP-T 0.305 vs 0.307 (best among baselines in CLIP-T is ZONE/WYS at 0.301-0.313 multi-turn).
- User study with 104 participants: ours wins 54.16% vs MagicBrush and 29.67% vs IP2P (ties 16.08% and 58.47%).
- Ablation: refined data and guidance loss each improve DINO-I over IP2P (0.645 to 0.671 with data only, 0.782 with loss only, 0.803 full).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add refined instructions to the training data (ablation) | Instruct-CLIP · MagicBrush | CLIP-I: 0.862 -> 0.911 | Helped |
| add refined instructions to the training data (ablation) | Instruct-CLIP · MagicBrush | DINO-I: 0.671 -> 0.803 | Helped |
| add refined instructions to the training data (ablation) | Instruct-CLIP · MagicBrush | CLIP-T: 0.299 -> 0.305 | Helped |
| add the Instruct-CLIP guidance loss (ablation) | Instruct-CLIP · MagicBrush | CLIP-I: 0.903 -> 0.911 | Helped |
| add the Instruct-CLIP guidance loss (ablation) | Instruct-CLIP · MagicBrush | DINO-I: 0.782 -> 0.803 | Helped |
| add the Instruct-CLIP guidance loss (ablation) | Instruct-CLIP · MagicBrush | CLIP-T: 0.303 -> 0.305 | Helped |
| add refined instructions to the training data (ablation) | Instruct-CLIP · ZONE | CLIP-T: 0.295 -> 0.297 | Helped |
| add the Instruct-CLIP guidance loss (ablation) | Instruct-CLIP · ZONE | CLIP-T: 0.298 -> 0.297 | Hurt |
Surprising findings
- CLIP-T, CLIP-I and DINO-I do not always match visual quality of edits, according to the authors.
- Ours trails MagicBrush on CLIP-I and DINO-I in the main benchmark table despite being preferred in user study over MagicBrush.
Limitations
- Inherits artifacts of Prompt-to-Prompt generated training images such as color bleeding and incomplete object addition.
- Struggles to remove objects present in the original image.
- Standard metrics (CLIP-T, CLIP-I, DINO-I) do not reliably reflect visual quality of edits.
Experimental setup
Datasets: InstructPix2Pix dataset MagicBrush ZONE
Benchmarks: MagicBrush ZONE InstructBrush
Metrics: CLIP-T CLIP-I DINO-I user preference
Baselines: InstructPix2Pix MagicBrush HIVE InstructInpaint Watch Your Steps ZONE
Builds on: CLIP DeCap InstructPix2Pix DINOv2 Stable Diffusion
Closest papers by meaning
- InsightEdit: Towards Better Instruction Following for Image Editing (CVPR 2025 · similarity 0.90)
- InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists (ICLR 2024 · similarity 0.88)
- SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models (CVPR 2024 · similarity 0.88)
- ImgEdit: A Unified Image Editing Dataset and Benchmark (NeurIPS 2025 · similarity 0.88)
- Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformer (NeurIPS 2025 · similarity 0.88)
- OmniEdit: Building Image Editing Generalist Models Through Specialist Supervision (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.