Research map / Image generation / Editing and style transfer
HQ-Edit: A High-Quality Dataset for Instruction-based Image Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds HQ-Edit, a ~200K instruction-based image editing dataset generated via GPT-4 expansion and DALL-E 3 diptych generation with GPT-4V rewriting, warping and filtering, and fine-tunes InstructPix2Pix on it to reach state-of-the-art editing scores.
Problem
Instruction-based image editing is limited by a scarcity of high-quality image pairs with editing instructions; prior synthetic datasets (e.g., InstructPix2Pix) use outdated generators, yielding low resolution and poor edit-image alignment.
Approach
Generate input and output images jointly as side-by-side diptychs with DALL-E 3 from GPT-4-expanded prompts, then decompose, warp (DIFT-based semantic correspondence), filter, and enrich instructions with GPT-4V rewriting and inverse edits; also proposes GPT-4V-based Alignment and Coherence metrics.
Key results
- InstructPix2Pix fine-tuned on HQ-Edit reaches Alignment 47.01 and Coherence 86.16, versus 34.71 and 80.52 for vanilla InstructPix2Pix and 43.77 / 84.19 for MagicBrush.
- The Alignment metric shows a correlation of 0.3592 with human scores, versus -0.1446 for CLIP directional similarity, on 1,651 human-rated pairs.
- Diptych-based generation plus rewrite, inverse, warp and filter post-processing progressively raises Alignment from 16.83 (raw) to 47.01.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune InstructPix2Pix on HQ-Edit rather than the vanilla model | InstructPix2Pix · 293-sample test set | Alignment: 34.71 to 47.01 | Helped |
| Fine-tune InstructPix2Pix on HQ-Edit rather than the vanilla model | InstructPix2Pix · 293-sample test set | Coherence: 80.52 to 86.16 | Helped |
| Use HQ-Edit instead of InstructPix2Pix, HIVE, or MagicBrush data | Editing dataset quality · Dataset samples | Alignment: 68.29; 9.85; 80.61 -> 92.80 | Helped |
| Use HQ-Edit instead of InstructPix2Pix, HIVE, or MagicBrush data | Editing dataset quality · Dataset samples | Coherence: 83.35; 84.65; 65.42 -> 91.87 | Helped |
| Use raw decomposed DALL-E 3 images (ablation) | InstructPix2Pix fine-tuning · Post-processing ablation | Alignment: 34.71 -> 16.83 | Hurt |
| Use raw decomposed DALL-E 3 images (ablation) | InstructPix2Pix fine-tuning · Post-processing ablation | Coherence: 80.52 -> 85.74 | Helped |
| Add instruction rewriting to RAW data (ablation) | InstructPix2Pix fine-tuning · Post-processing ablation | Alignment: 16.83 -> 28.62 | Helped |
| Add instruction rewriting to RAW data (ablation) | InstructPix2Pix fine-tuning · Post-processing ablation | Coherence: 85.74 -> 86.68 | Helped |
| Add inverse-edit augmentation to RAW + Rewrite (ablation) | InstructPix2Pix fine-tuning · Post-processing ablation | Alignment: 28.62 -> 34.42 | Helped |
| Add inverse-edit augmentation to RAW + Rewrite (ablation) | InstructPix2Pix fine-tuning · Post-processing ablation | Coherence: 86.68 -> 87.53 | Helped |
| Add warping to RAW + Rewrite + Inverse (ablation) | InstructPix2Pix fine-tuning · Post-processing ablation | Alignment: 34.42 -> 43.41 | Helped |
| Add warping to RAW + Rewrite + Inverse (ablation) | InstructPix2Pix fine-tuning · Post-processing ablation | Coherence: 87.53 -> 87.56 | Helped |
Surprising findings
- Fine-tuning on raw DALL-E 3 images raises Alignment but lowers Coherence relative to the InstructPix2Pix baseline (Alignment 16.83 vs 34.71 in the ablation, Coherence 85.74 vs 80.52, i.e., Coherence actually improved in the raw row).
- CLIP directional similarity showed negative correlation with human evaluation scores.
Limitations
- Metrics rely on GPT-4V judgments and a single human evaluation on 1,651 pairs.
- Filtering step is applied only for InstructPix2Pix fine-tuning, and warping can cause image distortion.
- Dataset is generated entirely by proprietary models (GPT-4, GPT-4V, DALL-E 3), limiting reproducibility.
Experimental setup
Datasets: Emu Edit test set (90 samples) online-collected seed samples (203) MagicBrush InstructPix2Pix dataset
Benchmarks: HQ-Edit test set (293 seed samples)
Metrics: Alignment (GPT-4V score 0-100) Coherence (GPT-4V score 0-100) CLIP directional similarity
Baselines: Imagic DiffEdit PromptInverse HIVE MagicBrush InstructPix2Pix
Builds on: InstructPix2Pix Self-instruct DIFT YOLOv8
Closest papers by meaning
- ImgEdit: A Unified Image Editing Dataset and Benchmark (NeurIPS 2025 · similarity 0.89)
- InsightEdit: Towards Better Instruction Following for Image Editing (CVPR 2025 · similarity 0.89)
- Multi-Reward as Condition for Instruction-based Image Editing (ICLR 2025 · similarity 0.88)
- AnyEdit: Mastering Unified High-Quality Image Editing for Any Idea (CVPR 2025 · similarity 0.87)
- Señorita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialists (NeurIPS 2025 · similarity 0.87)
- OmniEdit: Building Image Editing Generalist Models Through Specialist Supervision (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.