Research map / Image generation / Editing and style transfer
Towards Scalable Human-aligned Benchmark for Text-guided Image Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HATIE, a large-scale, fully automated benchmark for text-guided image editing built on GQA images, with a multi-criterion metric whose weights are fitted to user study judgments.
Problem
Text-guided image editing lacks a widely accepted, reproducible, scalable evaluation method; existing evaluations rely on small sets or manual user studies, and conventional metrics (CLIP, detection rate) align poorly with human judgment.
Approach
Builds a large editing query set from GQA scene-graph annotations filtered to COCO classes, and scores edits on five criteria (image quality, object/background fidelity, object/background consistency) using segmentation, CLIP, LPIPS, DINO, and FID, with weights fitted via grid search to user-study winning rates.
Key results
- HATIE metric winning-rate correlations with user study on test set reach Pearson r above 0.82 across all criteria (e.g., 0.9276 for object consistency).
- Conventional metrics align much worse with human judgment; e.g., CLIP alignment single-metric Pearson r = -0.34 on overall quality.
- On HATIE total score, P2P (0.6858) and MagicBrush (0.7329) are the top description-based and instruction-based models respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Aggregate multiple evaluation metrics into criteria and fit their weights to human judgments | HATIE · User study test set | Pearson correlation (Object Fidelity winning rates): 0.8208 vs -0.3410 | Helped |
| Aggregate multiple evaluation metrics into criteria and fit their weights to human judgments | HATIE · User study test set | Pearson correlation (Background Fidelity winning rates): 0.8971 | Helped |
| Aggregate multiple evaluation metrics into criteria and fit their weights to human judgments | HATIE · User study test set | Pearson correlation (Object Consistency winning rates): 0.9276 | Helped |
| Aggregate multiple evaluation metrics into criteria and fit their weights to human judgments | HATIE · User study test set | Pearson correlation (Background Consistency winning rates): 0.8857 | Helped |
| Aggregate multiple evaluation metrics into a Total Score and fit weights to human judgments | HATIE · User study test set | Pearson correlation (overall winning rates): 0.7143 vs 0.5058 | Helped |
Surprising findings
- CLIP alignment, the metric used in many prior editing papers, shows negative correlation with human overall-quality judgments in the user study.
Limitations
- Object removal for description-based models and movement/rotation tasks are excluded due to difficulty generating captions and reference points.
- Certain attribute types are excluded due to localization difficulty.
- Evaluation relies on a fixed set of 76 COCO object classes and automatic segmentation/detection models.
Experimental setup
Datasets: GQA COCO
Benchmarks: HATIE
Metrics: FID-based image quality CLIP alignment detection confidence size fidelity LPIPS DINO similarity L2 distance position score winning rate correlation (Pearson, Spearman, Kendall)
Baselines: Imagic Prompt-to-Prompt DiffEdit DiffuseIT FreeDiff MasaCtrl InstructPix2Pix InstructDiffusion MagicBrush TEdBench EditBench EditVal
Builds on: GQA COCO FID CLIP LPIPS
Closest papers by meaning
- I2EBench: A Comprehensive Benchmark for Instruction-based Image Editing (NeurIPS 2024 · similarity 0.88)
- ImgEdit: A Unified Image Editing Dataset and Benchmark (NeurIPS 2025 · similarity 0.87)
- VE-Bench: Subjective-Aligned Benchmark Suite for Text-Driven Video Editing Quality Assessment (AAAI 2025 · similarity 0.87)
- Balancing Preservation and Modification: A Region and Semantic Aware Metric for Instruction-Based Image Editing (ICML 2025 · similarity 0.86)
- EditBoard: Towards a Comprehensive Evaluation Benchmark for Text-Based Video Editing Models (AAAI 2025 · similarity 0.86)
- ViTeX-Bench: Benchmarking High-Fidelity Video Scene Text Editing (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.