Research map / Image generation / Editing and style transfer
Balancing Preservation and Modification: A Region and Semantic Aware Metric for Instruction-Based Image Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BPM, a region- and semantic-aware metric for instruction-based image editing that uses LLM parsing plus detection/segmentation to split images into edited and preserved regions, and also uses the localized masks as training-free guidance for editing models.
Problem
Automatic metrics for instruction-based image editing either need costly human evaluation or are adapted from other tasks; they evaluate the whole image holistically and fail to jointly account for the origin image, edited image, and instruction, yielding biased judgments of modification and preserv
Approach
Explicitly disentangle origin and edited images into editing-relevant and irrelevant regions (via LLM instruction parsing plus Grounding DINO and Grounded SAM), then score region size/position alignment with rule-based judges and score semantic compliance (directional CLIP similarity in the edited region) and preservation (L2 on the unedited region) separately, combined with weight 0.7/0.3.
Key results
- BPM reaches average human alignment 0.799 on local edits, versus 0.754 for GPT-4o and 0.702 for L2 (the best non-VLM baseline is L2 at 0.702).
- In the ground truth test BPM favors the ground-truth edit 87% of the time, versus 78% for GPT-4o and 13% for CLIPScore/CLIP-I combinations.
- BPM takes about 1.5 s per image versus 11.4 s for GPT-4o; mask-guided editing improves preservation scores for IP2P, FT_IP2P and MGIE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BPM instead of GPT-4o as the evaluation metric | image-editing metric evaluation · MagicBrush local edits | human alignment (average): 0.754 -> 0.799 | Helped |
| Use BPM instead of CLIPScore | image-editing metric evaluation · PIE-Bench global edits | human alignment (average): 0.817 -> 0.829 | Helped |
| Use BPM-overall instead of GPT-4o | Ground Truth test · Ground Truth test | GT preference proportion: 0.78 -> 0.87 | Helped |
| Add BPM localization mask guidance (ablation) | IP2P · mask-guided editing | preservation score: 0.792 -> 0.927 | Helped |
| Add BPM localization mask guidance (ablation) | IP2P · mask-guided editing | modification score: 0.318 -> 0.334 | Helped |
| Replace cropped-region CLIPScore with directional similarity (ablation) | BPM modification evaluation · Human alignment: IP2P vs FT_IP2P | alignment score: 0.635 -> 0.659 | Helped |
| Replace cropped-region CLIPScore with directional similarity (ablation) | BPM modification evaluation · Human alignment: DaLLE-2 vs IP2P | alignment score: 0.619 -> 0.670 | Helped |
| Replace cropped-region CLIPScore with directional similarity (ablation) | BPM modification evaluation · Human alignment: DALLE-2 vs FT_IP2P | alignment score: 0.479 -> 0.606 | Helped |
| Replace cropped-region CLIPScore with directional similarity (ablation) | BPM modification evaluation · Human alignment: DALLE-2 vs MGIE | alignment score: 0.536 -> 0.583 | Helped |
| Use region-aware S_preserve instead of L2 (ablation) | BPM preservation evaluation · Human alignment: IP2P vs FT_IP2P | alignment score: 0.816 -> 0.829 | Helped |
| Use directional S_modify instead of CLIPScore (ablation) | BPM modification evaluation · Human alignment: IP2P vs FT_IP2P | alignment score: 0.624 -> 0.659 | Helped |
| Set semantic weight alpha to 0.7 rather than 0.6 (ablation) | BPM · Human alignment (100-entry subset) | alignment score: 0.819 -> 0.822 | Helped |
Surprising findings
- Preservation metrics (L2, LPIPS, CLIP-I, DINOScore) favor excessively preserved images (noise-added) and CLIPScore favors excessively modified images in the ground truth test, showing holistic metrics are biased in opposite directions.
- Correlation with absolute per-model human scores is low for all metrics (average around 0.44 at best), so the authors switch to pairwise preference alignment.
Limitations
- The pipeline depends on detection and segmentation tools and the LLM parser, so errors in localization (97% accuracy on 100 samples) propagate into scores.
- Human alignment is evaluated on a relatively small set (960 annotations from 3 participants), and the absolute-score correlation is low.
- Rule-based thresholds for position and size judgments are hand-designed and limited to a fixed set of spatial relations.
Experimental setup
Datasets: MagicBrush PIE-Bench
Benchmarks: MagicBrush PIE-Bench
Metrics: alignment score with human preference winning rate in ground truth test inference speed
Baselines: L2 LPIPS CLIP-I DINOScore CLIPScore GPT-4o CLIPScore + CLIP-I
Builds on: CLIP Directional Similarity classifier-free guidance Grounding DINO Grounded SAM
Closest papers by meaning
- InsightEdit: Towards Better Instruction Following for Image Editing (CVPR 2025 · similarity 0.91)
- EdiVal-Agent: An Object-Centric Framework for Automated, Fine-Grained Evaluation of Multi-Turn Editing (ICLR 2026 · similarity 0.91)
- I2EBench: A Comprehensive Benchmark for Instruction-based Image Editing (NeurIPS 2024 · similarity 0.90)
- UltraEdit: Instruction-based Fine-Grained Image Editing at Scale (NeurIPS 2024 · similarity 0.89)
- Text-Vision Co-Instructed Image Editing (NeurIPS 2026 · similarity 0.89)
- Preserve or Modify? Context-Aware Evaluation for Balancing Preservation and Modification in Text-Guided Image Editing (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.