Research map / Vision-language and multimodal / Human and embodied modeling
Preserve or Modify? Context-Aware Evaluation for Balancing Preservation and Modification in Text-Guided Image Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AugCLIP, a context-aware metric for text-guided image editing that uses MLLM-extracted source/target attributes and a CLIP-space separating hyperplane to derive an ideal edited image embedding, balancing preservation and modification.
Problem
Existing metrics for text-guided image editing are context-blind: preservation metrics ignore the target text, modification metrics ignore the source image, and directional CLIP similarity is biased toward modification and attends to irrelevant regions.
Approach
Estimate the CLIP embedding of an ideally edited image as the minimum modification of the source embedding that a hyperplane (separating MLLM-extracted source vs. target attributes, weighted by importance and collision) classifies as target; score edits by cosine similarity to this ideal.
Key results
- AugCLIP reaches 0.883 s2AFC on CelebA, 0.831 on EditVal, and 0.857 on DreamBooth, the best among all compared metrics (CLIPdir: 0.673, 0.697, 0.357).
- On the ground-truth selection test, AugCLIP gets AccBoth 0.570 on TEdBench and 0.889 on MagicBrush, versus CLIPdir at 0.350 and 0.601.
- CLIPdir fails on 63% of TEdBench cases and 35.1% of MagicBrush cases by preferring excessively modified images over ground truth.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AugCLIP instead of CLIP-I | text-guided image editing metric · CelebA | s2AFC: 0.848 -> 0.883 | Helped |
| use AugCLIP instead of SC | text-guided image editing metric · EditVal | s2AFC: 0.764 -> 0.831 | Helped |
| use AugCLIP instead of CLIP-I | text-guided image editing metric · DreamBooth | s2AFC: 0.857 -> 0.857 | No clear effect |
| use AugCLIP instead of SC | text-guided image editing metric · TEdBench | AccBoth: 0.420 -> 0.570 | Helped |
| use AugCLIP instead of SC | text-guided image editing metric · MagicBrush | AccBoth: 0.936 -> 0.889 | Hurt |
| augment CLIPdir with source attributes (DS) (ablation) | CLIPdir · EditVal | s2AFC: 0.697 -> 0.629 | Hurt |
| augment CLIPdir with target attributes (DT) (ablation) | CLIPdir · CelebA | s2AFC: 0.673 -> 0.819 | Helped |
| augment CLIPdir with target attributes (DT) (ablation) | CLIPdir · TEdBench | AccBoth: 0.350 -> 0.420 | Helped |
| augment CLIPdir with source and target attributes (DS ∪ DT) (ablation) | CLIPdir · EditVal | s2AFC: 0.697 -> 0.607 | Hurt |
| augment CLIPdir with source and target attributes (DS ∪ DT) (ablation) | CLIPdir · MagicBrush | AccBoth: 0.601 -> 0.402 | Hurt |
| add attribute weighting strategy (ablation) | AugCLIP · EditVal | s2AFC: 0.787 -> 0.831 | Helped |
| add attribute weighting strategy (ablation) | AugCLIP · TEdBench | AccBoth: 0.400 -> 0.570 | Helped |
Surprising findings
- Naively combining preservation-centric and modification-centric metrics often degrades human alignment.
- Simply augmenting CLIPdir with extracted attributes does not reliably improve it, with gains inconsistent across datasets.
Limitations
- Extra computation from MLLM attribute extraction and hyperplane fitting versus CLIPdir.
- Depends on GPT-4V attribute quality; attributes are said to be generated with some randomness across seeds.
- Human study covers only three of the five benchmarks with 30-45 participants per dataset.
Experimental setup
Datasets: TEdBench EditVal MagicBrush DreamBooth CelebA
Benchmarks: TEdBench EditVal MagicBrush DreamBooth CelebA
Metrics: s2AFC human alignment AccBoth ground truth selection accuracy
Baselines: L2 LPIPS DINO similarity Segmentation Consistency CLIP-I CLIP-T CLIPdir
Builds on: CLIPdir (directional CLIP similarity) CLIP Relevancy maps
Closest papers by meaning
- Balancing Preservation and Modification: A Region and Semantic Aware Metric for Instruction-Based Image Editing (ICML 2025 · similarity 0.89)
- Prompt Augmentation for Self-supervised Text-guided Image Manipulation (CVPR 2024 · similarity 0.88)
- ImgEdit: A Unified Image Editing Dataset and Benchmark (NeurIPS 2025 · similarity 0.88)
- PairEdit: Learning Semantic Variations for Exemplar-based Image Editing (NeurIPS 2025 · similarity 0.87)
- Improving Editability in Image Generation with Layer-wise Memory (CVPR 2025 · similarity 0.87)
- TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Control (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.