Research map / Image generation / Editing and style transfer
Referring Image Editing: Object-level Image Editing via Referring Expressions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Referring Image Editing (RIE), a task where a single text prompt with a referring expression and an edit instruction edits one specific object, and proposes ReferDiffusion with a Region-Embedder Module, a region-control loss, and mixed-supervised training on a new RefCOCO-Edit dataset.
Problem
General and instruction-based diffusion image editors struggle to edit a specific object when other similar objects coexist in the scene, and region-based editors need manual masks or boxes.
Approach
Extends referring segmentation with generative editing: a Region-Embedder Module splits the prompt condition into vision, referring and editing embeddings, a region-control latent loss keeps background pixels unchanged, and a two-stage mixed-supervised strategy first self-supervises on referring segmentation data and then fine-tunes on RefCOCO-Edit.
Key results
- ReferDiffusion achieves CLIP-Sim_fg 25.06 and L2_bg 0.028, better than InstructPix2Pix (23.72, 0.031), InstructDiffusion (24.07, 0.044) and MagicBrush (24.32, 0.034) on RefCOCO-Edit.
- In the user study (50 participants), ReferDiffusion scores 3.63 on consistency versus 3.27 for the next best (MagicBrush).
- Ablation: adding Region-Embedder Module and self-supervised stage raises CLIP-Sim_fg from 24.28 (fine-tune) to 25.06 and lowers L2_bg from 0.049 to 0.028.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ReferDiffusion instead of MagicBrush | Referring image editing · RefCOCO-Edit | L2bg: 0.034 -> 0.028 | Helped |
| Use ReferDiffusion instead of MagicBrush | Referring image editing · RefCOCO-Edit | CLIP-Simfg: 24.32 -> 25.06 | Helped |
| Use ReferDiffusion instead of MagicBrush | Referring image editing · RefCOCO-Edit | AP: 4.62 -> 4.60 | Hurt |
| Use ReferDiffusion instead of MagicBrush | Referring image editing · User study | Consistency: 3.27 -> 3.63 | Helped |
| Add Region-Embedder to fine-tuned InstructPix2Pix (ablation) | ReferDiffusion · RefCOCO-Edit | AP: 4.56 -> 4.64 | Helped |
| Add Region-Embedder to fine-tuned InstructPix2Pix (ablation) | ReferDiffusion · RefCOCO-Edit | L2bg: 0.049 -> 0.032 | Helped |
| Add Region-Embedder to fine-tuned InstructPix2Pix (ablation) | ReferDiffusion · RefCOCO-Edit | CLIP-Simfg: 24.28 -> 24.17 | Hurt |
| Add Region-Control Loss to Region-Embedder model (ablation) | ReferDiffusion · RefCOCO-Edit | L2bg: 0.032 -> 0.026 | Helped |
| Add Region-Control Loss to Region-Embedder model (ablation) | ReferDiffusion · RefCOCO-Edit | CLIP-Simfg: 24.17 -> 24.72 | Helped |
| Add self-supervised training to Region-Control Loss model (ablation) | ReferDiffusion · RefCOCO-Edit | CLIP-Simfg: 24.72 -> 25.06 | Helped |
| Add self-supervised training to Region-Control Loss model (ablation) | ReferDiffusion · RefCOCO-Edit | L2bg: 0.026 -> 0.028 | Hurt |
Surprising findings
- Region-based editing with ground-truth masks (Blend Latent Diffusion) performs poorly, since it cannot adjust shape or context to the prompt.
Limitations
- Target identification is weaker than specialized referring segmentation models on long or complex prompts.
- Generated images can be unnatural in challenging scenes, depending on backbone and synthesized data.
- RefCOCO-Edit is small (400 images) and its reference outputs are produced by existing generative models.
Experimental setup
Datasets: RefCOCO RefCOCO-Edit
Benchmarks: RefCOCO-Edit
Metrics: Aesthetic Predictor score L2 background score CLIP similarity of foreground user study quality and consistency
Baselines: Blend Latent Diffusion InstructPix2Pix InstructDiffusion MagicBrush
Builds on: InstructPix2Pix Stable Diffusion Classifier-free guidance DragGAN
Closest papers by meaning
- An Item Is Worth a Prompt: Versatile Image Editing with Disentangled Control (AAAI 2025 · similarity 0.89)
- Early Semantic Grounding in Image Editing Models for Zero-Shot Referring Image Segmentation (NeurIPS 2026 · similarity 0.88)
- PAIR Diffusion: A Comprehensive Multimodal Object-Level Image Editor (CVPR 2024 · similarity 0.88)
- Object-Aware Inversion and Reassembly for Image Editing (ICLR 2024 · similarity 0.87)
- Prompt Augmentation for Self-supervised Text-guided Image Manipulation (CVPR 2024 · similarity 0.87)
- Prompt-Driven Referring Image Segmentation with Instance Contrasting (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.