Research map / Image generation / Editing and style transfer
Paint by Inpaint: Learning to Add Image Objects by Removing Them First
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds PIPE, a ~1M-pair mask-free object-addition dataset by inpainting-away objects from segmentation datasets and reversing the pairs, then trains an SD1.5-based instruction editing diffusion model on it.
Problem
Mask-free, text-instructed addition of objects to images is hard, and existing instruction-editing datasets (e.g., InstructPix2Pix) are synthetic and suffer source-target inconsistency.
Approach
Object removal is easier than addition, so remove objects from real images with a mask-based inpainting model, filter the outputs, and use (inpainted, original) pairs reversed as natural source-target addition training pairs with VLM-LLM generated instructions.
Key results
- On the MagicBrush test subset, the model reaches L1 .072, CLIP-I .900, DINO .852 versus IP2P .100 / .860 / .766 (Table 2).
- On the PIPE test set, L1 .057 and DINO .903 versus Hive .088 / .754 and IP2P .098 / .753 (Table 3).
- In a human study on 100 Conceptual Captions images, the model was preferred about 72.6% of the time over IP2P on average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the proposed model on PIPE rather than use IP2P | Object addition editing model · PIPE test set | L1: .098 -> .057 | Helped |
| Train the proposed model on PIPE rather than use IP2P | Object addition editing model · PIPE test set | CLIP-I: .861 -> .945 | Helped |
| Train the proposed model on PIPE rather than use IP2P | Object addition editing model · OPA | DINO: .647 -> .735 | Helped |
| Train on combined PIPE and IP2P data, then fine-tune on MagicBrush | IP2P general editing model · MagicBrush full general editing test set | L1: .082 -> .074 | Helped |
| Train on combined PIPE and IP2P data, then fine-tune on MagicBrush | IP2P general editing model · MagicBrush full general editing test set | CLIP-I: .896 -> .906 | Helped |
| Train on combined PIPE and IP2P data, then fine-tune on MagicBrush | IP2P general editing model · MagicBrush full general editing test set | DINO: .845 -> .866 | Helped |
Limitations
- Dataset is built with a Stable Diffusion inpainting model and filtered with CLIP heuristics, so inpainting artifacts may persist.
- Evaluated mainly on object addition; general editing gains are shown only when combined with IP2P data.
- Reference instructions use templates, limiting instruction diversity.
Experimental setup
Datasets: COCO Open Images LVIS RefCOCO RefCOCO+ RefCOCOg InstructPix2Pix dataset MagicBrush OPA Conceptual Captions
Benchmarks: PIPE test set OPA MagicBrush
Metrics: L1 L2 CLIP-I DINO CLIP-T CMMD directional CLIP similarity human preference
Baselines: InstructPix2Pix Hive VQGAN-CLIP SDEdit Null-Text Inversion Pix2PixZero Edit-Friendly DDPM MagicBrush fine-tuned IP2P
Builds on: Stable Diffusion 1.5 InstructPix2Pix MagicBrush
Closest papers by meaning
- Towards Enhanced Image Inpainting: Mitigating Unwanted Object Insertion and Preserving Color Consistency (CVPR 2025 · similarity 0.90)
- HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models (ICLR 2025 · similarity 0.89)
- PAIR Diffusion: A Comprehensive Multimodal Object-Level Image Editor (CVPR 2024 · similarity 0.88)
- Add-it: Training-Free Object Insertion in Images With Pretrained Diffusion Models (ICLR 2025 · similarity 0.88)
- CLIPAway: Harmonizing focused embeddings for removing objects via diffusion models (NeurIPS 2024 · similarity 0.88)
- RAD: Region-Aware Diffusion Models for Image Inpainting (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.