Research map / Image generation / Editing and style transfer
CLIPAway: Harmonizing focused embeddings for removing objects via diffusion models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLIPAway, a plug-and-play method that uses AlphaCLIP embeddings, an MLP projection to IP-Adapter space, and orthogonal vector subtraction of foreground from background embeddings to make diffusion inpainting remove objects without hallucinating replacements.
Problem
Diffusion-based inpainting often hallucinates or re-inserts the object the user wants removed, instead of filling the region with background, and existing object-removal fixes need specialized datasets or costly annotations.
Approach
Use AlphaCLIP region-focused embeddings for foreground and background, map them into the IP-Adapter embedding space with a trained MLP, and subtract the projection of the foreground embedding from the background embedding so the conditioning is orthogonal to the object; this is plugged into existing diffusion inpainters without an object-removal training set.</µcore_idea>
Key results
- SD-Inpaint + CLIPAway improves CLIP@1 from 70.45 to 84.82 and CLIP@3 from 57.14 to 74.42, with FID reduced from 59.21 to 57.32.
- Unipaint + CLIPAway reduces FID from 77.58 to 62.18 and raises CLIP@1 from 85.38 to 88.26.
- In a 20-participant user study, humans preferred CLIPAway 71.75% of the time versus 13.25% ZITS++, 11.25% SD-Inpaint, 3.75% Unipaint.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CLIPAway to SD-Inpaint | SD-Inpaint · COCO 2017 validation | FID: 59.21 -> 57.32 | Helped |
| Add CLIPAway to SD-Inpaint | SD-Inpaint · COCO 2017 validation | CMMD: 0.54 -> 0.53 | Helped |
| Add CLIPAway to SD-Inpaint | SD-Inpaint · COCO 2017 validation | CLIP Distance: 0.75 -> 0.81 | Helped |
| Add CLIPAway to SD-Inpaint | SD-Inpaint · COCO 2017 validation | CLIP@1: 70.45 -> 84.82 | Helped |
| Add CLIPAway to SD-Inpaint | SD-Inpaint · COCO 2017 validation | CLIP@3: 57.14 -> 74.42 | Helped |
| Add CLIPAway to SD-Inpaint | SD-Inpaint · COCO 2017 validation | CLIP@5: 49.88 -> 67.76 | Helped |
| Add CLIPAway to Blended Diff. | Blended Diff. · COCO 2017 validation | FID: 72.24 -> 61.66 | Helped |
| Add CLIPAway to Unipaint | Unipaint · COCO 2017 validation | CLIP Distance: 0.78 -> 0.84 | Helped |
| Use CLIPAway with SD-Inpaint | SD-Inpaint · COCO 2017 validation user study | participant preference: 11.25\% -> 71.75\% | Helped |
Surprising findings
- Providing the text prompt 'background' to diffusion inpainters did not change results compared with an empty prompt.
- Direct IP-Adapter use with the full input image as image prompt fills the erased area with the original object.
Limitations
- Diffusion inference is slower than GAN-based methods and not real-time.
- Shadows not included in the mask are not removed.
- Relies on the mask covering the object; the method is demonstrated on latent diffusion models only.
Experimental setup
Datasets: COCO 2017
Benchmarks: COCO 2017 validation
Metrics: FID CMMD CLIP Distance CLIP Accuracy (CLIP@1/3/5) user study preference
Baselines: ZITS++ MAT LaMa Blended Latent Diffusion Unipaint SD-Inpaint
Builds on: AlphaCLIP IP-Adapter Stable Diffusion inpainting Blended Latent Diffusion Unipaint
Closest papers by meaning
- Paint by Inpaint: Learning to Add Image Objects by Removing Them First (CVPR 2025 · similarity 0.88)
- In-N-Out: Lifting 2D Diffusion Prior for 3D Object Removal via Tuning-Free Latents Alignment (NeurIPS 2024 · similarity 0.87)
- DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing (CVPR 2024 · similarity 0.86)
- DesignEdit: Unify Spatial-Aware Image Editing via Training-free Inpainting with a Multi-Layered Latent Diffusion Framework (AAAI 2025 · similarity 0.86)
- Attentive Eraser: Unleashing Diffusion Model’s Object Removal Potential via Self-Attention Redirection Guidance (AAAI 2025 · similarity 0.86)
- DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.