Research map / Image generation / Autoregressive and latent generation

Attentive Eraser: Unleashing Diffusion Model’s Object Removal Potential via Self-Attention Redirection Guidance

AAAI 2025 · Image generation › Autoregressive and latent generation · approach: Compositional scene generation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Attentive Eraser is a tuning-free method that redirects diffusion self-attention (Attention Activation and Suppression plus Self-Attention Redirection Guidance) to remove masked objects and repaint plausible background in pre-trained Stable Diffusion models.

Problem

Diffusion-based object removal suffers from random artifacts, incomplete removal, and poor repainting of the masked foreground after object deletion, and existing fixes need costly fine-tuning or heavy prompt engineering.

Approach

Modify the self-attention maps inside a pre-trained SD U-net using the object mask so foreground tokens attend to background and not to themselves (AAS, with similarity suppression), then use this redirected prediction as a PAG-style guidance term during sampling (SARG).

Key results

• On OpenImages V5 (10k images), SD2.1+SIP with SARG reaches Local FID 5.835 and CLIP consensus 0.0734, versus 15.58 and 0.1347 without SARG, and best-in-class local FID among compared methods (LAMA 6.091).
• In a user study with 10 participants, SD2.1+SIP w/ SARG was preferred 44.9% of the time, versus 19.7% for LAMA and 10% for SD2.1-inpainting.
• In GPT-4o pairwise evaluation against LAMA over 3000 image pairs, the proposed method was selected 74.47% of the time.

Measured effects reported in the paper

Surprising findings

• Global metrics FID and LPIPS do not reflect removal effectiveness; the proposed method is only average on them (FID 7.352 vs. best 3.805).

Limitations

• Guidance requires two U-Net noise predictions per step, increasing inference time.
• When the mask is very large, few background references exist, causing reconstruction artifacts.
• Global metrics FID and LPIPS do not fully capture removal quality, and the method is only average on them.

Experimental setup

Datasets: OpenImages V5

Benchmarks: OpenImages V5

Metrics: FID LPIPS Local FID CLIP consensus CLIP score user preference GPT-4o selection rate

Baselines: SD-Inpaint LAMA Inst-Inpaint PowerPaint SD2.1 with SIP

Builds on: Perceptual Attention Guidance (PAG) Blended Latent Diffusion (BLD) DiffEdit DDIM inversion Self-Attention Guidance (SAG)

Closest papers by meaning

• CLIPAway: Harmonizing focused embeddings for removing objects via diffusion models (NeurIPS 2024 · similarity 0.86)
• Erase Diffusion: Empowering Object Removal Through Calibrating Diffusion Pathways (CVPR 2025 · similarity 0.86)
• Diffusion Handles Enabling 3D Edits for Diffusion Models by Lifting Activations to 3D (CVPR 2024 · similarity 0.85)
• Concept Pinpoint Eraser for Text-to-image Diffusion Models via Residual Attention Gate (ICLR 2025 · similarity 0.85)
• Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing (CVPR 2024 · similarity 0.85)
• Fine-Grained Erasure in Text-to-Image Diffusion-based Foundation Models (CVPR 2025 · similarity 0.85)