Research map / Image generation / Text-to-image diffusion
Concept Pinpoint Eraser for Text-to-image Diffusion Models via Residual Attention Gate
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Concept Pinpoint Eraser (CPE), which adds nonlinear Residual Attention Gates to cross-attention key/value projections, trained with an attention anchoring loss and adversarial residual embeddings (RARE), to erase celebrities, artistic styles, and explicit content from Stable Diffusion v1.4.
Problem
Concept erasing in text-to-image diffusion models must remove target concepts while preserving diverse remaining concepts and resisting adversarial attack prompts; prior cross-attention-only fine-tuning may distort remaining concepts.
Approach
Replace purely linear cross-attention weight updates with nonlinear, text-embedding-dependent Residual Attention Gates that only activate for target concepts, regularized by an attention anchoring loss on anchor concepts and trained adversarially with learnable residual embeddings.
Key results
- On 50-celebrity erasure, CPE reaches 0.37% GCD accuracy versus 0.09% for UCE, while keeping 100-celebrity CS at 34.82 (vs 34.83 for SD v1.4) and 88.26% ACC.
- On I2P explicit content, CPE detects 40 nudity instances in total, versus 89 for the second-best RECE and 743 for SD v1.4.
- CPE achieves 0.00% attack success rate on Ring-A-Bell for I2P nudity, but 30.28 on UnlearnDiff, where AdvUnlearn scores 21.13.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add ResAG modules (ablation) | CPE · 50 Celebrities | ACC: 9.83 -> 0.37 | Helped |
| add ResAG modules (ablation) | CPE · 100 Celebrities (remaining) | KID (×100): 0.34 -> 0.08 | Helped |
| add attention anchoring loss (ablation) | CPE · 100 Celebrities (remaining) | ACC: 72.54 -> 88.26 | Helped |
| add RARE adversarial training (ablation) | CPE · 50 Celebrities | ACC: 2.23 -> 0.37 | Helped |
| increase attention rank from 1 to 16 (ablation) | CPE · 50 Celebrities | ACC: 0.74 -> 0.37 | Helped |
| increase attention rank from 16 to 64 (ablation) | CPE · 50 Celebrities | ACC: 0.37 -> 0.31 | Helped |
| add anchor noise at scale 1.0 × 10^-3 (ablation) | CPE without Mixup · 100 Celebrities (remaining) | KID (×100): 0.19 -> 0.16 | Helped |
| apply Mixup to noisy anchor embeddings (ablation) | CPE · 100 Celebrities (remaining) | KID (×100): 0.16 -> 0.08 | Helped |
| use only similar expressions instead of the selected anchor combination (ablation) | CPE · 100 Celebrities (remaining) | KID (×100): 0.08 -> 0.22 | Hurt |
| use only similar celebrities instead of the selected anchor combination (ablation) | CPE · 50 Celebrities | ACC: 0.37 -> 0.63 | Hurt |
| erase explicit contents with CPE rather than RECE | Stable Diffusion v1.4 · I2P | Detected quantity (total): 89 -> 40 | Helped |
| erase Van Gogh style with CPE rather than AdvUnlearn | Stable Diffusion v1.4 · Vincent van Gogh style (UnlearnDiff attack) | ASR: 2 -> 0 | Helped |
Surprising findings
- Only updating cross-attention layers (a linear modification) is theoretically shown to be limited in preserving remaining concepts.
- Anchor-concept noise and mixup augmentation give a further preservation gain without harming erasure.
Limitations
- UnlearnDiff attack success rate on I2P nudity remains at 30.28%, behind AdvUnlearn's 21.13%.
- Anchor concept selection and hyperparameters (eta, lambda, rank) vary by experiment and require tuning.
Experimental setup
Datasets: COCO-30K I2P
Benchmarks: COCO-30K I2P Ring-A-Bell UnlearnDiff
Metrics: CLIP score GIPHY Celebrity Detector accuracy FID KID NudeNet detection count attack success rate
Baselines: FMN ESD-x ESD-u UCE MACE RECE AdvUnlearn
Builds on: ESD UCE MACE RECE AdvUnlearn FMN
Closest papers by meaning
- MACE: Mass Concept Erasure in Diffusion Models (CVPR 2024 · similarity 0.92)
- Localized Concept Erasure for Text-to-Image Diffusion Models Using Training-Free Gated Low-Rank Adaptation (CVPR 2025 · similarity 0.91)
- Erasing Undesirable Concepts in Diffusion Models with Adversarial Preservation (NeurIPS 2024 · similarity 0.91)
- All but One: Surgical Concept Erasing with Model Preservation in Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.90)
- ConceptPrune: Concept Editing in Diffusion Models via Skilled Neuron Pruning (ICLR 2025 · similarity 0.90)
- Unlearning Concepts in Diffusion Model via Concept Domain Correction and Concept Preserving Gradient (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.