Research map / Image generation / Text-to-image diffusion
Erasing Undesirable Concepts in Diffusion Models with Adversarial Preservation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes adversarial concept preservation: during concept erasure in Stable Diffusion, an inner max step finds the most parameter-sensitive remaining concept (via Gumbel-Softmax over a vocabulary) and preserves it, reducing side effects on other concepts.
Problem
Erasing undesirable concepts from text-to-image diffusion models degrades the model's ability to generate related or other concepts, and preserving a neutral concept (e.g., "a photo" or empty string) as an anchor is not an adequate safeguard.
Approach
A bilevel min-max objective that, at each step, adversarially searches over the remaining concept space (relaxed with a Gumbel-Softmax distribution over embeddings) for the concept most affected by the parameter update and explicitly preserves it alongside erasing the target concept.
Key results
- On Imagenette 5-class erasure, the method reaches 98.6% ESR-1 and 55.2% PSR-1, versus 44.2% PSR-1 for the best baseline (CA).
- On I2P nudity erasure at threshold 0.3, NER drops to 3.64% vs 6.87% for UCE and 5.32% for ESD, with FID 15.52 vs 15.98 for UCE.
- On artist-style erasure, the method reaches the lowest CLIP score on erased artists (21.57 vs 23.56 for ESD).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use adversarial concept preservation instead of baseline erasure | Stable Diffusion 1.4 · Imagenette object erasure | PSR-1: 44.2\pm9.7 -> 55.2\pm10.0 | Helped |
| use adversarial concept preservation instead of baseline erasure | Stable Diffusion 1.4 · Imagenette object erasure | ESR-1: 100\pm0.0 -> 98.6\pm1.1 | Mixed |
| use adversarial concept preservation instead of ESD | Stable Diffusion 1.4 · I2P nudity erasure | NER-0.3: 5.32 -> 3.64 | Helped |
| use adversarial concept preservation instead of UCE | Stable Diffusion 1.4 · COCO 30K validation | FID: 15.98 -> 15.52 | Helped |
| use adversarial concept preservation instead of ESD | Stable Diffusion 1.4 · Artistic styles to erase | CLIP: 23.56\pm4.73 -> 21.57\pm5.46 | Helped |
| increase search steps from N_iter=2 to N_iter=8 (ablation) | Adversarial concept preservation · Imagenette object erasure | PSR-1: 63.80 -> 54.12 | Hurt |
| reduce search space from K=100 to K=20 (ablation) | Adversarial concept preservation · Imagenette object erasure | ESR-1: 98.72 -> 96.76 | Hurt |
| use CLIP token vocabulary instead of Oxford 3000 (ablation) | Adversarial concept preservation · Imagenette object erasure | PSR-1: 63.80 -> 69.24 | Helped |
| use CLIP token vocabulary instead of Oxford 3000 (ablation) | Adversarial concept preservation · Imagenette object erasure | ESR-1: 98.72 -> 97.88 | Hurt |
| fine-tune non-cross-attention modules rather than cross-attention modules (ablation) | ESD · I2P nudity erasure | NER-0.3: 10.25 -> 5.32 | Helped |
| preserve the most sensitive concept rather than a fixed neutral concept | Adversarial concept preservation · Garbage truck erasure | CLIP alignment score | Helped |
Surprising findings
- Neutral concepts such as 'a photo' or the null string are often not the most affected by erasure; related concepts like 'person' or 'women' are more sensitive to removing 'nudity'.
- Adversarial concept search with continuous PGD collapses to background color information, motivating the discrete Gumbel-Softmax relaxation.
Limitations
- Computational cost grows with the size of the concept space since the full concept set embeddings are fed to the model, so K-subset restriction is needed.
- No reliable metric for concept erasure; CLIP alignment and classifier-based detection are imperfect, and CLIP is weak on NSFW content.
- Results depend on the choice of search vocabulary (Oxford 3000 vs CLIP tokens) and on which modules are fine-tuned.
Experimental setup
Datasets: Imagenette I2P prompts COCO 30K
Benchmarks: Imagenette I2P COCO 30K artist style prompts
Metrics: ESR-1 ESR-5 PSR-1 PSR-5 NER FID CLIP score LPIPS
Baselines: ESD UCE Concept Ablation AdvUnlearn
Builds on: ESD UCE Concept Ablation AdvUnlearn Gumbel-Softmax
Closest papers by meaning
- When Are Concepts Erased From Diffusion Models? (NeurIPS 2025 · similarity 0.93)
- Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Models (NeurIPS 2024 · similarity 0.92)
- STEREO: A Two-Stage Framework for Adversarially Robust Concept Erasing from Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.91)
- Concept Pinpoint Eraser for Text-to-image Diffusion Models via Residual Attention Gate (ICLR 2025 · similarity 0.91)
- MACE: Mass Concept Erasure in Diffusion Models (CVPR 2024 · similarity 0.90)
- Fantastic Targets for Concept Erasure in Diffusion Models and Where To Find Them (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.