Research map / Adversarial robustness and security / Attacks on text and graphs
STEREO: A Two-Stage Framework for Adversarially Robust Concept Erasing from Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STEREO, a two-stage concept erasure method for text-to-image diffusion models that uses min-max adversarial training to find adversarial prompts and then a single anchor-concept compositional erasing objective to robustly erase concepts while preserving utility.
Problem
Post-hoc concept erasure in text-to-image diffusion models is vulnerable to adversarial prompts and embedding-space inversion attacks, and existing robust erasure methods trade off utility for robustness.
Approach
Use iterative min-max adversarial training (erase-then-textual-inversion attack) only to discover a diverse set of adversarial prompts, then erase the concept in one fine-tuning pass using a compositional objective that adds anchor-concept prompts as positive guidance and the adversarial prompts as negative guidance.
Key results
- STEREO reaches 1.05% ASR on nudity under UnlearnDiff and 4.21% under CCE, versus 91.58%+ for SD v1.4 baselines ranges and up to 53.08%/46.31% for robust baselines RECE/RACE under CCE/UD.
- On the tench object under CCE, STEREO reduces ASR to 9.78% from 91.00% for AdvUnlearn and 97.20% for SD v1.4, with FID 16.49.
- STEREO's average FID is 16.1 and CLIP 30.5, deviating from SD v1.4 by 1.99 FID and 0.81 CLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use STEREO instead of AdvUnlearn | Stable Diffusion 1.4 · Nudity removal (CCE) | ASR (%): 66.31 -> 2.10 | Helped |
| use STEREO instead of AdvUnlearn | Stable Diffusion 1.4 · Van Gogh style removal (CCE) | ASR (%): 51.80 -> 17.00 | Helped |
| use STEREO instead of AdvUnlearn | Stable Diffusion 1.4 · Tench removal (CCE) | ASR (%): 91.00 -> 9.78 | Helped |
| add the REO stage after STE (ablation) | STEREO · Nudity removal (CCE) | ASR (%): 54.73 -> 4.21 | Helped |
| add the REO stage after STE (ablation) | STEREO · Nudity removal | FID: 50.32 -> 15.07 | Helped |
| replace the REO objective with ESD and adversarial prompts (ablation) | STEREO · Nudity removal (CCE) | ASR (%): 4.21 -> 35.78 | Hurt |
| replace the REO objective with ESD and adversarial prompts (ablation) | STEREO · Nudity removal | FID: 15.07 -> 38.06 | Hurt |
| replace the REO objective with AC and adversarial prompts (ablation) | STEREO · Nudity removal (CCE) | ASR (%): 4.21 -> 86.31 | Hurt |
| increase adversarial prompts from 0 to 1 (ablation) | STEREO · Nudity removal (CCE) | ASR (%): 46.31 -> 8.42 | Helped |
| increase adversarial prompts from 1 to 2 (ablation) | STEREO · Nudity removal (CCE) | ASR (%): 8.42 -> 4.21 | Helped |
Surprising findings
- Iterative adversarial training in the first stage alone destroys utility (FID 50.32), so the second single-stage erasure is needed to recover utility.
Limitations
- Struggles to erase multiple concepts simultaneously while keeping robustness
- Multiple min-max iterations incur higher computational time than closed-form robust erasure methods
Experimental setup
Datasets: I2P Gallery images generated by SD v1.4 GPT-4-generated anchor prompts
Benchmarks: I2P nudity prompts with Ring-A-Bell UnlearnDiff CCE attack on Van Gogh style CCE attack on tench object FID CLIP score
Metrics: ASR FID CLIP score
Baselines: ESD Ablating Concepts UCE MACE RECE RACE AdvUnlearn
Builds on: ESD Circumventing Concept Erasure Adaptive Projected Guidance Compositional inference Textual inversion
Closest papers by meaning
- Erasing Undesirable Concepts in Diffusion Models with Adversarial Preservation (NeurIPS 2024 · similarity 0.91)
- Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Models (NeurIPS 2024 · similarity 0.91)
- Concept Pinpoint Eraser for Text-to-image Diffusion Models via Residual Attention Gate (ICLR 2025 · similarity 0.89)
- ConceptPrune: Concept Editing in Diffusion Models via Skilled Neuron Pruning (ICLR 2025 · similarity 0.88)
- Semantic Surgery: Zero-Shot Concept Erasure in Diffusion Models (NeurIPS 2025 · similarity 0.88)
- All but One: Surgical Concept Erasing with Model Preservation in Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.