Research map / Image generation / Text-to-image diffusion
All but One: Surgical Concept Erasing with Model Preservation in Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a surgical fine-tuning method for concept erasing in Stable Diffusion that updates only the classifier-guidance term for the target concept while regularizing the unconditional score to preserve the model's utility.
Problem
Concept erasure fine-tuning in text-to-image diffusion models (e.g., removing nudity) severely degrades model utility: corrupted feature space, spatial/semantic drift from the same seed, and utility harm from many update steps.
Approach
Decompose the conditional score via classifier guidance and update only the guidance term for the erased concept (toward a user-chosen replacement concept), while a Lagrangian-derived penalty with stop-gradient anchors the unconditional score to the original checkpoint; the penalty weight lambda controls erasure-preservation trade-off.
Key results
- Achieves FID 13.19 (best among methods, vs. 13.59 for SD v1.4) and SSIM 0.762 against the original checkpoint, versus 0.231 for ESD and 0.309 for SDD.
- NudeNet nudity score 0.33% vs. 0.69% for SD v1.4; ESD (0.04%) and SDD (0.05%) erase more aggressively but with much lower SSIM.
- Trained for 200 steps, while ESD and SDD require about 1,000 or more iterations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use surgical concept-score updates with unconditional-score regularization | Ours vs. Stable Diffusion v1.4 · Nudity erasure / COCO utility | FID: 13.59 -> 13.19 | Helped |
| Use surgical concept-score updates with unconditional-score regularization | Ours vs. Stable Diffusion v1.4 · Nudity erasure / COCO utility | SSIM | Helped |
| Use surgical concept-score updates with unconditional-score regularization | Ours vs. ESD · Nudity erasure / COCO utility | NudeNet (%): 0.04 -> 0.33 | Hurt |
| Use surgical concept-score updates with unconditional-score regularization | Ours vs. ESD · Nudity erasure / COCO utility | KID: 0.00421 -> 0.00447 | Hurt |
| Increase the unconditional-score regularization coefficient to λ=1.5 (ablation) | Ours · Nudity erasure | spatial consistency | Helped |
| Increase the unconditional-score regularization coefficient to λ=1.5 (ablation) | Ours · Nudity erasure | erasure strength | Hurt |
| Set the unconditional-score regularization coefficient to λ=0 (ablation) | Ours · DDIM inversion concept purification | concept erasure | Helped |
| Use explicit guidance instead of implicit Prompt-to-Prompt guidance (ablation) | Ours · Nudity erasure | sample quality | Helped |
| Update the attention to-out layer rather than attention-map-producing weights (ablation) | Ours · 1,000 random objects | SSIM | Helped |
| Use cross-attention and to-out layers rather than cross-attention alone (ablation) | Ours · 1,000 random objects | SSIM | Helped |
| Train with 200 update steps rather than 1,000 or more | Ours vs. ESD and SDD · Nudity erasure | training iterations: 200 vs. 1,000 or more | Helped |
Surprising findings
- With lambda=0 (no unconditional penalty), DDIM inversion of real nude images conditioned on the erased concept or null token purifies them, a capability other baselines did not show.
Limitations
- Erasing painting styles either erases most styles uniformly or is too conservative under strong constraint.
- Explicit guidance is mostly necessary; implicit Prompt-to-Prompt guidance is sensitive to hyperparameters.
- Evaluation focuses mainly on nudity; the paper notes the NudeNet threshold was set to 0.7 due to false positives.
Experimental setup
Datasets: MS-COCO captions (for evaluation image generation)
Benchmarks: MS-COCO
Metrics: NudeNet score FID KID CLIP score SSIM
Baselines: ESD SDD Ablating Concepts Forget-Me-Not
Builds on: Erased Stable Diffusion (ESD) classifier-free guidance Ablating Concepts Semantic Guidance (SEGA) Prompt-to-Prompt
Closest papers by meaning
- Semantic Surgery: Zero-Shot Concept Erasure in Diffusion Models (NeurIPS 2025 · similarity 0.93)
- Localized Concept Erasure for Text-to-Image Diffusion Models Using Training-Free Gated Low-Rank Adaptation (CVPR 2025 · similarity 0.92)
- Precise, Fast, and Low-cost Concept Erasure in Value Space: Orthogonal Complement Matters (CVPR 2025 · similarity 0.91)
- MACE: Mass Concept Erasure in Diffusion Models (CVPR 2024 · similarity 0.91)
- CURE: Concept Unlearning via Orthogonal Representation Editing in Diffusion Models (NeurIPS 2025 · similarity 0.91)
- Fine-Grained Erasure in Text-to-Image Diffusion-based Foundation Models (CVPR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.