Research map / Image generation / Text-to-image diffusion
Growth Inhibitors for Suppressing Inappropriate Image Concepts in Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GIE, a fine-tuning-free concept erasure method for text-to-image diffusion that injects re-weighted cross-attention "growth inhibitors" for the target concept, with a small MLP adapter predicting the suppression scale.
Problem
Text-to-image diffusion models generate NSFW content and style-copyright infringements from implicit unsafe prompts that lack explicit unsafe words; existing erasure methods require per-concept fine-tuning, risk catastrophic forgetting, or are coarse.
Approach
Compute cross-attention maps for the target concept's tokens during denoising, re-weight them with a suppression scale, and inject them into the prompt's attention map group (and text embedding) so inappropriate image features are suppressed at the image level; an MLP adapter over first-step cross-attention mean values infers the scale, trained on GPT-4o-labeled samples.
Key results
- On I2P, GIE reaches an NRR of about 0.70-0.72 across prompt-length groups, versus 0.56-0.62 for the strongest baselines (ESD-u, Safe-CLIP) and 0.06-0.23 for SLD, ESD-x and SPM.
- On COCO-30k, GIE gets the highest CLIP score (26.433) among compared methods, while SPM has the best FID (13.566) and GIE scores 15.452.
- For object erasure, GIE achieves ORRs above 0.9 for car, airplane and cat (synonyms included) and at most 0.05 on non-target objects.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GIE instead of the strongest listed baseline, S-Clip | NSFW concept erasure · I2P-derived prompts, 1–20, 21–40, and 40+ tokens | NRR: 0.616 -> 0.712; 0.631 -> 0.720; 0.637 -> 0.698 | Helped |
| Use GIE instead of SPM | Image generation after concept erasure · COCO-30k | CLIP score: 26.221 -> 26.433 | Helped |
| Use GIE instead of SPM | Image generation after concept erasure · COCO-30k | FID: 13.566 -> 15.452 | Hurt |
| Erase Car using GIE | GIE · Car prompts (Car, Automobile, Motorcar) | ORR: 0.98; 1.00; 1.00 | Helped |
| Erase Airplane using GIE | GIE · Airplane prompts (Airplane, Aircraft, Aeroplane) | ORR: 0.98; 0.95; 0.96 | Helped |
| Erase Cat using GIE | GIE · Cat prompts (Cat, Kitten, Pussycat) | ORR: 0.95; 0.92; 0.94 | Helped |
| Replace the adapter-inferred suppression scale with fixed w = 10 (ablation) | GIE · I2P, target concept nude | CLIP score: 27.052 -> 26.231 | Hurt |
| Replace the adapter-inferred suppression scale with fixed w = 5 (ablation) | GIE · I2P, target concept nude | CLIP score: 27.052 -> 27.124 | No clear effect |
| Inject growth inhibitors before [EOT], after [SOT], or at a random position (ablation) | GIE · I2P, target concept nude | NRR | No clear effect |
| Use a fixed suppression scale instead of an adapter-inferred scale (ablation) | GIE · I2P, target concept nude | NRR | Mixed |
Surprising findings
- Reversing the order of tokens in text embeddings (happy/sad swap) yields identical images, which the authors use to argue that injected attention maps act additively regardless of position.
Limitations
- Adapter labels depend on GPT-4o judgments, and the suppression-scale range and interval were chosen heuristically.
- Fixed suppression scales either under-erase or distort images, so the adapter is needed; semantic similarity among target concepts can cause over-erasure and broken images.
- Main NSFW and style evaluations use SD v1.4 and a single nudity detector (NudeNet), with limited human evaluation.
Experimental setup
Datasets: I2P COCO-30k NSFW-prompt dataset (crawled images converted with BLIP-2)
Benchmarks: I2P COCO-30k
Metrics: NSFW removal rate (NRR) object removal rate (ORR) FID CLIP score
Baselines: SD v1.4 SD v2.1 SD + NEG SLD Safe-CLIP ESD-u ESD-x SPM
Builds on: classifier-free guidance ESD SPM Safe Latent Diffusion
Closest papers by meaning
- MACE: Mass Concept Erasure in Diffusion Models (CVPR 2024 · similarity 0.91)
- Detect-and-Guide: Self-regulation of Diffusion Models for Safe Text-to-Image Generation via Guideline Token Optimization (CVPR 2025 · similarity 0.91)
- ConceptPrune: Concept Editing in Diffusion Models via Skilled Neuron Pruning (ICLR 2025 · similarity 0.91)
- Comprehensive Assessment and Analysis for NSFW Content Erasure in Text-to-Image Diffusion models (NeurIPS 2025 · similarity 0.91)
- All but One: Surgical Concept Erasing with Model Preservation in Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.90)
- Localized Concept Erasure for Text-to-Image Diffusion Models Using Training-Free Gated Low-Rank Adaptation (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.