Research map / Image generation / Text-to-image diffusion
ConceptPrune: Concept Editing in Diffusion Models via Skilled Neuron Pruning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free concept erasure in Stable Diffusion by identifying and pruning concept-specific "skilled" FFN weight neurons (Wanda-based) and zeroing them out.
Problem
Undesired concepts (unsafe content, copyrighted artist styles, object classes, gender bias) learned by text-to-image diffusion models need to be removed; existing concept editing and unlearning methods require fine-tuning or token remapping that is vulnerable to adversarial jailbreaks.
Approach
Adapts the LLM pruning heuristic Wanda to diffusion FFN layers: compares Wanda importance scores of weights on target-concept prompts versus reference prompts to find "skilled" neurons, then permanently zeroes them out, aggregated over the first few denoising steps, without any weight update.
Key results
- On I2P, ConceptPrune generates nudity for 47 of 4703 prompts, a 94.1% decrease versus pre-trained SD (ESD 88%, UCE 85.6%).
- Under UnlearnDiffAtk, ConceptPrune reaches 2.0% top-1 ASR on Van Gogh style versus 94% for UCE and 56% for FMN.
- Pruning roughly 0.12% of total model weights (skilled neurons under 3% of FFN weights) erases target concepts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Prune skilled neurons for target objects | ConceptPrune on Stable Diffusion-v1.5 · ImageNette erased classes (average) | top-1 accuracy: 12.5 -> 1.8 | Helped |
| Prune skilled neurons for target objects | ConceptPrune on Stable Diffusion-v1.5 · ImageNette preserved classes (average) | top-1 accuracy: 66.9 -> 80.4 | Helped |
| Use ConceptPrune rather than UCE for multi-object erasure | Stable Diffusion-v1.5 · Multi-object erasure | accuracy on erased classes: 4% -> 7% | Hurt |
| Use ConceptPrune rather than UCE for multi-object erasure | Stable Diffusion-v1.5 · Multi-object erasure, COCO30k | FID: 17.7 -> 17.5 | Helped |
| Use ConceptPrune rather than UCE for multi-object erasure | Stable Diffusion-v1.5 · Multi-object erasure, COCO30k | CLIP score: 31.0 -> 29.9 | Mixed |
| Prune skilled neurons for artistic styles | ConceptPrune on Stable Diffusion-v1.5 · Artist style erasure | CLIP Accuracy | Mixed |
| Prune skilled neurons for nudity | ConceptPrune on Stable Diffusion-v1.5 · I2P nudity prompts | reduction in generated images with nudity: 88% -> 94.1% | Helped |
| Prune skilled neurons for nudity | ConceptPrune on Stable Diffusion-v1.5 · UnlearnDiffAtk, nudity | ASR | Hurt |
| Prune skilled neurons for gender concepts to reverse gender | ConceptPrune on Stable Diffusion-v1.5 · Winobias, 35 professions | gender reversal success rate: 87 ± 12% | Helped |
| Prune skilled neurons instead of leaving the model unedited | ConceptPrune on Stable Diffusion-v1.5 · MMA-Diffusion black-box attack | reduction in generated images with nudity: +95.6% | Helped |
Surprising findings
- Aggregating the pruning mask over only the first 10 denoising steps suffices to erase a concept while preserving the underlying object.
- Removing unskilled neurons distorts the reference concept while retaining the target concept, suggesting disentanglement of style and object neurons.
Limitations
- Erasing specific objects also inadvertently removes related fine-grained classes (e.g., dog breeds).
- Gender reversal sometimes removes the person from the image entirely.
- Erasing a very large number of objects may degrade overall generation quality.
Experimental setup
Datasets: COCO30k I2P ImageNette UnlearnDiffAtk prompts MMA-Diffusion adversarial prompts Winobias ChatGPT-generated artist prompt set
Benchmarks: COCO30k I2P ImageNette UnlearnDiffAtk MMA-Diffusion Ring-A-Bell Winobias
Metrics: CLIP similarity CLIP score FID top-1 accuracy attack success rate nudity reduction percentage gender reversal success rate
Baselines: UCE ESD Forget-Me-Not Concept Ablation SLD Stable Diffusion 2.0 MEMIT
Builds on: Wanda DiffPrune UCE ESD
Closest papers by meaning
- Unlearning Concepts in Diffusion Model via Concept Domain Correction and Concept Preserving Gradient (AAAI 2025 · similarity 0.92)
- Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Models (NeurIPS 2024 · similarity 0.91)
- Growth Inhibitors for Suppressing Inappropriate Image Concepts in Diffusion Models (ICLR 2025 · similarity 0.91)
- CURE: Concept Unlearning via Orthogonal Representation Editing in Diffusion Models (NeurIPS 2025 · similarity 0.91)
- MACE: Mass Concept Erasure in Diffusion Models (CVPR 2024 · similarity 0.90)
- Circumventing Concept Erasure Methods For Text-To-Image Generative Models (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.