Research map / Image generation / Text-to-image diffusion
One-dimensional Adapter to Rule Them All: Concepts Diffusion Models and Erasing Applications
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPM, a one-dimensional adapter injected into diffusion models to erase multiple concepts with Latent Anchoring fine-tuning and inference-time Facilitated Transport, transferable training-free across DM derivatives.
Problem
Concept erasure in text-to-image diffusion models via full or specification-based fine-tuning causes generation alterations and concept erosion on non-target concepts, especially with multiple erased concepts, and erased models are not transferable across derivative models.
Approach
Injects a lightweight rank-one-style adapter (erasing signal vector plus trainable regulator vector) into DM layers as a 'semi-permeable membrane'; Latent Anchoring samples distant concepts in the text embedding space to preserve them during fine-tuning, and Facilitated Transport gates each adapter's strength at inference by prompt-concept similarity.
Key results
- Multi-concept erasure (Snoopy, Mickey, Spongebob): SPM achieves FID of 33.19 on target-related evaluation vs. 137.23 (ESD) and 167.79 (SA), with FIDg 13.26 near the original 13.24.
- Nudity erasure on I2P reduces generated nudity by 85.3% relative to SD v1.4, compared with 69.5% (ConAbl) and 77.1% (SA) reductions as reported in the figure.
- SPM adds only about 0.0005x the parameter storage of tuning-based methods and reports 65.1% and 39.1% time reductions over ESD and ConAbl in the authors' cost model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SPM for Snoopy removal instead of ESD | SD v1.4 · Snoopy removal | CLIP Error Rate: 77.62 -> 20.12 | Hurt |
| Use SPM for Snoopy removal instead of ESD | SD v1.4 · Snoopy removal | FIDg: 13.68 -> 13.24 | Helped |
| Remove Snoopy and Mickey together rather than Snoopy alone | SPM on SD v1.4 · Snoopy and Mickey removal | FIDg: 13.24 -> 13.26 | No clear effect |
| Remove Snoopy, Mickey, and Spongebob together rather than Snoopy alone | SPM on SD v1.4 · Snoopy, Mickey, and Spongebob removal | FIDg: 13.24 -> 13.26 | No clear effect |
| Remove nudity using SPM | SD v1.4 · I2P nudity removal | change in detected exposed body parts: -85.3% | Helped |
| Transfer concept-specific SPMs to other models without model-specific re-tuning | SD v1.5, Chilloutmix, RealisticVision, and Dreamshaper-8 · Concept removal and non-target generation preservation | qualitative generation | Helped |
| Use SPM instead of ESD for 20 concepts across 5 models | 20 concepts, 5 models, 60 prompts · Erasing pipeline | time consumption: 70.25 -> 24.5 | Helped |
Surprising findings
- SD v2.x data-filtering reduces inappropriate generations by less than 30% relative to SD v1.4.
Limitations
- Concept erasure evaluated mainly on object, style, and nudity concepts; sensitivity to hyperparameters alpha, lambda, eta is deferred to appendix.
- Facilitated Transport relies on CLIP similarity and token overlap, which may miss semantically related but lexically different prompts.
Experimental setup
Datasets: I2P COCO-30k Caption
Benchmarks: I2P COCO-30k Caption
Metrics: CLIP Score CLIP Error Rate FID FIDg NudeNet exposed body part counts generation time
Baselines: ESD Concept Ablation Selective Amnesia SLD
Builds on: ESD Concept Ablation Selective Amnesia LoRA classifier-free guidance concept negation
Closest papers by meaning
- Precise, Fast, and Low-cost Concept Erasure in Value Space: Orthogonal Complement Matters (CVPR 2025 · similarity 0.90)
- Fantastic Targets for Concept Erasure in Diffusion Models and Where To Find Them (ICLR 2025 · similarity 0.90)
- All but One: Surgical Concept Erasing with Model Preservation in Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.89)
- When Are Concepts Erased From Diffusion Models? (NeurIPS 2025 · similarity 0.88)
- Memories of Forgotten Concepts (CVPR 2025 · similarity 0.88)
- EraseAnything: Enabling Concept Erasure in Rectified Flow Transformers (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.