Research map / Image generation / Text-to-image diffusion
Continuous Concepts Removal in Text-to-image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CCRT, a knowledge-distillation method for continuously removing concepts (artist styles, improper content, IP) from Stable Diffusion while preserving text-image alignment, using a genetic-algorithm-plus-LLM-fuzzing search for calibration prompts.
Problem
Existing concept-removal methods for text-to-image diffusion models degrade text-image alignment (entity forgetting) when concepts must be removed one after another in a continual fashion.
Approach
Distill from the frozen original Stable Diffusion as teacher while removing concepts in the student, with a calibration prompt set evolved by a genetic algorithm (crossover plus LLM-based mutation fuzzing) and ranked by a Misalignment Distance hardness metric to anchor the semantic space.
Key results
- Average RR-CLS 0.753 and RR-LLM 0.874 on continuous artistic style removal across four artists, versus 0.406/0.150 for the next best method (MACE).
- At an average removal rate of 0.87, CLIP score improves from 21.698 to 25.005 over the state of the art.
- Human study shows only CCRT balances concept removal, text-image alignment, other-concept preservation, and image quality; ESD removes concepts comparably but misaligns text.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CCRT instead of SD for continuous removal of eroticism and violence | Stable Diffusion v1.4 · Improper content removal | RR-CLS: 0.39 -> 0.99 | Helped |
| Use CCRT instead of SD for violence removal | Stable Diffusion v1.4 · Violence removal | RR-CLS: 0.51 -> 0.93 | Helped |
| Use CCRT instead of the strongest baseline | Stable Diffusion v1.4 · Continuous artistic style removal (average) | RR-CLS: 0.406 -> 0.753 | Helped |
| Use CCRT instead of the strongest baseline | Stable Diffusion v1.4 · Continuous artistic style removal (average) | RR-LLM: 0.154 -> 0.874 | Helped |
| Use CCRT instead of MACE | Stable Diffusion v1.4 · Van Gogh + Picasso + Monet + Cezanne removal | RR-LLM: 0.077 -> 0.918 | Helped |
| Use CCRT instead of ESD | Stable Diffusion v1.4 · Continuous artistic style removal | CLIP-Score: +2.23 in CLIP-Score | Helped |
| Use CCRT instead of ESD as removal progresses | Stable Diffusion v1.4 · Continuous artistic style removal | CLIP-Score: +5.01 in CLIP-Score | Helped |
| Use CCRT instead of ESD | Stable Diffusion v1.4 · Continuous artistic style removal | VQA-Score: +0.03 in VQA-Score | Helped |
| Use CCRT instead of ESD as removal progresses | Stable Diffusion v1.4 · Continuous artistic style removal | VQA-Score: +0.16 in VQA-Score | Helped |
| Use CCRT instead of the existing state-of-the-art | Stable Diffusion v1.4 · Continuous concept removal | CLIP-Score: 21.698 -> 25.005 | Helped |
Surprising findings
- ESD removes target concepts about as well as CCRT, but only because it damages the semantic structure, causing large text-image misalignment.
Limitations
- Evaluated only on Stable Diffusion v1.4 and a limited set of concepts (four artists, two improper content types, two IPs).
- Relies on GPT-4 for mutation fuzzing, prompt generation, and RR-LLM scoring, adding dependence on a proprietary model.
Experimental setup
Datasets: ImageNet class labels I2P
Benchmarks: I2P
Metrics: RR-CLS RR-LLM CLIP-Score VQA-Score human ranking
Baselines: ESD UCE MACE SPM
Builds on: ESD UCE knowledge distillation classifier-free guidance negative guidance
Closest papers by meaning
- Six-CD: Benchmarking Concept Removals for Text-to-image Diffusion Models (CVPR 2025 · similarity 0.89)
- Growth Inhibitors for Suppressing Inappropriate Image Concepts in Diffusion Models (ICLR 2025 · similarity 0.88)
- CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching (NeurIPS 2024 · similarity 0.88)
- Precise, Fast, and Low-cost Concept Erasure in Value Space: Orthogonal Complement Matters (CVPR 2025 · similarity 0.88)
- All but One: Surgical Concept Erasing with Model Preservation in Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.88)
- Semantic Surgery: Zero-Shot Concept Erasure in Diffusion Models (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.