Research map / Image generation / Text-to-image diffusion
Efficient Fine-Tuning and Concept Suppression for Pruned Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a bilevel optimization framework that jointly fine-tunes a pruned Stable Diffusion 2.1 via distillation while unlearning undesired concepts (artist styles, NSFW content), outperforming two-stage distill-then-unlearn pipelines.
Problem
Pruned diffusion models recovered via knowledge distillation inherit undesirable concepts (copyrighted artist styles, NSFW content) from the base model, and sequentially fine-tuning then unlearning is inefficient and suboptimal.
Approach
Reformulates pruned-model fine-tuning as a bilevel problem where the lower level performs diffusion and distillation fine-tuning and the upper level performs concept unlearning (ESD-style negative guidance), solved with a first-order penalty-based double-loop method so that unlearning and restoration are optimized jointly.
Key results
- Bilevel method reduces artist-style CLIP similarity to 26.28 vs 30.78 for Distilled+ESD, with CSD 39.04 vs 61.45 and FID 22.24 vs 30.38.
- On COCO, bilevel achieves CLIP 29.19 and FID 22.24, better than two-stage baselines with ESD, UCE, and ConceptPrune.
- On NSFW removal, Ours (ESD) reaches 94.32 on MMA and 91.60 on Ring-a-Bell versus baselines, with FID 26.80 vs 29.56-41.55 for baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace two-stage distillation then ESD with joint bilevel fine-tuning and concept removal | pruned Stable Diffusion 2.1 · Artist Erasure | CSD score: 61.45 -> 39.04 | Helped |
| Replace two-stage distillation then ESD with joint bilevel fine-tuning and concept removal | pruned Stable Diffusion 2.1 · Artist Erasure | CP score: 84.0 -> 97.6 | Helped |
| Replace two-stage distillation then ESD with joint bilevel fine-tuning and concept removal | pruned Stable Diffusion 2.1 · MS-COCO-2017 validation | FID: 30.38 -> 22.24 | Helped |
| Replace two-stage distillation then ESD with joint bilevel fine-tuning and concept removal | pruned Stable Diffusion 2.1 · Artist Erasure | CLIP similarity: 30.78 -> 26.28 | Helped |
| Use bilevel ESD rather than two-stage ESD | pruned Stable Diffusion 2.1 · MMA adversarial NSFW prompts | NSFW removal: 93.70 -> 91.60 | Hurt |
| Use bilevel ESD rather than two-stage ESD | pruned Stable Diffusion 2.1 · Ring-a-Bell adversarial NSFW prompts | NSFW removal: 77.27 -> 94.32 | Helped |
| Add distillation during fine-tuning of the pruned model (ablation) | APTP-pruned Stable Diffusion 2.1 · MS-COCO-2017 validation | FID | Helped |
| Use pruning instead of random initialization (ablation) | small diffusion model with distillation · MS-COCO-2017 fine-tuning | convergence speed | Helped |
| Use standard two-stage fine-tuning followed by concept unlearning instead of bilevel optimization | pruned Stable Diffusion 2.1 · Artist style removal | generation quality | Hurt |
| Use bilevel ESD instead of two-stage ESD | pruned Stable Diffusion 2.1 · MS-COCO-2017 validation | FID: 32.47 -> 26.80 | Helped |
Surprising findings
- Standard distillation-based fine-tuning of a pruned model retains the base model's artist styles and NSFW content even when such content is absent from the fine-tuning data.
Limitations
- Only ESD is used as the upper-level unlearning method in main experiments; stronger unlearning methods are only explored in the appendix.
- Continual unlearning of multiple concepts is left to future work.
- Evaluated on a single base model (Stable Diffusion 2.1) with one pruning method (APTP).
Experimental setup
Datasets: MS-COCO-2017 I2P Ring-A-Bell MMA
Benchmarks: MS-COCO-2017 validation (FID, CLIP) artist style erasure (Van Gogh, Picasso, Monet) I2P NSFW prompts Ring-A-Bell MMA
Metrics: CLIP similarity CP score CSD score FID NudeNet detection percentage
Baselines: Stable Diffusion 2.1 Distilled Model + ESD Distilled Model + UCE Distilled Model + ConceptPrune Distilled Model (Eq. 8)
Builds on: APTP ESD UCE ConceptPrune BK-SDM
Closest papers by meaning
- A First-order Generative Bilevel Optimization Framework for Diffusion Models (ICML 2025 · similarity 0.90)
- Diffusion Tuning: Transferring Diffusion Models via Chain of Forgetting (NeurIPS 2024 · similarity 0.89)
- DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecture (CVPR 2025 · similarity 0.88)
- Enabling Preference-driven Unlearning in Few-step Distilled Text-to-Image Diffusion Models (NeurIPS 2026 · similarity 0.88)
- One-Step Offline Distillation of Diffusion-based Models via Koopman Modeling (NeurIPS 2025 · similarity 0.87)
- Attention-Driven Training-Free Efficiency Enhancement of Diffusion Models (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.