Research map / Image generation / Text-to-image diffusion
DuMo: Dual Encoder Modulation Network for Precise Concept Erasure
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DuMo, a frozen-U-Net concept-erasure method for Stable Diffusion that adds a copied-encoder eraser on skip connections plus timestep-layer modulation factors to erase NSFW, cartoon, and artist concepts while preserving non-target generation.
Problem
Concept erasure in text-to-image diffusion models (NSFW content, copyrighted cartoon characters and artist styles) typically alters backbone U-Net features, which damages the structure and generative ability for non-target concepts.
Approach
Keep the pretrained U-Net frozen and intervene only on skip-connection features via an Eraser with Prior Knowledge (EPR) module (a copied encoder with zero-convolution outputs), then calibrate its outputs with timestep- and layer-specific modulation factors (TLMO) to balance erasure and preservation.
Key results
- Explicit content erasure: 34 nude body parts detected by NudeNet vs 41 for the next best (SDD) and 646 for SD v1.4.
- Cartoon concept removal: average LPIPS_da of 0.428 vs SOTA 0.315 (as stated in abstract).
- Artistic style erasure: average LPIPS_da of 0.387 vs SOTA 0.299 (as stated in abstract).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DuMo for explicit content erasure instead of SDD | SD v1.4 · I2P | total nude body parts: 41 -> 34 | Helped |
| Use DuMo instead of UCE | SD v1.4 · COCO-30k | CS: 31.26 -> 30.87 | Hurt |
| Use DuMo instead of UCE | SD v1.4 · COCO-30k | FID: 1.82 -> 2.06 | Hurt |
| Use DuMo instead of SPM | SD v1.4 · Cartoon concept removal: Snoopy | LPIPS_da: 0.329 -> 0.425 | Helped |
| Use DuMo instead of SPM | SD v1.4 · Cartoon concept removal: Snoopy and Mickey | LPIPS_da: 0.316 -> 0.416 | Helped |
| Use DuMo instead of SPM | SD v1.4 · Cartoon concept removal: Snoopy, Mickey and Spongebob | LPIPS_da: 0.302 -> 0.444 | Helped |
| Use DuMo instead of SPM | SD v1.4 · Artistic style erasure: Van Gogh | LPIPS_da: 0.245 -> 0.358 | Helped |
| Use DuMo instead of SPM | SD v1.4 · Artistic style erasure: Rembrandt | LPIPS_da: 0.352 -> 0.415 | Helped |
| Add timestep modulation (ablation) | EPR for Van Gogh erasure · Van Gogh | LPIPS_da: 0.184 -> 0.318 | Helped |
| Add layer modulation (ablation) | EPR for Van Gogh erasure · Van Gogh | LPIPS_da: 0.184 -> 0.358 | Helped |
| Add EPR instead of directly fine-tuning encoder cross-attention layers (ablation) | SD v1.4 for Van Gogh erasure · Van Gogh | LPIPS_u: 0.24 -> 0.033 | Helped |
| Add TLMO to EPR (ablation) | SD v1.4 for Van Gogh erasure · Van Gogh | LPIPS_da: 0.426 -> 0.358 | Hurt |
Surprising findings
- Authors report the method preserves non-target concepts better without any preservation loss in explicit content erasure.
- Layer group 4 of the EPR module has zero scale for Van Gogh erasure, indicating some layers contribute negligibly to erasure.
Limitations
- Evaluation limited to SD v1.4 for main experiments and a few concepts.
- Modulation factors are tuned per concept and require a second fine-tuning stage.
Experimental setup
Datasets: I2P COCO-30K
Benchmarks: I2P COCO-30K NudeNet
Metrics: NudeNet detected nude parts CLIP Score FID LPIPS_e LPIPS_u LPIPS_da
Baselines: ESD UCE SLD-Med SA CA SDD RECE MACE SPM
Builds on: ControlNet ESD FreeU
Closest papers by meaning
- Semantic Surgery: Zero-Shot Concept Erasure in Diffusion Models (NeurIPS 2025 · similarity 0.88)
- Precise, Fast, and Low-cost Concept Erasure in Value Space: Orthogonal Complement Matters (CVPR 2025 · similarity 0.88)
- MACE: Mass Concept Erasure in Diffusion Models (CVPR 2024 · similarity 0.88)
- Circumventing Concept Erasure Methods For Text-To-Image Generative Models (ICLR 2024 · similarity 0.88)
- Concept Pinpoint Eraser for Text-to-image Diffusion Models via Residual Attention Gate (ICLR 2025 · similarity 0.87)
- Localized Concept Erasure for Text-to-Image Diffusion Models Using Training-Free Gated Low-Rank Adaptation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.