Research map / Image generation / Text-to-image diffusion
Concept Replacer: Replacing Sensitive Concepts in Diffusion Models via Precision Localization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Concept Replacer, which fine-tunes a Stable Diffusion U-Net copy as a few-shot concept localizer and uses a training-free Dual Prompts Cross-Attention to replace a sensitive concept only inside the localized region.
Problem
Existing concept removal methods for diffusion models act globally during generation and alter non-target regions, causing inconsistency with the original model's outputs.
Approach
Localize the target concept during denoising with a few-shot fine-tuned U-Net-based localizer built from cross- and self-attention maps, then apply a dual-prompt cross-attention that uses a replacing prompt only inside the mask and the original prompt elsewhere.
Key results
- Localization: average mIoU 78.1 on CelebAMask-HQ with 10 samples (vs SLiMe 75.7) and 70.2 with 1 sample (vs SLiMe 69.6); 69.29 average on Pascal-Car 10-shot vs SLiMe 68.3.
- Non-target consistency for nudity replacement: MSE 45.86, PSNR 34.18, SSIM 0.828 vs ESD-u (MSE 99.71, PSNR 28.15) and SLD (MSE 104.81).
- COCO 30K FID 15.15 and CLIP 30.67, close to original SD (14.50 / 31.32).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the Concept Replacer localizer instead of SLiMe | Stable Diffusion concept localization · CelebAMask-HQ 10-shot | mIoU: 78.0 -> 78.1 | Helped |
| Use the Concept Replacer localizer instead of SLiMe | Stable Diffusion concept localization · CelebAMask-HQ 1-shot | mIoU: 69.6 -> 70.2 | Helped |
| Use the Concept Replacer localizer instead of SLiMe | Stable Diffusion concept localization · Pascal-Car 10-shot | mIoU: 68.3 -> 69.29 | Helped |
| Use the Concept Replacer localizer instead of SLiMe | Stable Diffusion concept localization · Pascal-Car 1-shot | mIoU: 61.6 -> 64.3 | Helped |
| Apply Concept Replacer rather than Stable Diffusion | Stable Diffusion image generation · COCO 30K | CLIP: 31.32 -> 30.67 | No clear effect |
| Use localized concept replacement rather than ESD-u | Stable Diffusion nudity replacement · Nudity replacement non-target regions | MSE: 99.7132 -> 45.8641 | Helped |
| Use localized concept replacement rather than ESD-u | Stable Diffusion nudity replacement · Nudity replacement non-target regions | PSNR: 28.1489 -> 34.1830 | Helped |
| Use localized concept replacement rather than ESD-u | Stable Diffusion nudity replacement · Nudity replacement non-target regions | SSIM: 0.4816 -> 0.8284 | Helped |
Limitations
- Localizer mIoU on some classes (e.g., Pascal-Car 1-shot, CelebA eyebrow) is lower than baselines; 1-shot average on CelebA is only marginally above SLiMe.
- Evaluation of concept removal focuses only on nudity via NudeNet; other unsafe concepts are shown qualitatively.
- Non-target consistency is measured against predicted masks rather than ground-truth masks.
Experimental setup
Datasets: CelebAMask-HQ Pascal-Car I2P COCO
Benchmarks: CelebAMask-HQ Pascal-Car I2P COCO 30K
Metrics: mIoU NudeNet detection reduction FID-30k CLIP score MSE PSNR SSIM
Baselines: ReGAN SegDDPM SegGPT SLiMe SLD ESD-u Stable Diffusion v2.1
Builds on: SLiMe Prompt-to-Prompt DiffSegmenter Stable Diffusion ESD
Closest papers by meaning
- Unlearning Concepts in Diffusion Model via Concept Domain Correction and Concept Preserving Gradient (AAAI 2025 · similarity 0.90)
- Localized Concept Erasure for Text-to-Image Diffusion Models Using Training-Free Gated Low-Rank Adaptation (CVPR 2025 · similarity 0.89)
- CURE: Concept Unlearning via Orthogonal Representation Editing in Diffusion Models (NeurIPS 2025 · similarity 0.88)
- Easing Concept Bleeding in Diffusion via Entity Localization and Anchoring (ICML 2024 · similarity 0.88)
- All but One: Surgical Concept Erasing with Model Preservation in Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.88)
- Semantic Surgery: Zero-Shot Concept Erasure in Diffusion Models (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.