Research map / Image generation / Text-to-image diffusion
Localized Concept Erasure for Text-to-Image Diffusion Models Using Training-Free Gated Low-Rank Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GLoCE, a training-free, closed-form low-rank projection with a learned-from-few-generations gate that erases localized target concepts (celebrities, explicit content, artistic styles) from text-to-image diffusion models while preserving other regions.
Problem
Fine-tuning-based concept erasure in text-to-image diffusion models degrades fidelity of remaining concepts when a target concept appears in a localized region of an image alongside other concepts; prior methods also remain vulnerable to adversarial regeneration attacks.
Approach
Injects low-rank projection matrices into diffusion layers, derived in closed form from PCA of a few generated images (inspired by LEACE concept scrubbing), plus a sigmoid gate on the principal-component energy of tokens that opens only for target concepts; no gradient training is performed.
Key results
- On 50 celebrity erasure with localized prompts, GLoCE reaches H_cc 96.97 with Acc_t 1.17 and Acc_r 95.17, versus MACE at 85.95 H_cc.
- On I2P explicit content, GLoCE detects only 8 nudity instances versus 111 for MACE and 743 for SD v1.4.
- Under Ring-A-Bell attack GLoCE achieves 0.00% attack success rate versus 4.26% for MACE; UnlearnDiff ASR 39.44% vs 65.46% for RECE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply GLoCE instead of MACE for localized erasure of 50 celebrities | SD v1.4 · Localized celebrities: 50 Celebrities Erased | Hcc: 85.95 -> 96.97 | Helped |
| Apply GLoCE instead of MACE for celebrity erasure | SD v1.4 · Celebrities erasure | target ACC: 3.29 -> 0.95 | Helped |
| Apply GLoCE instead of RECE for explicit-content erasure | SD v1.4 · I2P | detected explicit contents (total): 89 -> 8 | Helped |
| Apply GLoCE instead of MACE against Ring-A-Bell | SD v1.4 · I2P adversarial attacks: Ring-A-Bell | ASR: 4.26 -> 0.00 | Helped |
| Apply GLoCE instead of RECE against UnlearnDiff | SD v1.4 · I2P adversarial attacks: UnlearnDiff | ASR: 65.46 -> 39.44 | Helped |
| add target principal components Vtar (ablation) | GLoCE · Localized celebrities: 50 Celebrities Erased | target ACC: 2.17 -> 1.17 | Helped |
| add mapping principal components Vmap (ablation) | GLoCE · Localized celebrities: 50 Celebrities Erased | target ACC: 3.00 -> 1.17 | Helped |
| add mapping bias b* (ablation) | GLoCE · Localized celebrities: 50 Celebrities Erased | target ACC: 6.67 -> 1.17 | Helped |
| add gate s(X) (ablation) | GLoCE · Localized celebrities: 50 Celebrities Erased | remaining ACC: 74.83 -> 95.17 | Helped |
| increase η from 0.5 to 3.0 (ablation) | GLoCE · Localized celebrities: 50 Celebrities Erased | target ACC: 4.83 -> 0.83 | Helped |
| increase η from 0.5 to 3.0 (ablation) | GLoCE · Localized celebrities: 50 Celebrities Erased | remaining ACC: 98.17 -> 91.17 | Hurt |
| increase gate tolerance τ1 from 0.5 to 5.0 (ablation) | GLoCE · Localized celebrities: 50 Celebrities Erased | target ACC: 1.67 -> 6.50 | Hurt |
Surprising findings
- Removing the gate mechanism raised fidelity-related accuracy only partially, while removing the bias b* caused the largest efficacy drop, showing the mapping bias is central to erasure.
- Baselines like ESD-u and UCE achieve strong erasure of target celebrities but severely degrade remaining celebrities' fidelity on localized prompts.
Limitations
- Designed for localized erasure, but gate and anchor selection rely on predefined concept pools and hyperparameters (r1, r2, r3, eta, tau1).
- Evaluation depends on domain-specific classifiers (GIPHY Celebrity Detector, NudeNet) as accuracy proxies.
- Artistic style erasure is not localized, so the method is applied outside its primary design target.
Experimental setup
Datasets: I2P prompts COCO-30K GIPHY celebrity list (200 celebrities from MACE) Word2Vec characters (64) artistic styles (100 from MACE)
Benchmarks: localized celebrity erasure prompts 50 and 100 celebrity erasure I2P COCO-30K Ring-A-Bell UnlearnDiff 100 artistic styles erasure
Metrics: GCD accuracy (Acc_t, Acc_r) harmonic mean H_cc NudeNet detection count CLIP score FID KID attack success rate
Baselines: FMN ESD-x ESD-u UCE MACE RECE
Builds on: LEACE MACE UCE ESD LoRA
Closest papers by meaning
- All but One: Surgical Concept Erasing with Model Preservation in Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.92)
- Precise, Fast, and Low-cost Concept Erasure in Value Space: Orthogonal Complement Matters (CVPR 2025 · similarity 0.92)
- MACE: Mass Concept Erasure in Diffusion Models (CVPR 2024 · similarity 0.92)
- Semantic Surgery: Zero-Shot Concept Erasure in Diffusion Models (NeurIPS 2025 · similarity 0.91)
- Fine-Grained Erasure in Text-to-Image Diffusion-based Foundation Models (CVPR 2025 · similarity 0.91)
- Concept Pinpoint Eraser for Text-to-image Diffusion Models via Residual Attention Gate (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.