Research map / Image generation / Text-to-image diffusion
MACE: Mass Concept Erasure in Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MACE, a finetuning framework that erases up to 100 concepts from Stable Diffusion v1.4 by combining closed-form cross-attention refinement, per-concept LoRA training with concept-focal importance sampling, and closed-form LoRA fusion.
Problem
Concept erasure in text-to-image diffusion models struggles to scale beyond a handful of concepts while balancing generality (blocking synonyms) and specificity (preserving unrelated concepts).
Approach
Removes residual concept information from co-existing prompt words via closed-form cross-attention key/value refinement, trains a LoRA per concept with attention-map suppression on timesteps weighted toward early-mid denoising steps (CFIS), and fuses LoRAs with a closed-form objective that avoids inter-concept interference.
Key results
- Erasing 100 celebrities: MACE reaches harmonic mean Hc 89.78 with Acce 4.31 and Accs 84.56, versus 46.72 for the naive ablation baseline configuration.
- On I2P explicit content, MACE yields 111 total NudeNet detections versus 743 for SD v1.4 and 123 for ESD-u, with MS-COCO FID 13.42.
- On erasing 100 artistic styles, MACE achieves Ha 5.99, the highest among compared methods (UCE 4.39, ESD-x 0.32).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MACE instead of ESD-x for single-object erasure | SD v1.4 · CIFAR-10, average across 10 classes | Ho (%): 76.91 -> 92.61 | Helped |
| Use MACE instead of ESD-u for explicit-content erasure | SD v1.4 · I2P | Detected explicit content (Total): 123 -> 111 | Helped |
| Use MACE for erasing 100 artistic styles | SD v1.4 · 100 artistic styles | Ha: 4.39 -> 5.99 | Helped |
| Use MACE rather than SD v1.4 for regular-content generation after explicit-content erasure | SD v1.4 · MS-COCO 30K | FID: 14.04 -> 13.42 | Helped |
| Use MACE rather than SD v1.4 for regular-content generation after explicit-content erasure | SD v1.4 · MS-COCO 30K | CLIP: 31.34 -> 29.41 | Hurt |
| Remove LoRA training, fusion, and CFIS, leaving closed-form refinement only (ablation) | MACE · 100 celebrities | Hc (%): 46.72 -> 89.78 | Hurt |
| Replace closed-form LoRA fusion with naive LoRA fusion and remove CFIS (ablation) | MACE · 100 celebrities | Accs (%): 32.16 -> 84.56 | Hurt |
| Remove CFIS while retaining closed-form LoRA fusion (ablation) | MACE · 100 celebrities | Hc (%): 70.21 -> 89.78 | Hurt |
| Remove CFIS while retaining closed-form LoRA fusion (ablation) | MACE · 100 celebrities | Acce (%): 18.70 -> 4.31 | Hurt |
Surprising findings
- Naive LoRA fusion severely hurts specificity: the ablation with naive fusion reaches Accs of only 32.16% on 100 celebrities.
- Retraining SD v2.1 on a curated dataset yields only a minor improvement in explicit content generation over SD v1.4 (586 vs 743 detections).
Limitations
- Harmonic mean declines as the number of erased concepts grows from 10 to 100, suggesting limits for thousands of concepts.
- Requires Grounded-SAM segmentation of training images to build masks for attention loss.
- Evaluated only on Stable Diffusion v1.4 with fixed evaluation classifiers (CLIP, GCD, NudeNet).
Experimental setup
Datasets: MS-COCO CIFAR-10 I2P Image Synthesis Style Studies Database
Benchmarks: CIFAR-10 object erasure celebrity erasure (GCD) I2P explicit content artistic style erasure MS-COCO 30K FID/CLIP
Metrics: Acc_e Acc_s Acc_g harmonic mean FID CLIP score NudeNet detected quantity CLIPe/CLIPs
Baselines: ESD-u ESD-x FMN SLD-M UCE AC
Builds on: ESD UCE LoRA Grounded-SAM
Closest papers by meaning
- Semantic Surgery: Zero-Shot Concept Erasure in Diffusion Models (NeurIPS 2025 · similarity 0.92)
- Localized Concept Erasure for Text-to-Image Diffusion Models Using Training-Free Gated Low-Rank Adaptation (CVPR 2025 · similarity 0.92)
- Concept Pinpoint Eraser for Text-to-image Diffusion Models via Residual Attention Gate (ICLR 2025 · similarity 0.92)
- Growth Inhibitors for Suppressing Inappropriate Image Concepts in Diffusion Models (ICLR 2025 · similarity 0.91)
- All but One: Surgical Concept Erasing with Model Preservation in Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.91)
- Precise, Fast, and Low-cost Concept Erasure in Value Space: Orthogonal Complement Matters (CVPR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.