Research map / Image generation / Text-to-image diffusion

Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Models

NeurIPS 2024 · Image generation › Text-to-image diffusion · approach: Concept erasure in diffusion models · 19 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Integrates adversarial training into diffusion-model concept erasure (AdvUnlearn), robustifying the text encoder with a utility-retaining regularizer so unlearned models resist adversarial prompt attacks while keeping image quality.

Problem

Concept-erased (unlearned) text-to-image diffusion models remain vulnerable to adversarial prompt attacks that regenerate erased concepts such as nudity, objects, and art styles.

Approach

Formulates concept erasure as bi-level adversarial training (ESD unlearning as upper level, adversarial soft-prompt generation as lower level), adds a retain-prompt regularizer to preserve utility, and optimizes the text encoder instead of the UNet so the robust encoder can be plugged into other diffusion models.

Key results

• On nudity unlearning, AdvUnlearn reduces ASR under UnlearnDiffAtk from 100% (SD v1.4) to 21.13% with FID 19.34, versus ESD at 73.24% ASR and FID 18.18.
• For Van Gogh style unlearning, AdvUnlearn achieves 2% ASR with FID 16.96, compared with 36% for the next best method ESD.
• The text encoder trained by AdvUnlearn on SD v1.4 transfers to SD v1.5, lowering ASR from 95.74% to 16.20% without additional fine-tuning.

Measured effects reported in the paper

Surprising findings

• Naive adversarial training of ESD (AT-ESD) cuts ASR but raises FID from 18.18 to 26.48, showing a significant utility loss.
• Optimizing the text encoder with plain ESD gives very low ASR (3.52%) but FID rises to 59.10, whereas AdvUnlearn on the text encoder keeps FID at 19.34.

Limitations

• Adversarial training with K-step attacks costs roughly K times more computation; the fast FGSM variant loses robustness.
• The utility-retaining regularization adds extra computation per iteration.
• Nudity unlearning still leaves ASR around 21%, higher than SalUn's 11.27%.

Experimental setup

Datasets: I2P COCO ImageNet Imagenette WikiArt

Benchmarks: I2P COCO Imagenette WikiArt

Metrics: ASR FID CLIP score

Baselines: ESD FMN AC UCE SalUn ScissorHands EraseDiff SPM

Builds on: ESD adversarial training UnlearnDiffAtk TRADES fast adversarial training

Closest papers by meaning

• Erasing Undesirable Concepts in Diffusion Models with Adversarial Preservation (NeurIPS 2024 · similarity 0.92)
• ConceptPrune: Concept Editing in Diffusion Models via Skilled Neuron Pruning (ICLR 2025 · similarity 0.91)
• Unlearning Concepts in Diffusion Model via Concept Domain Correction and Concept Preserving Gradient (AAAI 2025 · similarity 0.91)
• STEREO: A Two-Stage Framework for Adversarially Robust Concept Erasing from Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.91)
• Towards Robust Knowledge Unlearning: An Adversarial Framework for Assessing and Improving Unlearning Robustness in Large Language Models (AAAI 2025 · similarity 0.91)
• Direct Unlearning Optimization for Robust and Safe Text-to-Image Models (NeurIPS 2024 · similarity 0.90)