Research map / Image generation / Fast diffusion sampling

Data Unlearning in Diffusion Models

ICLR 2025 · Image generation › Fast diffusion sampling · approach: Diffusion theory and design space · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SISS, an importance-sampled, subtracted-score loss that interpolates between naive deletion fine-tuning and NegGrad to unlearn specific training datapoints from diffusion models while preserving quality.

Problem

Efficiently unlearning specific training datapoints (data unlearning) from diffusion models without retraining, while preserving generation quality; existing concept-unlearning methods need anchor prompts/classes that are unavailable in this setting.

Approach

Combine naive deletion (fine-tune on retained data) with NegGrad (gradient ascent on the unlearning set) into one loss via importance sampling with a defensive mixture distribution, so one forward pass suffices; a superfactor scales the NegGrad term and is set so its gradient norm is about 10% of the retain term's.

Key results

• On CelebA-HQ, SISS (λ=0.5) keeps FID at 25.1 (pretrained 30.3) while cutting SSCD from 0.87 to 0.36; NegGrad and EraseDiff raise FID to 334.3 and 117.8.
• On MNIST T-Shirt, SISS (λ=0.5) and SISS (No IS) reduce T-shirt generation rate from 0.74% to 0%, with IS of 9.2 vs 9.6 for the pretrained model.
• On Stable Diffusion, SISS mitigated memorization on nearly 90% of tested prompts while preserving CLIP-IQA quality.

Measured effects reported in the paper

Surprising findings

• Naive deletion and SISS (λ=0) behave similarly to each other, while λ=0.5 unlearns with preserved quality, despite the lemma stating gradient estimators are equal in expectation for all λ.

Limitations

• Stable Diffusion experiments require manually modified 'partially-memorized' prompts, a step that is hard to automate.
• SISS with λ=0 is numerically unstable on Stable Diffusion, so it was excluded from that experiment.
• Legal sufficiency is uncertain since the copyrighted data is part of the unlearning process itself.

Experimental setup

Datasets: CelebA-HQ MNIST Fashion-MNIST LAION (memorized images)

Benchmarks: CelebA-HQ MNIST T-Shirt Stable Diffusion memorization prompts

Metrics: FID Inception Score CLIP-IQA SSCD similarity negative log-likelihood fraction of T-shirts generated memorized sample count

Baselines: Naive deletion NegGrad EraseDiff Retrain SISS (No IS)

Builds on: NegGrad DDPM defensive mixture distribution EraseDiff

Closest papers by meaning

• The Illusion of Unlearning: The Unstable Nature of Machine Unlearning in Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.91)
• Training Data Attribution in Diffusion Models via Mirrored Unlearning and Noise-Consistent Skew (NeurIPS 2026 · similarity 0.90)
• Machine Unlearning under Overparameterization (NeurIPS 2025 · similarity 0.90)
• Enabling Preference-driven Unlearning in Few-step Distilled Text-to-Image Diffusion Models (NeurIPS 2026 · similarity 0.89)
• Score Forgetting Distillation: A Swift, Data-Free Method for Machine Unlearning in Diffusion Models (ICLR 2025 · similarity 0.89)
• The Emergence of Reproducibility and Consistency in Diffusion Models (ICML 2024 · similarity 0.89)