Research map / Image generation / Text-to-image diffusion
Boosting Alignment for Post-Unlearning Text-to-Image Generative Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes restricted-gradient unlearning (RG) plus LLM-driven data diversification (RGD) that removes target classes/concepts from diffusion models while preserving text-image alignment.
Problem
Approximate machine unlearning in text-to-image diffusion models removes harmful concepts or classes but degrades text-image alignment on retained concepts because forgetting and retaining objectives conflict and retain sets lack diversity.
Approach
Computes a restricted gradient, a update direction that gives monotonic decrease of both forgetting and remaining losses (equivalent to gradient surgery/PCGrad-style projection), and builds diverse forget/retain sets via LLM-generated prompts with concept words removed.
Key results
- On CIFAR-10 class-wise forgetting, RGD achieves UA 1.000 with RA 0.771 and FID 6.539, versus GradDiff at RA 0.734 and FID 14.09.
- On nudity removal, RGD reduces exposed body parts to zero versus 598 for SD, 48 for ESD-u, and 3 for SalUn, with test-set CLIP alignment 0.350 vs SD 0.352.
- In SD v3 (2B) nudity removal, RGD gets alignment 0.362/0.370 vs SD v3 0.364/0.371 while SalUn collapses to 0.079/0.088.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the restricted gradient instead of direct gradient aggregation | EDM class-wise unlearning · CIFAR-10 class-wise forgetting | FID: 14.09$_{±2.531}$ -> 9.813$_{±1.863}$ | Helped |
| add diverse remaining data (ablation) | RG on EDM · CIFAR-10 class-wise forgetting | FID: 9.813$_{±1.863}$ -> 6.539$_{±0.994}$ | Helped |
| add diverse remaining data (ablation) | RG on EDM · CIFAR-10 class-wise forgetting | RA: 0.752$_{±0.018}$ -> 0.771$_{±0.016}$ | Helped |
| add the restricted gradient (ablation) | GradDiffD · Nudity removal, Stable Diffusion | CLIP alignment, Dr test: 0.348 (0.004) -> 0.350 (0.002) | Helped |
| add data diversification (ablation) | RG · Nudity removal, Stable Diffusion | CLIP alignment, Dr test: 0.320 (0.032) -> 0.351 (0.001) | Helped |
| use balanced sampling across all classes instead of samples from only two similar classes (ablation) | RGD · CIFAR-10 diversity-controlled experiment | RA: 0.239±0.071 -> 0.675±0.019 | Helped |
| use balanced sampling across all classes instead of samples from only two similar classes (ablation) | RGD · CIFAR-10 diversity-controlled experiment | FID: 94.259±28.217 -> 10.456±1.976 | Helped |
| use balanced sampling across all classes instead of samples from only two similar classes (ablation) | RG · CIFAR-10 diversity-controlled experiment | RA: 0.205±0.138 -> 0.463±0.059 | Helped |
| use balanced sampling across all classes instead of samples from only two similar classes (ablation) | RG · CIFAR-10 diversity-controlled experiment | FID: 131.247±46.049 -> 47.797±7.231 | Helped |
| use balanced sampling across all classes instead of samples from only two similar classes (ablation) | GradDiff · CIFAR-10 diversity-controlled experiment | RA: 0.106±0.086 -> 0.201±0.043 | Helped |
| use balanced sampling across all classes instead of samples from only two similar classes (ablation) | GradDiff · CIFAR-10 diversity-controlled experiment | FID: 156.021±31.901 -> 95.287±16.279 | Helped |
| add diverse remaining data (ablation) | RG · Nudity removal, Stable Diffusion | CLIP alignment gap, Dr test: 0.032 -> 0.001 | Helped |
Surprising findings
- SalUn achieves strong nudity suppression but its alignment score drops sharply, indicating overfitting to the retain set and loss of output diversity.
- Gradient surgery style projection, which yields a restricted gradient, is shown to be equivalent to optimizing a constrained objective that guarantees monotonic decrease of both losses.
Limitations
- Hyperparameter sensitivity (lambda and alpha) remains a challenge.
- Retain set sampling strategy for Stable Diffusion is not deeply studied.
- Theoretical guarantee relies on first-order Taylor approximation with small learning rates.
Experimental setup
Datasets: CIFAR-10 I2P prompts LLM-generated prompts (nudity, Van Gogh artist removal)
Benchmarks: CIFAR-10 I2P nudity prompts Artist style removal prompts
Metrics: unlearning accuracy remaining accuracy FID NudeNet exposed-body-part detections CLIP alignment score
Baselines: Finetune GradDiff SalUn ESD ESD-u ESD-x SD
Builds on: Erased Stable Diffusion (ESD) SalUn gradient surgery EDM
Closest papers by meaning
- Machine Unlearning for Image-to-Image Generative Models (ICLR 2024 · similarity 0.92)
- The Illusion of Unlearning: The Unstable Nature of Machine Unlearning in Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.90)
- Fine-Grained Erasure in Text-to-Image Diffusion-based Foundation Models (CVPR 2025 · similarity 0.90)
- Enabling Preference-driven Unlearning in Few-step Distilled Text-to-Image Diffusion Models (NeurIPS 2026 · similarity 0.90)
- UnlearnCanvas: Stylized Image Dataset for Enhanced Machine Unlearning Evaluation in Diffusion Models (NeurIPS 2024 · similarity 0.90)
- Unlearning Concepts in Diffusion Model via Concept Domain Correction and Concept Preserving Gradient (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.