Research map / Image generation / Text-to-image diffusion
UnlearnCanvas: Stylized Image Dataset for Enhanced Machine Unlearning Evaluation in Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces UnlearnCanvas, a high-resolution stylized image dataset (60 styles x 20 objects) with a standardized automated evaluation framework, and benchmarks 9 diffusion-model unlearning methods on style/object unlearning, adversarial robustness, style-object combinations, and sequential unlearning.
Problem
Existing evaluations of diffusion model unlearning rely on limited, manually chosen targets, neglect retainability (especially cross-domain), and struggle to classify generated artistic styles precisely, so they give incomplete or biased assessments.
Approach
A stylized dataset with dual style and object labels (60 styles x 20 objects, stylized from shared seed images) lets unlearning be scored by classifier accuracy on in-domain and cross-domain prompts, so that unlearning effectiveness and retainability can be measured automatically and precisely.
Key results
- ESD achieves 98.58% style UA but only 80.97% style IRA and 93.96% CRA, while UCE reaches similar UA (98.40%) with much lower retainability (IRA 60.22%, CRA 47.71%), showing UA alone is misleading.
- Under UnlearnDiffAtk adversarial prompts, all methods drop below 60% UA; UCE falls from over 90% to below 50%.
- Sequential unlearning shows a catastrophic retaining failure for UCE, with RA dropping from 81.42% to 29.38% after the second request.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate retainability (IRA/CRA) alongside UA rather than UA alone | DM unlearning methods (ESD vs UCE) · UnlearnCanvas style unlearning | UA vs IRA/CRA | No clear effect |
| Cross-domain retainability is harder to keep than in-domain | DM unlearning methods (UCE) · UnlearnCanvas object unlearning | IRA vs CRA: CRA 34.67% vs IRA 39.35% | Hurt |
| Style-object combination unlearning (finer scale) versus single style/object unlearning | DM unlearning methods · Style-object combinations | UA: over 20% lower UA than single-target settings | Hurt |
| Adversarial prompts (UnlearnDiffAtk) applied to unlearned models | DM unlearning methods (UCE) · Adversarial robustness | UA under attack: UA drops from over 90% to below 50% | Hurt |
| Sequential unlearning of 6 styles | ESD and UCE · Sequential unlearning | retain accuracy (RA): UCE RA drops from 81.42% to 29.38% after second request | Hurt |
Surprising findings
- UCE achieves over 90% UA in normal settings but falls below 50% under adversarial prompts, so normal-condition UA does not imply robustness.
- Cross-domain retainability (CRA) is consistently harder to preserve than in-domain retainability (IRA) across methods.
Limitations
- Benchmark focuses on Stable Diffusion models and text-to-image generation only.
- Error bars omitted; results averaged over many unlearning cases with reported variance below 1%.
- Style-object combination evaluation uses 50 randomly selected of 1200 possible combinations.
Experimental setup
Datasets: UnlearnCanvas WikiArt Pexels seed photos I2P
Benchmarks: UnlearnCanvas
Metrics: UA IRA CRA FID Rob. SC OC UP time memory
Baselines: ESD FMN UCE CA SalUn SEOT SPM EDiff SHS
Builds on: Stable Diffusion ESD UnlearnDiffAtk ViT
Closest papers by meaning
- The Illusion of Unlearning: The Unstable Nature of Machine Unlearning in Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.93)
- Machine Unlearning for Image-to-Image Generative Models (ICLR 2024 · similarity 0.90)
- Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Models (NeurIPS 2024 · similarity 0.90)
- Unlearning Concepts in Diffusion Model via Concept Domain Correction and Concept Preserving Gradient (AAAI 2025 · similarity 0.90)
- Boosting Alignment for Post-Unlearning Text-to-Image Generative Models (NeurIPS 2024 · similarity 0.90)
- Score Forgetting Distillation: A Swift, Data-Free Method for Machine Unlearning in Diffusion Models (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.