Research map / Image generation / Text-to-image diffusion
Can Protective Perturbation Safeguard Personal Data from Being Exploited by Stable Diffusion?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically evaluates adversarial protective perturbations (AdvDM, Anti-DreamBooth, Glaze, Improved-AdvDM) against Stable Diffusion fine-tuning under a practical threat model, and proposes GrIDPure, a grid-based iterative diffusion purification method that removes the perturbations.
Problem
Protective adversarial perturbations are proposed to prevent Stable Diffusion from learning personal faces and artistic styles, but their effectiveness under realistic conditions (varied fine-tuning methods, mixed protected/unprotected data, natural transformations, purification) is unclear.
Approach
Evaluate protective perturbations under a practical threat model where the exploiter can choose fine-tuning methods, mix protected and clean images, apply natural transformations, and purify images; then propose GrIDPure, which applies small-step SDEdit-based DiffPure over overlapping grid crops at high resolution and blends with the original to preserve structure.
Key results
- Under practical settings, protection often fails: e.g., on CelebA-HQ AdvDM with LoRA (w/ text encoder) yields FID 424.7 and CLIP 0.2316 versus clean 119.8/0.7378, but JPEG/Gaussian blur transformations recover quality (e.g., CLIP 0.617 with JPEG under EoT-free attack setting in Table 3).
- GrIDPure achieves SSIM 0.9199 and PSNR 30.60 on AdvDM-perturbed images versus 0.6378 SSIM and 22.24 PSNR for DiffPure (t=100).
- Stable Diffusion fine-tuned on GrIDPure-purified data generates images with CLIP 0.8526 (CelebA AdvDM) versus 0.7400 for DiffPure t=100, indicating purified images are learnable.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add AdvDM protective perturbations to training images | Stable Diffusion fine-tuned with LoRA and text encoder training · CelebA-HQ | FID: 119.8 -> 424.7 | Hurt |
| Add Anti-DreamBooth protective perturbations to training images | Stable Diffusion fine-tuned with LoRA and text encoder training · CelebA-HQ | FID: 119.8 -> 439.1 | Hurt |
| Add Improved-AdvDM protective perturbations to training images | Stable Diffusion fine-tuned with LoRA and text encoder training · CelebA-HQ | FID: 119.8 -> 100.5 | No clear effect |
| Add AdvDM protective perturbations to training images | Stable Diffusion fine-tuned with LoRA without text encoder training · CelebA-HQ | FID: 157.2 -> 226.5 | Hurt |
| Do not train the text encoder during LoRA fine-tuning (ablation) | Stable Diffusion · CelebA-HQ | FID: 119.8 -> 157.2 | Mixed |
| Purify AdvDM-protected images with GrIDPure before fine-tuning | Stable Diffusion · CelebA-HQ | CLIP-Score: 0.2316 -> 0.8526 | Helped |
| Purify Anti-DreamBooth-protected images with GrIDPure before fine-tuning | Stable Diffusion · CelebA-HQ | CLIP-Score: 0.2804 -> 0.7773 | Helped |
| Purify AdvDM-protected images with GrIDPure before fine-tuning | Stable Diffusion · WikiArt | CLIP-Score: 0.337 -> 0.8758 | Helped |
| Apply JPEG compression to AdvDM-protected images | Stable Diffusion fine-tuned with LoRA · CelebA-HQ | CLIP-Score: 0.232 -> 0.617 | Mixed |
| Apply Gaussian blur to AdvDM-protected images | Stable Diffusion fine-tuned with LoRA · CelebA-HQ | FID: 424.7 -> 119.2 | Helped |
| Optimize AdvDM perturbations with EoT (ablation) | AdvDM · CelebA-HQ, JPEG transformation | FID: 129.8 -> 137.1 | Mixed |
| Purify protected images using GrIDPure rather than DiffPure (ablation) | AdvDM-protected images · CelebA-HQ | SSIM: 0.6978 -> 0.9199 | Helped |
Surprising findings
- Textual Inversion, which only trains the text encoder, is the least robust to protective perturbations and yields nearly illegible images.
- Adaptive attacks that integrate DiffPure into perturbation optimization still fail to keep protection robust against DiffPure.
Limitations
- GrIDPure is slower than other purification methods due to grid cropping.
- Purification works less well on oil paintings because it can remove oil texture intertwined with perturbations.
- Evaluation focuses on AdvDM and Anti-DreamBooth as main perturbation methods; other methods evaluated only in some experiments.
Experimental setup
Datasets: CelebA-HQ WikiArt
Benchmarks: CelebA-HQ WikiArt
Metrics: FID CLIP-Score precision SSIM PSNR
Baselines: AdvDM Anti-DreamBooth Improved-AdvDM Glaze DiffPure
Builds on: DiffPure Anti-DreamBooth AdvDM SDEdit Expectation over Transformation
Closest papers by meaning
- Perturb a Model, Not an Image: Towards Robust Privacy Protection via Anti-Personalized Diffusion Models (NeurIPS 2025 · similarity 0.91)
- Prompt-Agnostic Adversarial Perturbation for Customized Diffusion Models (NeurIPS 2024 · similarity 0.90)
- Nearly Zero-Cost Protection Against Mimicry by Personalized Diffusion Models (CVPR 2025 · similarity 0.89)
- Anti-Diffusion: Preventing Abuse of Modifications of Diffusion-Based Models (AAAI 2025 · similarity 0.88)
- Adversarial Perturbations Cannot Reliably Protect Artists From Generative AI (ICLR 2025 · similarity 0.88)
- PID: Prompt-Independent Data Protection Against Latent Diffusion Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.