Research map / Image generation / Editing and style transfer
Anti-Diffusion: Preventing Abuse of Modifications of Diffusion-Based Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Anti-Diffusion, an adversarial-noise privacy defense against both tuning (DreamBooth/LoRA) and editing (MasaCtrl/DiffEdit) diffusion methods, using prompt tuning and a semantic disturbance loss, plus a new Defense-Edit dataset.
Problem
Images published online can be abused by diffusion-based personalization (tuning) and editing methods; prior defenses like Anti-DreamBooth depend on manually chosen prompts and ignore editing methods.
Approach
Adds adversarial noise via PGD while jointly optimizing a learnable text embedding (prompt tuning) to match the image, using a UNet reverse loss and a semantic disturbance loss that pushes cross-attention maps toward zero, and alternately updates the UNet on protected samples to simulate malicious training.
Key results
- On VGGFace2 DreamBooth, Anti-Diffusion reaches FDFR 0.62 and ISM 0.15 versus 0.60 and 0.24 for Anti-DB, with PSNR 35.91.
- Ablation on DreamBooth shows PT and SDL are complementary: the full model reaches FDFR 0.62 vs 0.50 for the baseline.
- On Defense-Edit, Anti-Diffusion gives the lowest CLIP Score for MasaCtrl (26.44) and DiffEdit (25.25) among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Anti-Diffusion instead of Anti-DB | DreamBooth defense on VGGFace2 · VGGFace2 DreamBooth | FDFR: 0.60 -> 0.62 | Helped |
| Use Anti-Diffusion instead of Anti-DB | DreamBooth defense on CelebA-HQ · CelebA-HQ DreamBooth | ISM: 0.41 -> 0.26 | Helped |
| Use Anti-Diffusion instead of Anti-DB | LoRA defense on VGGFace2 · VGGFace2 LoRA | BRISQUE: 37.47 -> 39.26 | Helped |
| Use Anti-Diffusion instead of Anti-DB | MasaCtrl image-editing defense · Defense-Edit MasaCtrl | CLIP Score: 27.42 -> 26.44 | Helped |
| Use Anti-Diffusion instead of Anti-DB | DiffEdit image-editing defense · Defense-Edit DiffEdit | CLIP Score: 26.69 -> 25.25 | Helped |
| Add PT to the fixed-prompt method (ablation) | Anti-Diffusion · DreamBooth | FDFR: 0.50 -> 0.52 | Helped |
| Add SDL to the fixed-prompt method (ablation) | Anti-Diffusion · DreamBooth | FDFR: 0.50 -> 0.53 | Helped |
| Combine PT and SDL rather than use neither (ablation) | Anti-Diffusion · DreamBooth | FDFR: 0.50 -> 0.62 | Helped |
| Combine PT and SDL rather than use neither (ablation) | Anti-Diffusion · DreamBooth | ISM: 0.22 -> 0.15 | Helped |
| Replace the zero SDL attention target with a random noise map (ablation) | Anti-Diffusion · DreamBooth | FDFR: 0.62 -> 0.58 | Hurt |
| Replace the zero SDL attention target with a diagonal matrix (ablation) | Anti-Diffusion · DreamBooth | FDFR: 0.62 -> 0.59 | Hurt |
| Apply Anti-Diffusion using SD v2.1 to DreamBooth trained with SD v1.5 rather than matching versions | Anti-Diffusion · VGGFace2 DreamBooth, defense v2.1 and test v1.5 | FDFR: 0.06 -> 0.89 | Helped |
Surprising findings
- Protection transfers across SD versions: noise crafted on v2.1 still yields FDFR 0.89 when attacking v1.5 DreamBooth.
Limitations
- Defense-Edit is small, with only 50 image-prompt pairs.
- Evaluation relies on a noise budget of 0.05 and a fixed set of metrics and prompts.
Experimental setup
Datasets: VGGFace2 CelebA-HQ TEdBench
Benchmarks: VGGFace2 CelebA-HQ Defense-Edit
Metrics: PSNR FDFR ISM SER-FQA BRISQUE FID NIQE CLIP Score
Baselines: Photo Guard MIST PID Anti-DreamBooth
Builds on: Anti-DreamBooth Projected Gradient Descent Latent Diffusion Models
Closest papers by meaning
- DiffusionGuard: A Robust Defense Against Malicious Diffusion-based Image Editing (ICLR 2025 · similarity 0.93)
- Toward effective protection against diffusion-based mimicry through score distillation (ICLR 2024 · similarity 0.90)
- Distraction is All You Need: Memory-Efficient Image Immunization against Diffusion-Based Image Editing (CVPR 2024 · similarity 0.90)
- Towards Safe Concept Transfer of Multi-Modal Diffusion via Causal Representation Editing (NeurIPS 2024 · similarity 0.89)
- PID: Prompt-Independent Data Protection Against Latent Diffusion Models (ICML 2024 · similarity 0.89)
- Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.