Research map / Image generation / Editing and style transfer
DiffusionGuard: A Robust Defense Against Malicious Diffusion-based Image Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiffusionGuard, an adversarial-noise defense against mask-based diffusion inpainting edits that perturbs the early denoising step with a noise-norm maximization loss and uses mask augmentation for mask robustness, plus the InpaintGuardBench benchmark.
Problem
Existing adversarial protections against diffusion image editing fail when adversaries use different, freely chosen masks with inpainting models, so protected images can still be edited maliciously.
Approach
Inpainting models reveal fine details already at the first denoising step, so the method maximizes the l2 norm of the initial predicted noise (one forward pass) under an l_inf budget, and trains with randomly contour-shrunk augmented masks so the perturbation generalizes to unseen mask shapes.
Key results
- On InpaintGuardBench unseen masks, DiffusionGuard reaches CLIP directional similarity 21.84 vs 23.30 for PhotoGuard (unprotected 24.29) at 16/255 budget.
- Under 90-second compute budget DiffusionGuard matches PhotoGuard's PSNR at roughly 11 seconds of optimization, per the paper's claim.
- Human survey win rate of DiffusionGuard over PhotoGuard is about 17% higher in the gap, on both seen and unseen masks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace PhotoGuard with DiffusionGuard | Stable Diffusion Inpainting protection · InpaintGuardBench Seen | PSNR: 12.87 -> 12.60 | Helped |
| Replace PhotoGuard with DiffusionGuard | Stable Diffusion Inpainting protection · InpaintGuardBench Seen | CLIP directional similarity: 21.17 -> 18.95 | Helped |
| Replace PhotoGuard with DiffusionGuard | Stable Diffusion Inpainting protection · InpaintGuardBench Seen | ImageReward: -1.537 -> -1.807 | Helped |
| Replace PhotoGuard with DiffusionGuard | Stable Diffusion Inpainting protection · InpaintGuardBench Seen | CLIP similarity: 27.89 -> 26.55 | Helped |
| Replace PhotoGuard with DiffusionGuard | Stable Diffusion Inpainting protection · InpaintGuardBench Unseen | PSNR: 14.53 -> 13.19 | Helped |
| Replace PhotoGuard with DiffusionGuard | Stable Diffusion Inpainting protection · InpaintGuardBench Unseen | CLIP directional similarity: 23.30 -> 21.84 | Helped |
| Replace PhotoGuard with DiffusionGuard | Stable Diffusion Inpainting protection · InpaintGuardBench Unseen | ImageReward: -1.357 -> -1.557 | Helped |
| Replace PhotoGuard with DiffusionGuard | Stable Diffusion Inpainting protection · InpaintGuardBench Unseen | CLIP similarity: 30.30 -> 29.05 | Helped |
| Use early-stage perturbation loss instead of image-space or reconstruction loss (ablation) | DiffusionGuard · InpaintGuardBench Seen | Protection metrics | Helped |
| Add mask augmentation (ablation) | DiffusionGuard · InpaintGuardBench Unseen | Protection metrics | Helped |
| Use mask-dependent protection rather than global mask-free perturbation (ablation) | DiffusionGuard versus AdvDM · InpaintGuardBench Unseen | Protection metrics | Helped |
Surprising findings
- Inpainting models produce fine details such as facial features already at the first denoising step, unlike standard diffusion models.
Limitations
- Mask-robustness depends on defender sharing a notion of the sensitive region with the attacker.
- Protection is weaker when the noise budget is reduced in some baseline comparisons, and purification remains a risk.
Experimental setup
Datasets: InpaintGuardBench DreamBooth
Benchmarks: InpaintGuardBench
Metrics: PSNR CLIP directional similarity ImageReward CLIP similarity
Baselines: PhotoGuard AdvDM Mist SDS(-)
Builds on: PhotoGuard AdvDM Stable Diffusion Inpainting projected gradient descent
Closest papers by meaning
- Anti-Diffusion: Preventing Abuse of Modifications of Diffusion-Based Models (AAAI 2025 · similarity 0.93)
- Pixel Is Not a Barrier: An Effective Evasion Attack for Pixel-Domain Diffusion Models (AAAI 2025 · similarity 0.92)
- BlurGuard: A Simple Approach for Robustifying Image Protection Against AI-Powered Editing (NeurIPS 2025 · similarity 0.91)
- Targeted Attack Improves Protection against Unauthorized Diffusion Customization (ICLR 2025 · similarity 0.91)
- Distraction is All You Need: Memory-Efficient Image Immunization against Diffusion-Based Image Editing (CVPR 2024 · similarity 0.90)
- AdvI2I: Adversarial Image Attack on Image-to-Image Diffusion Models (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.