Research map / Adversarial robustness and security / Attacks on text and graphs
Targeted Attack Improves Protection against Unauthorized Diffusion Customization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ACE, a targeted adversarial attack on latent diffusion models (with a chaotic-pattern target, and an ACE+ variant adding a latent-target term) that protects images against unauthorized LoRA and SDEdit customization more effectively than untargeted baselines.
Problem
Existing protections against unauthorized diffusion customization rely on untargeted adversarial attacks, which under small perturbation budgets only add patterns to parts of customized outputs and fail to degrade image quality enough.
Approach
Adapt targeted attacks to diffusion models by minimizing the score-function error toward a fixed chaotic-pattern target (ACE), rather than maximizing error as untargeted attacks do; the consistent target unifies the chaotic pattern learned during customization, and ACE+ adds a latent-distance target term.
Key results
- On CelebA-HQ LoRA, ACE achieves FDFR 1.00 (no faces detected in outputs) versus 0.33 for the strongest baseline ASPL.
- On WikiArt SDEdit, ACE* reaches MS-SSIM 0.13 and CLIP-SIM 73.92 versus 0.30 and 87.25 for ASPL.
- An artist user survey covering 1304 artists found ACE produced the worst customization image in 55% of pairwise comparisons against ASPL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ACE instead of ASPL | Stable Diffusion 1.5 protection · CelebA-HQ SDEdit | MS-SSIM: 0.82 -> 0.73 | Helped |
| Use ACE instead of ASPL | Stable Diffusion 1.5 protection · CelebA-HQ SDEdit | CLIP-SIM: 84.12 -> 74.70 | Helped |
| Use ACE instead of ASPL | Stable Diffusion 1.5 protection · CelebA-HQ LoRA | CLIP-IQA: 33.62 -> 31.46 | Hurt |
| Use ACE instead of ASPL | Stable Diffusion 1.5 protection · CelebA-HQ LoRA | FDFR: 0.33 -> 1.00 | Helped |
| Use ACE instead of ASPL | Stable Diffusion 1.5 protection · WikiArt SDEdit | MS-SSIM: 0.30 -> 0.23 | Helped |
| Use ACE instead of ASPL | Stable Diffusion 1.5 protection · WikiArt SDEdit | CLIP-SIM: 87.25 -> 76.13 | Helped |
| Use ACE instead of ASPL | Stable Diffusion 1.5 protection · WikiArt LoRA | CLIP-IQA: 46.74 -> 40.54 | Hurt |
| Add the latent-target objective to ACE (ablation) | ACE protection · CelebA-HQ SDEdit | MS-SSIM: 0.73 -> 0.69 | Helped |
| Add the latent-target objective to ACE (ablation) | ACE protection · CelebA-HQ LoRA | FDFR: 1.00 -> 0.07 | Hurt |
| Replace ACE's selected target with the new repeated NeurIPS-logo target (ACE*) (ablation) | ACE protection · WikiArt LoRA | CLIP-IQA: 40.54 -> 76.50 | Helped |
| Increase target pattern density (ablation) | ACE target · LoRA | protection effectiveness | Mixed |
| Increase target pattern contrast (ablation) | ACE target · LoRA | protection effectiveness | Helped |
Surprising findings
- Targeted attacks with a fixed chaotic target outperform untargeted attacks, which previous targeted attacks (ASPL-T) failed to do.
- On Stable Diffusion 3, ACE mainly makes LoRA outputs deviate semantically to irrelevant persons rather than degrading quality.
Limitations
- Evaluated on only two customization methods (LoRA+DreamBooth and SDEdit) and SD1.x-family backbones primarily.
- Gaussian-noise purification remains effective at recovering quality, and DiffPure degrades customization output semantics.
- Input updating dominates runtime (about 70 s per image on RTX 3080Ti, 27% slower than ASPL).
Experimental setup
Datasets: CelebA-HQ WikiArt
Benchmarks: CelebA-HQ WikiArt
Metrics: MS-SSIM CLIP-SIM CLIP-IQA FDFR ISM
Baselines: AdvDM PhotoGuard PhotoGuard+ ASPL ASPL-T
Builds on: ASPL AdvDM PhotoGuard LoRA DreamBooth
Closest papers by meaning
- DiffusionGuard: A Robust Defense Against Malicious Diffusion-based Image Editing (ICLR 2025 · similarity 0.91)
- Perturb a Model, Not an Image: Towards Robust Privacy Protection via Anti-Personalized Diffusion Models (NeurIPS 2025 · similarity 0.90)
- SimAC: A Simple Anti-Customization Method for Protecting Face Privacy against Text-to-Image Synthesis of Diffusion Models (CVPR 2024 · similarity 0.89)
- Pixel Is Not a Barrier: An Effective Evasion Attack for Pixel-Domain Diffusion Models (AAAI 2025 · similarity 0.89)
- Prompt-Agnostic Adversarial Perturbation for Customized Diffusion Models (NeurIPS 2024 · similarity 0.89)
- Anti-Diffusion: Preventing Abuse of Modifications of Diffusion-Based Models (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.