Research map / Image generation / Autoregressive and latent generation
Toward effective protection against diffusion-based mimicry through score distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SDS-accelerated and gradient-descent variants of latent-diffusion adversarial protection (SDS(-), SDST) that halve compute/memory while matching or beating AdvDM, Mist and PhotoGuard against SDEdit, inpainting and textual inversion.
Problem
Protecting images against diffusion-based mimicry (SDEdit, inpainting, textual inversion) with adversarial perturbations is too computationally and memory expensive for individual users, and the mechanism of attacking latent diffusion models is not well understood.
Approach
Argues the encoder, not the denoiser, is the vulnerable point of LDMs; approximates the semantic-loss gradient with Score Distillation Sampling (dropping denoiser Jacobian), which halves cost, and shows that minimizing (rather than maximizing) the semantic loss yields more natural perturbations that blur edits.
Key results
- SDS halves computation: SDS-based attacks run in about 30s and ~8G VRAM versus about 65s and ~16G VRAM for AdvDM and Mist.
- SDS(-) achieves the best LPIPS and PSNR and the top human-evaluation score (4.55) among the compared protections in the perturbation-quality table.
- SDS(+) gives the strongest textual-inversion protection, matching AdvDM while being much cheaper.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SDS to AdvDM semantic-loss optimization | AdvDM · Perturbation generation | TIME: ~ 65s -> ~ 30s | Helped |
| Apply SDS to AdvDM semantic-loss optimization | AdvDM · Perturbation generation | VRAM: ~16G -> ~8G | Helped |
| Apply SDS to AdvDM semantic-loss optimization | AdvDM · Perturbation quality | SSIM: 0.714 -> 0.719 | Helped |
| Replace gradient ascent with gradient descent on semantic loss (ablation) | AdvDM · SDEdit | FID-score: 251.82 -> 199.45 | Hurt |
| Add SDS to AdvDM with gradient ascent (ablation) | AdvDM · SDEdit | FID-score: 251.82 -> 242.53 | No clear effect |
| Replace gradient ascent with gradient descent and add SDS (ablation) | AdvDM · SDEdit | HumanEval: 2.03 -> 4.34 | Helped |
| Add textual loss to SDS(-), lambda=1 (ablation) | SDS(-) · SDEdit, strength i | FID-score: 206.01 -> 346.75 | Helped |
| Vary textual-loss weight from lambda=1 to lambda=5 (ablation) | SDST · SDEdit, strength iii | FID-score: 365.04 -> 322.80 | Hurt |
| Minimize rather than maximize semantic loss (ablation) | SDS protection · SDEdit and perturbation appearance | HumanEval: SDS(+) HumanEval -; SDS(-) HumanEval 4.34 | Helped |
| Directly attack the LDM latent instead of perturbing the input image (ablation) | LDM · SDEdit | protection effectiveness | Hurt |
Surprising findings
- Minimizing the semantic loss (gradient descent) yields effective protection that blurs edits and produces more natural perturbations than maximizing it.
- Directly attacking the denoiser in latent space with large budgets fails to fool it, suggesting the denoiser is robust and the encoder is the real bottleneck.
Limitations
- Focuses only on latent diffusion models; pixel-space diffusion attacks were not successful in their tests.
- Evaluation datasets are small (100-200 images per subset) and human evaluation is based on 53 respondents.
- Defenses such as Adv-Clean, crop-and-resize and JPEG compression were not fully countered by any method.
Experimental setup
Datasets: WikiArt self-collected anime, portrait, artwork and landscape subsets (100-200 images each)
Benchmarks: SDEdit image inpainting textual inversion
Metrics: FID IA-score LPIPS PSNR SSIM VRAM time human preference rank
Baselines: AdvDM PhotoGuard Mist AdvDM(-)
Builds on: AdvDM PhotoGuard Mist Score Distillation Sampling SDEdit
Closest papers by meaning
- Nearly Zero-Cost Protection Against Mimicry by Personalized Diffusion Models (CVPR 2025 · similarity 0.91)
- Anti-Diffusion: Preventing Abuse of Modifications of Diffusion-Based Models (AAAI 2025 · similarity 0.90)
- DiffusionGuard: A Robust Defense Against Malicious Diffusion-based Image Editing (ICLR 2025 · similarity 0.89)
- Towards Black-Box Membership Inference Attack for Diffusion Models (ICML 2025 · similarity 0.89)
- Score Forgetting Distillation: A Swift, Data-Free Method for Machine Unlearning in Diffusion Models (ICLR 2025 · similarity 0.88)
- MimicDiffusion: Purifying Adversarial Perturbation via Mimicking Clean Diffusion Model (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.