Research map / Adversarial robustness and security / Attacks on text and graphs

Prompt-Agnostic Adversarial Perturbation for Customized Diffusion Models

NeurIPS 2024 · Adversarial robustness and security › Attacks on text and graphs · approach: Diffusion-based adversarial purification · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PAP, a prompt-agnostic adversarial perturbation that models the prompt embedding distribution with a Laplace approximation and maximizes expected diffusion loss over sampled prompts to protect images from customized diffusion (DreamBooth/LoRA/TI) tampering.

Problem

Existing adversarial protections for customized text-to-image diffusion models are prompt-specific: perturbations optimized for one training prompt fail when attackers use unseen prompts, leaving personal portraits and artworks exposed.

Approach

Model the distribution of attacker prompt embeddings around a reference prompt as a Gaussian via Laplace approximation (mean from momentum-based loss minimization, variance from a Taylor/Hessian estimate with an identity-matrix simplification), then craft perturbations by Monte Carlo maximization of the diffusion loss expected over sampled prompts.

Key results

• On Celeb-HQ, PAP reaches FID 249.9 and CLIP-I 0.5539 versus 233.3 and 0.6371 for the second-best Anti-DreamBooth, with LPIPS improved by 13.55% over Anti-DB.
• On Wikiart style imitation, PAP attains FID 448.3, CLIP-I 0.5641 and BRISQUE 38.47, outperforming AdvDM, Anti-DB and IAdvDM across all metrics.
• PAP also defends LoRA and Textual Inversion customizations, raising FID from 152.5 to 375.8 for LoRA and from 144.3 to 281.8 for TI.

Measured effects reported in the paper

Surprising findings

• Setting the text sampling steps to 0 (no prompt sampling) yields the Anti-DreamBooth-level results, and performance keeps improving with more sampling steps up to 25 with diminishing returns.
• Using the empty string as the initial prompt causes a significant performance drop, which the authors attribute to the reference prompt serving as a crucial prior for parameter estimation.

Limitations

• The Hessian inverse is approximated by a scaled identity matrix, with a cosine-dissimilarity bound of 0.0909 under standard settings that depends on assumed values of D and l.
• The error bounds and the Laplace approximation rely on loose theoretical arguments; the authors acknowledge the upper bound is very coarse.
• Per-image optimization takes about 4 to 6 minutes on an A800 GPU, and text sampling steps add substantial cycle time (7.4s at N=15 versus 0.3s at N=0).

Experimental setup

Datasets: CelebA-HQ VGGFace2 Wikiart

Benchmarks: CelebA-HQ VGGFace2 Wikiart

Metrics: FID CLIP-I LPIPS LAION aesthetic BRISQUE CLIP

Baselines: AdvDM Anti-DreamBooth IAdvDM Photoguard Glaze

Builds on: AdvDM Anti-DreamBooth DreamBooth momentum iterative attack DiffPure

Closest papers by meaning

• Perturb a Model, Not an Image: Towards Robust Privacy Protection via Anti-Personalized Diffusion Models (NeurIPS 2025 · similarity 0.90)
• Can Protective Perturbation Safeguard Personal Data from Being Exploited by Stable Diffusion? (CVPR 2024 · similarity 0.90)
• Targeted Attack Improves Protection against Unauthorized Diffusion Customization (ICLR 2025 · similarity 0.89)
• DiffusionGuard: A Robust Defense Against Malicious Diffusion-based Image Editing (ICLR 2025 · similarity 0.89)
• CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models (ICML 2025 · similarity 0.88)
• PID: Prompt-Independent Data Protection Against Latent Diffusion Models (ICML 2024 · similarity 0.88)