Research map / Image generation / Text-to-image diffusion
Perturbing Attention Gives You More Bang for the Buck: Subtle Imaging Perturbations That Efficiently Fool Customized Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAAT, a cross-attention-focused PGD-style adversarial perturbation that alternately updates cross-attention W_K/W_V and the input perturbation to disrupt customized diffusion fine-tuning on face images.
Problem
Protecting users' images from being exploited to fine-tune customized latent diffusion models (DreamBooth, Textual Inversion, SVDiff, Custom Diffusion) for generating fake images, with existing attacks (Anti-DreamBooth, Mist) being slow and less general.
Approach
Instead of running PGD on a fixed model as Anti-DreamBooth and Mist do, CAAT jointly optimizes the perturbation and only the cross-attention key/value projections (W_K, W_V) of the LDM, exploiting the observation that cross-attention layers change most during customized fine-tuning while having few parameters.
Key results
- CAAT achieved 12 best and 3 second-best results out of 16 metrics in Table 1 across Custom Diffusion, DreamBooth, SVDiff, and Textual Inversion.
- On DreamBooth, CAAT reduced face detection success rate to 0.64 versus 0.81 for Anti-DreamBooth and 0.99 for Mist.
- CAAT trains in about 2.5 minutes on an RTX 3090 versus about 5.5 minutes for Anti-DreamBooth and 5 minutes for Mist.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CAAT instead of baseline attackers | Custom Diffusion · Custom Diffusion | IR: 0.03 -> -0.36 | Helped |
| Use CAAT instead of Anti-DreamBooth | DreamBooth · DreamBooth | FID: 307 -> 371 | Helped |
| Use CAAT instead of Mist | SVDiff · SVDiff | FID: 317 -> 355 | Helped |
| Use CAAT instead of Anti-DreamBooth | Textual Inversion · Textual Inversion | FR: 0.50 -> 0.43 | Helped |
| Increase perturbation budget η from 0.05 to 0.10 (ablation) | CAAT · DreamBooth | FID: 251 -> 371 | Helped |
| Increase perturbation budget η from 0.10 to 0.15 (ablation) | CAAT · DreamBooth | FID: 371 -> 401 | Helped |
| Train CAAT on Stable Diffusion v2.1 and evaluate on Stable Diffusion v1.4 (ablation) | CAAT · Custom Diffusion | FID: 224 -> 313 | Helped |
| Train CAAT on Stable Diffusion v2.1 and evaluate on Stable Diffusion v1.5 (ablation) | CAAT · Textual Inversion | FID: 259 -> 441 | Helped |
| Increase the number of perturbed images from 0 to 4 of 4 (ablation) | CAAT · DreamBooth | FID: 179 -> 371 | Helped |
| Increase the number of perturbed images from 0 to 4 of 4 (ablation) | CAAT · Textual Inversion | FID: 242 -> 396 | Helped |
Surprising findings
- Anti-DreamBooth and Mist perform poorly on Custom Diffusion, with face detection success rates of 1.00 in Table 1.
Limitations
- Large perturbation budgets (eta=0.15) introduce visually perceptible noise
- Perturbing one or fewer of four input images has limited effect
- Evaluated only on face images from CelebA-HQ with a small set of subjects
Experimental setup
Datasets: CelebA-HQ
Benchmarks: CelebA-HQ
Metrics: Face Detection Success Rate (FR) Face Similarity (FS) ImageReward (IR) FID
Baselines: Anti-DreamBooth Mist
Builds on: PGD Custom Diffusion Anti-DreamBooth Mist
Closest papers by meaning
- Where and How to Perturb: On the Design of Perturbation Guidance in Diffusion and Flow Models (NeurIPS 2025 · similarity 0.90)
- CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models (ICML 2025 · similarity 0.89)
- Intriguing Properties of Diffusion Models: An Empirical Study of the Natural Attack Capability in Text-to-Image Generative Models (CVPR 2024 · similarity 0.89)
- Pixel Is Not a Barrier: An Effective Evasion Attack for Pixel-Domain Diffusion Models (AAAI 2025 · similarity 0.89)
- MimicDiffusion: Purifying Adversarial Perturbation via Mimicking Clean Diffusion Model (CVPR 2024 · similarity 0.88)
- From Trojan Horses to Castle Walls: Unveiling Bilateral Data Poisoning Effects in Diffusion Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.