Research map / Adversarial robustness and security / Attacks on text and graphs
CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that adversarial protective perturbations on latent diffusion models work mainly by distorting latent representations, and proposes CAT, which trains LoRA adapters on the latent autoencoder with a reconstruction loss on protected images to undo the distortion and defeat protections.
Problem
Protective perturbations that prevent unauthorized customization of latent diffusion models are assumed robust, but their robustness against adaptive attacks has received limited study, and existing purification-based attacks are costly and can alter image content.
Approach
Identify latent-representation distortion as the main reason protective perturbations work, then attack the latent autoencoder via lightweight LoRA adapters trained with a reconstruction loss on protected images, so the realigned latents let DreamBooth/LoRA customization succeed; adapters are detachable at inference.
Key results
- On VGGFace2 with DreamBooth, CAT-both raises FSS for AdvDM(+) from 0.390 baseline to 0.534 and for Mist from 0.035 to 0.557.
- On CelebA-HQ with DreamBooth, CAT-both improves FSS for Anti-DreamBooth from 0.471 to 0.662 and Mist from 0.160 to 0.572.
- CAT-de (decoder-only adapters) has little effect on protection, supporting the claim that encoder-side latent distortion is the key factor.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CAT adapters to both encoder and decoder | DreamBooth customization of protected images · CelebA-HQ, AdvDM(+) | FSS: 0.340 -> 0.643 | Helped |
| Add CAT adapters to the encoder only | DreamBooth customization of protected images · CelebA-HQ, AdvDM(+) | FQS: 0.244 -> 0.448 | Helped |
| Add CAT adapters to the encoder only | DreamBooth customization of protected images · VGGFace2, AdvDM(+) | FSS: 0.390 -> 0.560 | Helped |
| Add CAT adapters to the encoder only | DreamBooth customization of protected images · VGGFace2, AdvDM(+) | FQS: 0.257 -> 0.600 | Helped |
| Replace CAT with Noisy-Upscaling purification | DreamBooth customization of protected images · CelebA-HQ, AdvDM(+) | FSS: 0.643 -> 0.531 | Hurt |
| Replace CAT with Gaussian filtering | DreamBooth customization of protected images · CelebA-HQ, AdvDM(+) | FSS: 0.643 -> 0.492 | Hurt |
| Add CAT adapters to the encoder only | DreamBooth style mimicry · WikiArt, AdvDM(+) | CLIP-IQA: 0.343 -> 0.621 | Helped |
| Add CAT adapters to the decoder only (ablation) | DreamBooth customization of protected images · Object-driven synthesis across evaluated protections | FSS | No clear effect |
| Use adapter rank 128 instead of rank 4 (ablation) | CAT-both · VGGFace2, AdvDM(+), DreamBooth | FSS: 0.525 -> 0.544 | Helped |
| Use adapter rank 128 instead of rank 4 (ablation) | CAT-both · VGGFace2, AdvDM(+), DreamBooth | FQS: 0.372 -> 0.460 | Helped |
Surprising findings
- Decoder-only adapters (CAT-de) barely reduce protection effectiveness, indicating the encoder-side latent distortion is the key mechanism.
- Adapter-based CAT outperforms optimization-based purification (Noisy-Upscaling) and Gaussian filtering on most protections, while being cheaper.
Limitations
- The mechanism claim (latent distortion as primary cause) is based on empirical observation and the authors acknowledge other factors may contribute.
- Evaluated on only five identities per dataset and a single artist, with a single pretrained model (Stable Diffusion v2.1).
Experimental setup
Datasets: CelebA-HQ VGGFace2 WikiArt
Benchmarks: CelebA-HQ VGGFace2 WikiArt
Metrics: FSS FQS FID CLIP-IQA
Baselines: AdvDM Mist SDS SDST Glaze Anti-DreamBooth MetaCloak Noisy-Upscaling Gaussian filtering
Builds on: LoRA DreamBooth Latent Diffusion Models Noisy-Upscaling
Closest papers by meaning
- Adv-Diffusion: Imperceptible Adversarial Face Identity Attack via Latent Diffusion Model (AAAI 2024 · similarity 0.90)
- Perturbing Attention Gives You More Bang for the Buck: Subtle Imaging Perturbations That Efficiently Fool Customized Diffusion Models (CVPR 2024 · similarity 0.89)
- Robust Classification via a Single Diffusion Model (ICML 2024 · similarity 0.89)
- MimicDiffusion: Purifying Adversarial Perturbation via Mimicking Clean Diffusion Model (CVPR 2024 · similarity 0.89)
- DiffusionGuard: A Robust Defense Against Malicious Diffusion-based Image Editing (ICLR 2025 · similarity 0.89)
- AdvAD: Exploring Non-Parametric Diffusion for Imperceptible Adversarial Attacks (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.