Research map / Adversarial robustness and security / Attacks on text and graphs

Variance as a Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models

ICML 2025 · Adversarial robustness and security › Attacks on text and graphs · approach: Diffusion-based adversarial purification

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes variance-driven adversarial perturbations on the VAE encoder of latent diffusion models, using Laplace (LA) and Lagrange-entropy (LE) losses, to erase identity semantics from customized fine-tuning with 30 PGD steps.

Problem

Existing adversarial protections for customized (few-shot fine-tuned) latent diffusion models only add noise distortions and fail to completely erase identity semantics; they are also slow and computationally expensive, and transfer poorly.

Approach

Identifies VAE latent variance as the key factor controlling distortion and identity erasure, and designs losses (LA: constant-gradient absolute loss; LE: entropy plus Lagrange constraint) whose gradient signs align with variance growth, so PGD rapidly drives latents toward pure-noise images.

Key results

• LE and LA reach ISM 0 and FDFR 1 on CelebA-HQ and VGGFace2 against DreamBooth, versus 0.069 ISM and 0.938 FDFR for the best baseline PID.
• Protection completes in about 7-8 seconds using 4.5 GB GPU memory on an A100, versus 241 s and 4582 MB for PID and 1843 s for MetaCloak.
• Transfer to LoRA on SD2.1 gives ISM 0 and FDFR 1 for LE and LA, while on SDXL, SD3.5 and FLUX.1-dev transfer is weaker (e.g., SDXL ISM 0.171 for LE).

Measured effects reported in the paper

Surprising findings

• Continuing optimization past 30 steps (to 50) makes images shift from pure noise to chaotic scene-like visuals, so the authors terminate at 30 steps.
• Using either the entropy term or the Lagrange term alone in LE fails to achieve effective attack, implying the combination is necessary.

Limitations

• Transferability to SDXL, SD3.5 and FLUX.1-dev is weaker than on SD2.1, with lower FDFR (e.g., 0.204 for LE on FLUX.1-dev).
• Evaluation is limited to two facial datasets with 50 identities each and DreamBooth/LoRA fine-tuning.
• Optimization beyond 30 steps degrades the protection behavior, requiring a hand-picked stopping point.

Experimental setup

Datasets: CelebA-HQ VGGFace2

Benchmarks: CelebA-HQ VGGFace2

Metrics: ISM FDFR Brisque LPIPS PSNR SSIM

Baselines: AdvDM ASPL Mist MetaCloak SimAC DisDiff SDS- PID

Builds on: PGD DreamBooth PID Mist

Closest papers by meaning

• Adv-Diffusion: Imperceptible Adversarial Face Identity Attack via Latent Diffusion Model (AAAI 2024 · similarity 0.89)
• LoRID: Low-Rank Iterative Diffusion for Adversarial Purification (AAAI 2025 · similarity 0.89)
• Step Vulnerability Guided Mean Fluctuation Adversarial Attack against Conditional Diffusion Models (AAAI 2024 · similarity 0.88)
• CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models (ICML 2025 · similarity 0.88)
• Hiding Images in Diffusion Models by Editing Learned Score Functions (CVPR 2025 · similarity 0.88)
• Towards Understanding the Robustness of Diffusion-Based Purification: A Stochastic Perspective (ICLR 2025 · similarity 0.88)