Research map / Adversarial robustness and security / Attacks on text and graphs

Harnessing Global-Local Collaborative Adversarial Perturbation for Anti-Customization

CVPR 2025 · Adversarial robustness and security › Attacks on text and graphs · approach: Diffusion-based adversarial purification · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GoodAC, an anti-customization method that adds adversarial perturbations combining global feature-correlation disruption (block-angle transformation on UNet features) with local facial-attribute edge distortion to protect face images from DreamBooth/LoRA customization of Stable Diffusion.

Problem

Latent diffusion model personalization (DreamBooth, LoRA) enables unauthorized face theft and misuse; existing anti-customization perturbations are weak at blocking concept transfer and semantic/facial-attribute theft.

Approach

Jointly disrupt global spatial correlations of UNet perceptual features via block-angle transformation (rearrange plus rotation) and locally distort facial-attribute edges identified via text-inversion descriptions and edge detection, combined with SimAC's whole-feature loss under PGD.

Key results

• On CelebA-HQ, GoodAC reduces ISM by up to about 89% relative to the strongest baseline (SimAC) under 'a photo of sks person' (0.03 vs 0.31); FDFR exceeds 95% under several prompts.
• On VGGFace2, ISM is reduced by over 50% and FDFR exceeds 90% for multiple prompts relative to SimAC.
• Ablation on the Eiffel Tower prompt shows Lperceptual alone (ISM 0.01) beats SimAC (0.04), and combining with Ledge gives FDFR 96.67.

Measured effects reported in the paper

Surprising findings

• The local edge-based loss alone performs slightly worse than SimAC, while it adds value when combined with the global loss.

Limitations

• Evaluated only on open-source fine-tuning-based LDMs; proprietary or non-fine-tuning models unexplored.
• May be vulnerable to specially designed customization techniques.
• Black-box transfer and LoRA results only qualitative and in supplementary material.

Experimental setup

Datasets: CelebA-HQ VGGFace2

Benchmarks: CelebA-HQ VGGFace2

Metrics: ISM FDFR BRISQUE SER-FQA

Baselines: AdvDM Anti-DreamBooth SimAC

Builds on: DreamBooth SimAC Anti-DreamBooth AdvDM LoRA

Closest papers by meaning

• SimAC: A Simple Anti-Customization Method for Protecting Face Privacy against Text-to-Image Synthesis of Diffusion Models (CVPR 2024 · similarity 0.90)
• Targeted Attack Improves Protection against Unauthorized Diffusion Customization (ICLR 2025 · similarity 0.88)
• Perturb a Model, Not an Image: Towards Robust Privacy Protection via Anti-Personalized Diffusion Models (NeurIPS 2025 · similarity 0.88)
• Prompt-Agnostic Adversarial Perturbation for Customized Diffusion Models (NeurIPS 2024 · similarity 0.87)
• Adversarial Perturbations Cannot Reliably Protect Artists From Generative AI (ICLR 2025 · similarity 0.86)
• Edit Away and My Face Will not Stay: Personal Biometric Defense against Malicious Generative Editing (CVPR 2025 · similarity 0.85)