Research map / Adversarial robustness and security / Black-box and transfer attacks

DiffAM: Diffusion-based Adversarial Makeup Transfer for Facial Privacy Protection

CVPR 2024 · Adversarial robustness and security › Black-box and transfer attacks · approach: Transfer attacks on face recognition · 47 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DiffAM, a two-stage diffusion-based makeup transfer framework that first removes makeup via text-guided CLIP fine-tuning and then transfers adversarial makeup from a reference image to protect faces against black-box face recognition.

Problem

Face images shared on social media are vulnerable to unauthorized face recognition; existing adversarial protections either degrade visual quality (noise/patch) or are GAN-based makeup methods with poor fine-grained control and low black-box transferability.

Approach

Use a fine-tuned diffusion model to remove makeup from a reference image under CLIP text guidance, giving a deterministic non-makeup/makeup pair whose CLIP-space direction guides fine-grained adversarial makeup transfer, combined with an ensemble attack loss over surrogate FR models and early-stopped DDIM steps to preserve identity and background.

Key results

• Average black-box ASR of 77.88% across four FR models, a 12.98% improvement over the best prior method (CLIP2Protect, 64.90%).
• FID of 26.10 versus 34.44 for AMT-GAN and 37.12 for CLIP2Protect; PSNR 20.53 dB and SSIM 0.886.
• Average confidence of about 70 on Face++ and about 50 on Aliyun for protected images.

Measured effects reported in the paper

Surprising findings

• Reducing DDIM inversion steps from T=200 to T=60 improved FID (47.32 to 26.10), suggesting that more steps introduce unwanted facial attribute changes.

Limitations

• Relies on fine-tuning a diffusion model per image pair, which is computationally costly.
• Black-box ASR on FaceNet and some models remains moderate (e.g., 64.67% on FaceNet for CelebA-HQ).

Experimental setup

Datasets: MT (Makeup Transfer) dataset CelebA-HQ LADN

Benchmarks: CelebA-HQ LADN Face++ Aliyun

Metrics: attack success rate FID PSNR SSIM confidence score

Baselines: PGD MI-FGSM TI-DIM TIP-IM Adv-Makeup AMT-GAN CLIP2Protect

Builds on: ADM DDIM inversion BeautyGAN pixel-level histogram matching loss Asyrp/CLIP directional loss (StyleDiffusion-style text-guided editing) AMT-GAN ensemble attack

Closest papers by meaning

• Enhancing Facial Privacy Protection via Weakening Diffusion Purification (CVPR 2025 · similarity 0.90)
• Recoverable Facial Identity Protection via Adaptive Makeup Transfer Adversarial Attacks (AAAI 2025 · similarity 0.89)
• Transferable Facial Privacy Protection against Blind Face Restoration via Domain-Consistent Adversarial Obfuscation (ICML 2024 · similarity 0.88)
• SimAC: A Simple Anti-Customization Method for Protecting Face Privacy against Text-to-Image Synthesis of Diffusion Models (CVPR 2024 · similarity 0.86)
• FSFM: A Generalizable Face Security Foundation Model via Self-Supervised Facial Representation Learning (CVPR 2025 · similarity 0.86)
• Adv-Diffusion: Imperceptible Adversarial Face Identity Attack via Latent Diffusion Model (AAAI 2024 · similarity 0.86)