Research map / Adversarial robustness and security / Attacks on text and graphs

Enhancing Diffusion-based Unrestricted Adversarial Attacks via Adversary Preferences Alignment

NeurIPS 2025 · Adversarial robustness and security › Attacks on text and graphs · approach: Vision-language model robustness · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Reframes diffusion-based unrestricted adversarial example generation as adversary preference alignment, decoupling visual consistency (LoRA fine-tuning with a similarity reward) from attack effectiveness (dual-path gradient guidance plus diffusion augmentation against a substitute classifier) to improve black-box transferability.

Problem

Unrestricted adversarial attacks built on diffusion models must balance visual consistency with black-box attack effectiveness, but these conflicting objectives cause unstable joint optimization and reward hacking.

Approach

Treat adversarial example generation as preference alignment with two sequential stages: first align visual consistency by fine-tuning LoRA on the input image with a differentiable similarity reward, then optimize the image latent or prompt embedding with trajectory-level and step-wise attack rewards from a substitute classifier, plus a diffusion augmentation that mixes intermediate and final outp

Key results

• On ImageNet-compatible data with RN-50 as substitute, APA-GC reaches 88.02% average black-box ASR across target models, versus 59.49% for ACA.
• With MobileViT-s as substitute, APA-GC achieves 77.48% average black-box ASR versus 56.90% for ACA.
• Against defenses with ViT-B as substitute, APA-GC reaches 70.20% average ASR versus 57.47% for ACA.

Measured effects reported in the paper

Surprising findings

• Joint single-stage optimization with a consistency penalty suffers reward hacking, with attack success falling as the consistency weight increases.
• Prompt-embedding optimization (APA-GC-P) yields the best visual consistency (LPIPS 0.09) but lower attack success than latent optimization.

Limitations

• Attack guidance is applied only in the final steps and tuning of T and T_a trades off attack strength against time and image quality.
• Visual consistency is measured by metrics that the authors say cannot fully capture consistency, relying partly on qualitative comparison.

Experimental setup

Datasets: ImageNet-compatible Dataset ImageNet validation set

Benchmarks: ImageNet-compatible Dataset

Metrics: attack success rate LPIPS SSIM CLIP Score NIMA-AVA CNN-IQA

Baselines: SAE cAdv tAdv ColorFool NCF ACA DiffPGD-MI

Builds on: ACA DiffPGD Stable Diffusion LoRA DDIM inversion Diffusion Model Alignment via Differentiable Rewards (HPA)

Closest papers by meaning

• AdvPaint: Protecting Images from Inpainting Manipulation via Adversarial Attention Disruption (ICLR 2025 · similarity 0.87)
• CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models (ICML 2025 · similarity 0.85)
• AdvI2I: Adversarial Image Attack on Image-to-Image Diffusion Models (ICML 2025 · similarity 0.85)
• Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models (CVPR 2025 · similarity 0.85)
• Boosting Adversarial Transferability with Spatial Adversarial Alignment (NeurIPS 2025 · similarity 0.85)
• Variance as a Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models (ICML 2025 · similarity 0.85)