Research map / Adversarial robustness and security / Attacks on text and graphs

Improving Zero-Shot Adversarial Robustness in Vision-Language Models by Closed-form Alignment of Adversarial Path Simplices

ICML 2025 · Adversarial robustness and security › Attacks on text and graphs · approach: Vision-language model robustness

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AdvSimplex, an adversarial fine-tuning method that aligns CLIP predictions over closed-form second-order statistics of adversarial simplices, approximating infinite sampling of the simplex via a Taylor-expansion upper bound.

Problem

Adversarial fine-tuning of CLIP for zero-shot robustness typically aligns only individual adversarial samples with clean ones, ignoring intermediate adversaries along the attack trajectory; explicitly sampling simplices formed by them is computationally prohibitive.

Approach

Taylor-expand the CLIP output around the clean sample and upper-bound the alignment loss using closed-form second-order statistics (Σx) of uniform points in simplices formed by the clean vertex and consecutive intermediate adversaries, requiring only Jacobian and Hessian-vector products at clean inputs.

Key results

• Average robust accuracy (PGD-20, eps=2/255) across 15 datasets reaches 35.68% vs 30.94% for FARE and 27.33% for TeCoA (ViT-B/32).
• Average clean accuracy 60.23% vs 56.68% for FARE, approaching standard CLIP's 64.90%.
• Closed-form simplex training takes 4.1 hours per epoch vs 13.6 hours for sampling-based alignment with comparable robustness.

Measured effects reported in the paper

Surprising findings

• Adding a tetrahedron (Q=4) simplex variant gives only marginal further gains over the triangle (AdvSimplex) version, 36.49 vs 35.68 PGD robust accuracy.

Limitations

• Relies on a Taylor expansion with negligible higher-order remainder, valid only for sufficiently small epsilon.
• Requires Hessian-vector products, adding cost relative to standard point-wise adversarial fine-tuning (one HVP ~ 2-4 Jacobian evaluations).
• Gains over the simpler clean-adversarial alignment baseline are modest for some components (e.g., cross-product term).

Experimental setup

Datasets: ImageNet

Benchmarks: ImageNet STL10 CIFAR-10 CIFAR-100 SUN397 Stanford Cars Food101 OxfordPet Flower102 DTD EuroSAT FGVC Aircraft

Metrics: zero-shot clean accuracy zero-shot robust accuracy AUC CIDEr recall@1

Baselines: TeCoA PMG-FT FARE Standard CLIP

Builds on: TeCoA TRADES PGD adversarial training FARE PMG-FT

Closest papers by meaning

• On the Zero-shot Adversarial Robustness of Vision-Language Models: A Truly Zero-shot and Training-free Approach (CVPR 2025 · similarity 0.93)
• One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models (CVPR 2024 · similarity 0.89)
• Few-Shot Adversarial Prompt Learning on Vision-Language Models (NeurIPS 2024 · similarity 0.89)
• Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness (CVPR 2024 · similarity 0.89)
• R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning (CVPR 2025 · similarity 0.89)
• Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models (ICML 2024 · similarity 0.88)