Research map / Adversarial robustness and security / Certified and empirical robustness
Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PMG-AFT, an adversarial fine-tuning method for CLIP that uses an auxiliary branch distilling the frozen original CLIP's adversarial outputs plus a clean-output regularizer, improving zero-shot adversarial robustness and clean accuracy.
Problem
Adversarial fine-tuning of CLIP overfits to the downstream dataset, degrading zero-shot generalization and clean accuracy under adversarial attacks.
Approach
Keep the frozen pre-trained CLIP image encoder as a guide: a KL-divergence loss aligns the fine-tuned model's adversarial output distribution with the original model's adversarial output, and a regularizer aligns adversarial with clean outputs, on top of TeCoA-style text-supervised adversarial training.
Key results
- PMG-AFT improves average zero-shot robust accuracy (PGD-10, eps=1/255) to 31.95% vs 26.96% for FT-TeCoA, a 4.99% gain.
- Average zero-shot clean accuracy of 55.71% for PMG-AFT vs 46.99% for FT-TeCoA (+8.72%), close to the original CLIP's 59.98%.
- Under AutoAttack (eps=1/255), PMG-AFT averages 17.91% robust accuracy vs 10.33% for FT-TeCoA and 1.74% for CLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PMG-AFT instead of FT-TeCoA | CLIP ViT-B/32 fine-tuned on TinyImageNet · 16 zero-shot datasets, average (PGD-10, ε=1/255) | robust accuracy: +4.99% | Helped |
| use PMG-AFT instead of FT-TeCoA | CLIP ViT-B/32 fine-tuned on TinyImageNet · 16 zero-shot datasets, average clean accuracy | accuracy: 55.71 vs 46.99 | Helped |
| use PMG-AFT instead of FT-TeCoA | CLIP ViT-B/32 fine-tuned on TinyImageNet · 16 zero-shot datasets, average (AutoAttack, ε=1/255) | robust accuracy: 10.33 -> 17.91 | Helped |
| add the generalization information branch (Lgeneral) (ablation) | FT-TeCoA adversarial fine-tuning · 16 zero-shot datasets, average clean accuracy | accuracy: +11.29% | Helped |
| add the generalization information branch (Lgeneral) (ablation) | FT-TeCoA adversarial fine-tuning · 16 zero-shot datasets, average (PGD-10, ε=1/255) | robust accuracy: +1.48% | Helped |
| add the clean-example regularization loss (Lclean) to Lgeneral (ablation) | FT-TeCoA adversarial fine-tuning with Lgeneral · 16 zero-shot datasets, average (PGD-10, ε=1/255) | robust accuracy: 28.44% -> 32.04% | Helped |
| add the clean-example regularization loss (Lclean) to Lgeneral (ablation) | FT-TeCoA adversarial fine-tuning with Lgeneral · 16 zero-shot datasets, average clean accuracy | accuracy: 58.28% -> 55.72% | Hurt |
| replace output-layer KL divergence with L2 distance (ablation) | PMG-AFT · 16 zero-shot datasets, average | robust accuracy: 31.95 -> 27.23 | Hurt |
| replace output-layer KL divergence with L2 distance (ablation) | PMG-AFT · 16 zero-shot datasets, average | clean accuracy: 55.71 -> 48.21 | Hurt |
| use penultimate feature layer with L2 distance instead of output-layer KL divergence (ablation) | PMG-AFT · 16 zero-shot datasets, average | robust accuracy: 31.95 -> 22.90 | Hurt |
| use penultimate feature layer with L2 distance instead of output-layer KL divergence (ablation) | PMG-AFT · 16 zero-shot datasets, average | clean accuracy: 55.71 -> 38.32 | Hurt |
| use penultimate feature layer with cosine distance instead of output-layer KL divergence (ablation) | PMG-AFT · 16 zero-shot datasets, average | robust accuracy: 31.95 -> 24.53 | Hurt |
Surprising findings
- Adding the generalization branch alone raised average clean accuracy from 46.99% to 58.28% while robust accuracy rose only modestly, and the regularization loss reduced clean accuracy to 55.72% while increasing robust accuracy to 32.04%.
Limitations
- Overhead from the second (frozen original CLIP) forward pass increases training time relative to FT-TeCoA.
- TinyImageNet evaluation is not strictly zero-shot, and PMG-AFT trails FT-TeCoA on TinyImageNet itself.
- Experiments are limited to CLIP ViT-B/32 and mostly to a single perturbation bound for the main results.
Experimental setup
Datasets: TinyImageNet CIFAR100
Benchmarks: CIFAR10 CIFAR100 STL10 SUN397 Food101 OxfordPets Flowers102 DTD EuroSAT FGVC Aircraft TinyImageNet ImageNet
Metrics: zero-shot robust accuracy zero-shot clean accuracy average accuracy
Baselines: FT-TeCoA FT-Standard CLIP VP-TeCoA interpolation (model parameter interpolation)
Builds on: TeCoA PGD adversarial training CLIP
Closest papers by meaning
- On the Zero-shot Adversarial Robustness of Vision-Language Models: A Truly Zero-shot and Training-free Approach (CVPR 2025 · similarity 0.91)
- Improving Zero-Shot Adversarial Robustness in Vision-Language Models by Closed-form Alignment of Adversarial Path Simplices (ICML 2025 · similarity 0.89)
- Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models (CVPR 2024 · similarity 0.88)
- CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIP (CVPR 2025 · similarity 0.87)
- Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.