Research map / Vision-language and multimodal / CLIP and zero-shot
Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FARE, an unsupervised adversarial fine-tuning of the CLIP ViT-L/14 vision encoder that preserves original embeddings so robustness transfers to LVLMs (OpenFlamingo, LLaVA) and zero-shot classification without retraining the downstream models.
Problem
Frozen CLIP vision encoders used in large vision-language models are highly vulnerable to adversarial image perturbations, enabling stealthy targeted attacks, misinformation and jailbreaks; existing supervised robust CLIP (TeCoA) degrades clean and zero-shot performance.
Approach
Fine-tune the CLIP image encoder without labels by minimizing the worst-case squared L2 distance between adversarial and original CLIP embeddings (feature-preserving adversarial training), so the robust encoder can be plugged into existing LVLMs and zero-shot classifiers without retraining downstream components.
Key results
- FARE^2 CLIP in OpenFlamingo-9B averages 51.4 clean CIDEr/VQA vs 46.4 for TeCoA^2 at 2/255 robustness, with robust average 19.7 vs 17.9.
- In LLaVA-1.5-7B, FARE^4 achieves 0% stealthy targeted attack success at eps=4/255 while the original CLIP breaks 100% of cases.
- On zero-shot classification, FARE^2 averages 67.0 clean vs 60.0 for TeCoA^2 across 13 datasets, close to original CLIP's 73.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace supervised TeCoA fine-tuning with unsupervised FARE at ε=2/255 | OpenFlamingo-9B with CLIP ViT-L/14 · average over COCO, Flickr30k, TextVQA, VQAv2 | clean average score: 46.4 -> 51.4 | Helped |
| replace supervised TeCoA fine-tuning with unsupervised FARE at ε=2/255 | LLaVA-1.5-7B with CLIP ViT-L/14 · average over COCO, Flickr30k, TextVQA, VQAv2 | clean average score: 61.6 -> 71.1 | Helped |
| train FARE at ε=4/255 rather than ε=2/255 | LLaVA-1.5-7B with CLIP ViT-L/14 · average over COCO, Flickr30k, TextVQA, VQAv2, ε=4/255 evaluation | average score: 20.1 -> 26.3 | Helped |
| use FARE² instead of TeCoA² | CLIP ViT-L/14 zero-shot classification · average over 13 zero-shot datasets, clean | accuracy: 60.0 -> 67.0 | Helped |
| use FARE⁴ instead of TeCoA⁴ | CLIP ViT-L/14 zero-shot classification · average over 13 zero-shot datasets, ε=2/255 | accuracy: 42.3 -> 45.9 | Helped |
| use FARE⁴ instead of TeCoA⁴ | CLIP ViT-L/14 zero-shot classification · average over 13 zero-shot datasets, ε=4/255 | accuracy: 31.9 -> 32.4 | Helped |
| use FARE² instead of TeCoA² | LLaVA-1.5-7B · POPE mean | F1-score: 75.9 -> 80.8 | Helped |
| use FARE² instead of TeCoA² | LLaVA-1.5-7B · SQA-I | accuracy: 61.1 -> 63.4 | Helped |
| replace CLIP with FARE² | LLaVA-1.5-7B · targeted attack at ε=2/255 | mean attack success rate: 100% -> 0% | Helped |
| replace CLIP with FARE⁴ | LLaVA-1.5-7B · targeted attack at ε=4/255 | mean attack success rate: 100% -> 0% | Helped |
| increase LR from 1e-5 to 1e-4 (ablation) | ViT-B/32 FARE · average zero-shot clean | accuracy: 48.6 -> 44.4 | Hurt |
| increase LR from 1e-5 to 1e-4 (ablation) | ViT-B/32 FARE · ImageNet adversarial evaluation at ε=2/255 and ε=4/255 | accuracy: 29.6 -> 34.2; 14.8 -> 20.2 | Helped |
Surprising findings
- Robust CLIP models yield POPE hallucination F1 and SQA results close to the original CLIP, while TeCoA causes notably more hallucinations.
- Adversarial images from OpenFlamingo and LLaVA transfer well between them with clean CLIP, but the transfer fails once the vision encoder is robust.
Limitations
- Robustness restricted to the visual input; language-side defenses not addressed.
- Not evaluated on instruction following, explainability, or perception tasks.
- Jailbreak evaluation is preliminary and may overestimate robustness.
Experimental setup
Datasets: ImageNet
Benchmarks: COCO Flickr30k TextVQA VQAv2 ImageNet zero-shot suite (13 datasets) POPE SQA-I stealthy targeted attacks on LLaVA jailbreaking attack
Metrics: CIDEr VQA accuracy clean accuracy robust accuracy F1-score attack success rate
Baselines: TeCoA original CLIP
Builds on: CLIP TeCoA PGD adversarial training AutoAttack
Closest papers by meaning
- SLADE: Shielding against Dual Exploits in Large Vision-Language Models (CVPR 2025 · similarity 0.90)
- CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIP (CVPR 2025 · similarity 0.90)
- One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models (CVPR 2024 · similarity 0.90)
- On the Zero-shot Adversarial Robustness of Vision-Language Models: A Truly Zero-shot and Training-free Approach (CVPR 2025 · similarity 0.89)
- R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning (CVPR 2025 · similarity 0.89)
- Robustness in Both Domains: CLIP Needs a Robust Text Encoder (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.