Research map / Vision-language and multimodal / CLIP and zero-shot
Robustness in Both Domains: CLIP Needs a Robust Text Encoder
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LEAF, an efficient adversarial finetuning method for CLIP text encoders using a batched, constant-cost character-level attack, to make text embeddings robust and combine them with robust image encoders (FARE) for bimodal robustness.
Problem
CLIP text encoders are vulnerable to character-level adversarial perturbations, which shift embeddings and degrade downstream models (text-to-image generation, retrieval, LMMs), while robustness of text encoders has been unexplored.
Approach
Extends FARE's embedding-preserving adversarial loss to the text domain (Levenshtein-bounded perturbations with semantic constraints) and replaces the costly Charmer attack with a batched attack evaluating a constant number of random perturbations per sentence, giving roughly an order-of-magnitude speedup.
Key results
- On AG-News at k=1, adversarial accuracy of CLIP-ViT-L/14 improved from 44.5% to 63.3% with LEAF text encoder (bimodal with robust image encoder reaches 63.2%).
- Attack per batch: LEAF rho=20 takes 1.83 s vs 15.17 s for Charmer-1 and 118.19 s for Charmer-20 on ViT-B-32, with comparable adversarial accuracy (58.3% vs 60.2%).
- On MS-COCO text-to-image retrieval at k=2, OpenCLIP-ViT-H/14 with robust encoders reaches 49.31 Recall@1 under Charmer attack vs 39.26 for non-robust.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the text encoder with LEAF while retaining the FARE image encoder | CLIP-ViT-L/14 · AG-News, Charmer-20 k=1 | adversarial accuracy: 44.5 -> 63.2 | Helped |
| replace the text encoder with LEAF while retaining the FARE image encoder | OpenCLIP-ViT-H/14 · AG-News, Charmer-20 k=1 | adversarial accuracy: 37.5 -> 53.3 | Helped |
| replace the text encoder with LEAF while retaining the FARE image encoder | OpenCLIP-ViT-g/14 · AG-News, Charmer-20 k=1 | adversarial accuracy: 32.9 -> 46.3 | Helped |
| replace CLIP text encoder with LEAF | CLIP-ViT-L/14 · MS-COCO retrieval, Charmer-Con k=2 | Recall@1: 30.66 -> 40.22 | Helped |
| replace CLIP text encoder with LEAF | OpenCLIP-ViT-H/14 · MS-COCO retrieval, Charmer-Con k=2 | Recall@1: 39.26 -> 49.31 | Helped |
| replace CLIP text encoder with LEAF | CLIP-ViT-L/14 · MS-COCO text embedding inversion | Word Recall: 34.4 -> 46.4 | Helped |
| replace Charmer-1 with the LEAF training-time attack (ρ=20) | CLIP-ViT-B-32 finetuning · AG-News, 1-epoch finetuning | attack time per batch (s): 15.17_{(± 28.98)} -> 1.83_{(± 0.11)} | Helped |
| increase ρ from 1 to 50 without semantic constraints (ablation) | LEAF text finetuning · AG-News, k=1 | adversarial accuracy: 57.6 -> 66.3 | Helped |
| increase ρ from 1 to 50 without semantic constraints (ablation) | LEAF text finetuning · ImageNet, k=1 | clean accuracy: 74.7 -> 65.5 | Hurt |
| add semantic constraints to training attacks at ρ=50 (ablation) | LEAF text finetuning · ImageNet, k=1 | clean accuracy: 65.5 -> 72.6 | Helped |
| increase ρ from 1 to 50 with semantic constraints (ablation) | LEAF text finetuning · AG-News, k=1 | adversarial accuracy: 54.4 -> 63.2 | Helped |
Surprising findings
- Unconstrained character attacks produce semantically changed captions (e.g., bear->beer), so semantic constraints are needed for text-to-image and retrieval tasks.
- Robust text encoders alone (without robust image encoders) leave image-domain robustness at 0, so both encoders must be robust for bimodal robustness.
Limitations
- Image and text encoders are finetuned in isolation; joint training not explored.
- Token-level attacks not trained against since they often change semantics.
- Largest models (OpenCLIP-ViT-bigG image encoder, EVA-CLIP) were not trained due to compute constraints.
Experimental setup
Datasets: DataComp-small ImageNet AG-News MS-COCO
Benchmarks: ImageNet AG-News MS-COCO CalTech101 CIFAR10 CIFAR100 EuroSAT ImageNet-R ImageNet-Sketch STL-10
Metrics: zero-shot accuracy adversarial accuracy Recall@1 Recall@5 CLIPScore BLEU word recall token recall cosine similarity
Baselines: FARE Charmer-20 Charmer-1 CLIP-ViT-L/14 OpenCLIP-ViT-H/14 OpenCLIP-ViT-g/14
Builds on: FARE TeCoA Charmer PGD adversarial training
Closest papers by meaning
- CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIP (CVPR 2025 · similarity 0.90)
- Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models (ICML 2024 · similarity 0.88)
- Enhancing CLIP Robustness via Cross-Modality Alignment (NeurIPS 2025 · similarity 0.87)
- Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models (NeurIPS 2024 · similarity 0.87)
- One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models (CVPR 2024 · similarity 0.87)
- FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.