Research map / Vision-language and multimodal / CLIP and zero-shot
Enhancing CLIP Robustness via Cross-Modality Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COLA, a training-free test-time defense for CLIP that projects adversarial image features onto a text-induced subspace and aligns multi-view image and multi-description text distributions via optimal transport.
Problem
CLIP zero-shot classifiers are highly vulnerable to adversarial perturbations, which push image embeddings away from text embeddings and scatter local structure, causing near-collapse of accuracy.
Approach
Project adversarial image embeddings onto the SVD subspace spanned by LLM-augmented class text features, then embed this projection into an entropy-weighted optimal transport cost between augmented image views and class description distributions, with no retraining.
Key results
- On ImageNet under PGD (eps=1/255, ViT-B/32), COLA reaches 50.0% robust accuracy vs 40.0% for TTC and 1.1% for CLIP, with 62.8% clean accuracy.
- On the 9-dataset benchmark under PGD, the robust accuracy of the projection-based OT variant rises from 2.4% (CLIP) to 46.2%.
- On ViT-L/14, COLA gains 35.8% robust accuracy on ImageNet over TTC under PGD attacks, per the paper's text.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add COLA | CLIP ViT-B/32 · ImageNet and its variants (PGD) | average accuracy: 20.9 -> 23.1 | Helped |
| add COLA | CLIP ViT-B/32 · ImageNet-A (PGD) | robust accuracy: 15.4 -> 22.7 | Helped |
| add COLA | CLIP ViT-B/16 · 9-datasets (PGD) | robust accuracy: 13.9 -> 20.7 | Helped |
| add COLA | CLIP ViT-B/16 · ImageNet (PGD) | robust accuracy: 20.1 -> 32.1 | Helped |
| add COLA | CLIP ViT-B/16 · IN-Variants (PGD) | robust accuracy: 17.9 -> 24.5 | Helped |
| replace the original OT cost matrix with the projection-based cost matrix (ablation) | OT alignment · 9-datasets (PGD) | robust accuracy: 44.9 -> 46.2 | Helped |
| replace the original OT cost matrix with the projection-based cost matrix (ablation) | OT alignment · ImageNet (PGD) | robust accuracy: 46.3 -> 50.0 | Helped |
| replace the original OT cost matrix with the projection-based cost matrix (ablation) | OT alignment · IN-Variants (PGD) | robust accuracy: 33.2 -> 37.8 | Helped |
| replace the original OT cost matrix with the projection-based cost matrix (ablation) | OT alignment · 9-datasets (CW) | robust accuracy: 35.2 -> 42.3 | Helped |
| replace the original OT cost matrix with the projection-based cost matrix (ablation) | OT alignment · ImageNet (CW) | robust accuracy: 34.5 -> 42.3 | Helped |
| replace the original OT cost matrix with the projection-based cost matrix (ablation) | OT alignment · IN-Variants (CW) | robust accuracy: 30.7 -> 33.0 | Helped |
| increase the number of image augmentations (ablation) | COLA · Caltech101 and ImageNet | robust accuracy | Helped |
Surprising findings
- Projection onto the text-induced subspace restores image-text similarity to near clean levels, as shown in similarity distribution analysis.
- The paper's proof of the cosine distortion bound relies on a first-order approximation and a loose general-case argument, which is a weakness rather than a finding.
Limitations
- The text-induced subspace may encode dataset-specific priors that limit generalization to unseen domains.
- Stronger defenses could provoke more adaptive attacks; evaluation under adaptive attacks is not addressed.
- Theoretical cosine-distortion bound relies on small-perturbation Taylor approximations and assumptions such as equal perturbations.
Experimental setup
Datasets: ImageNet Caltech101 SUN397 DTD EuroSAT Pets Cars Flowers Food101 Aircraft ImageNetV2 ImageNet-Sketch
Benchmarks: Pets Flowers Aircraft DTD EuroSAT Cars Food101 SUN397 Caltech101 ImageNet ImageNet-V2 ImageNet-Sketch
Metrics: clean accuracy robust accuracy
Baselines: CLIP TeCoA PMG FARE Anti-Adversary Hedge Defence TTE TTC
Builds on: CLIP Sinkhorn algorithm TTC LLM-generated class descriptions
Closest papers by meaning
- CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIP (CVPR 2025 · similarity 0.89)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.89)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.89)
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.88)
- Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.