Research map / Adversarial robustness and security / Certified and empirical robustness

CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIP

CVPR 2025 · Adversarial robustness and security › Certified and empirical robustness · approach: Adversarial training with feature consistency · 8 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Test-time Counterattacks (TTC), a training-free PGD-based defense that pushes adversarial CLIP images away from their embeddings at inference, gated by a τ-thresholded 'false stability' check.

Problem

CLIP is highly vulnerable to adversarial perturbations in zero-shot classification; existing robustness methods (adversarial finetuning, adversarial prompt tuning) require costly training, overfit, and reduce clean accuracy.

Approach

Adversarial images are 'falsely stable' to small random noise; the method uses CLIP's own pre-trained vision encoder to craft a counterattack perturbation maximizing L2 embedding drift from the original image embedding, applied only when the τ stability check indicates an adversarial input, with weighted aggregation over PGD steps.

Key results

• TTC improves average robust accuracy across 16 datasets from 2.70% (original CLIP) to 39.17% at attack budget 1/255, with clean accuracy dropping only 1.76 points (61.51 to 59.75).
• At attack budget 4/255, TTC reaches 20.63% average robust accuracy versus 0.09% for CLIP, at a cost of 5.52 points clean accuracy.
• Applying TTC on FARE raises average robust accuracy from 20.00% to 33.89% at 1/255 on top of the adversarially finetuned model.

Measured effects reported in the paper

Surprising findings

• Adversarial images produced by maximizing classification loss are more stable to small random noise than clean images ('false stability'), the opposite of the expected fragility.
• Adversarial finetuning greatly reduces CLIP's sensitivity to pixel-space variations, which limits the gains of test-time counterattacks on finetuned models.

Limitations

• Gains are smaller on adversarially finetuned models such as TeCoA and PMG-AFT due to reduced encoder expressiveness.
• Inference cost increases with counterattack steps, and the optimal number of steps depends on unknown attack strength.
• Test-time defenses may be circumvented by adaptive attacks; the appendix discusses possible adaptive attacks but the defense is not robust to them by design.

Experimental setup

Datasets: TinyImageNet CIFAR10 CIFAR100 STL10 ImageNet Caltech101 Caltech256 OxfordPets Flowers102 FGVCAircraft StanfordCars SUN397

Benchmarks: CIFAR10 CIFAR100 STL10 ImageNet Caltech101 Caltech256 OxfordPets Flowers102 FGVCAircraft StanfordCars SUN397 Country211

Metrics: robust accuracy under 10-step PGD clean accuracy

Baselines: CLIP-FT TeCoA PMG-AFT FARE Random noise Test-time transformation ensembling Anti-adversary Hedge Defense

Builds on: Schlarmann et al. unsupervised attack (FARE) Hedge Defense Anti-adversary TeCoA PMG-AFT

Closest papers by meaning

• CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classification (ICLR 2025 · similarity 0.91)
• Robustness in Both Domains: CLIP Needs a Robust Text Encoder (NeurIPS 2025 · similarity 0.90)
• Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models (ICML 2024 · similarity 0.90)
• On the Zero-shot Adversarial Robustness of Vision-Language Models: A Truly Zero-shot and Training-free Approach (CVPR 2025 · similarity 0.90)
• TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models (CVPR 2025 · similarity 0.89)
• Enhancing CLIP Robustness via Cross-Modality Alignment (NeurIPS 2025 · similarity 0.89)