Research map / Adversarial robustness and security / Attacks on text and graphs
CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLIPure, which purifies adversarial image embeddings in CLIP's multi-modal latent space (via a DaLLE-2 diffusion prior or a training-free cosine-similarity likelihood) to get adversarially robust zero-shot classification.
Problem
Zero-shot image classifiers like CLIP are highly vulnerable to adversarial attacks, and existing defenses either adversarially train (hurting clean accuracy and generalizing poorly to unseen attacks) or run pixel-space diffusion purification (slow and less robust).
Approach
Formulate purification risk as a KL divergence between forward attack and reverse purification SDE joint distributions, and conclude that purifying in CLIP's multi-modal latent space with unit-vector normalization lowers that risk; CLIPure-Cos uses cosine similarity to a blank template as the likelihood, avoiding generative models.
Key results
- CLIPure-Cos reaches 91.1% robust accuracy on CIFAR-10 under AutoAttack l_inf=8/255, up from 71.7% for the prior best (LM-EDM).
- CLIPure-Cos reaches 72.6% robust accuracy on ImageNet under l_inf=4/255, versus 59.6% for AT-Swin-L.
- On 13 zero-shot datasets at l_inf=4/255, CLIPure-Cos averages 67.4% robustness, a 108% relative improvement over FARE (32.4%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use cosine similarity instead of DiffusionPrior for CLIP latent-space purification (ablation) | CLIPure · CIFAR-10, AutoAttack ℓ∞=8/255 | robust accuracy: 88.0 -> 91.1 | Helped |
| Use cosine similarity instead of DiffusionPrior for CLIP latent-space purification (ablation) | CLIPure · ImageNet, AutoAttack ℓ∞=4/255 | robust accuracy: 65.0 -> 72.6 | Helped |
| Use cosine similarity instead of DiffusionPrior for CLIP latent-space purification (ablation) | CLIPure · 13 zero-shot datasets, AutoAttack ℓ∞=2/255 | average robust accuracy: 64.0 -> 68.8 | Helped |
| Use cosine similarity instead of DiffusionPrior for CLIP latent-space purification (ablation) | CLIPure · 13 zero-shot datasets, AutoAttack ℓ∞=4/255 | average robust accuracy: 63.3 -> 67.4 | Helped |
| Normalize image embeddings to unit vectors during purification (ablation) | CLIPure · ImageNet | robust accuracy | Helped |
| Use CLIPA ViT-H-14 instead of the standard CLIP backbone (ablation) | CLIPure-Cos · ImageNet, AutoAttack ℓ∞=4/255 | robust accuracy: 74.4 -> 79.3 | Helped |
Surprising findings
- Purifying unnormalized CLIP embeddings failed to exceed 10% robustness on ImageNet, showing vector magnitude must be normalized out.
- CLIPure-Cos, a discriminative cosine-similarity likelihood with no generative model, outperforms diffusion-based CLIPure-Diff and pixel-space purifiers.
Limitations
- Results reported with create_graph=False, which the authors acknowledge may cause gradient obfuscation rather than a true adaptive attack.
- Robustness on ImageNet and CIFAR-10 relies on evaluation with 1000 sampled test examples.
- Purification with CLIPure-Diff relies on an off-the-shelf DaLLE-2 DiffusionPrior and a chosen timestep range (900-1000).
Experimental setup
Datasets: CIFAR-10 ImageNet CalTech101 StanfordCars CIFAR100 DTD EuroSAT FGVC Aircraft Flowers ImageNet-R ImageNet-S PCAM
Benchmarks: CIFAR-10 ImageNet CalTech101 StanfordCars CIFAR100 DTD EuroSAT FGVC Aircraft Flowers ImageNet-R ImageNet-S PCAM
Metrics: clean accuracy robust accuracy average zero-shot robustness inference time
Baselines: DiffPure LM-EDM FARE TeCoA AT-DDPM AT-EDM MixedNUTS LM-StableDiffusion DiffPure-DaLLE2.Decoder LM-DaLLE2.Decoder
Builds on: CLIP DaLLE-2 DiffPure likelihood maximization Score SDE
Closest papers by meaning
- CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIP (CVPR 2025 · similarity 0.91)
- Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models (ICML 2024 · similarity 0.88)
- A Sober Look at the Robustness of CLIPs to Spurious Features (NeurIPS 2024 · similarity 0.87)
- On the Zero-shot Adversarial Robustness of Vision-Language Models: A Truly Zero-shot and Training-free Approach (CVPR 2025 · similarity 0.87)
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.87)
- Better Safe than Sorry: Pre-training CLIP against Targeted Data Poisoning and Backdoor Attacks (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.