Research map / Vision-language and multimodal / CLIP and zero-shot
A Sober Look at the Robustness of CLIPs to Spurious Features
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CounterAnimal, an iNaturalist-based animal dataset whose easy/hard background splits expose CLIP's reliance on spurious background features, and gives a theory that the CLIP objective cannot remove them.
Problem
Existing CLIP robustness benchmarks (ImageNet variants) were curated for ImageNet-trained models and may not reflect CLIP's reliance on spurious features learned from web-scale data like LAION.
Approach
Build a natural-image test set split by background into easy and hard groups per class, where the groups are selected using a CLIP proxy model's accuracy drop, so the benchmark measures CLIP-specific spurious correlations that ImageNet models are less affected by.
Key results
- CLIP-LAION400M-ViT-B/32 zero-shot accuracy drops from 97.62% (easy) to 70.91% (hard) on the ice bear class, and CLIP models across backbones and pre-train sets show drops of roughly 9 to 30 points.
- ImageNet models show smaller drops than CLIP models (e.g., ConvNext-L drop 7.65 vs ViT-B/32 LAION400M drop 30.18), reversing the impression from ImageNet variant benchmarks.
- Larger backbones (e.g., ViT-bigG/14 drop 10.61) and high-quality data (DataComp1B and DFN2B ViT-L/14 drops 9.39 and 10.22) reduce the drop, while scaling pre-train data alone does not.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a larger CLIP backbone | CLIP on CounterAnimal · CounterAnimal 1 vs. 1000 | accuracy drop (easy to hard): 30.18 -> 17.59 | Helped |
| Scale up pre-training data alone | CLIP on CounterAnimal · CounterAnimal 1 vs. 1000 | accuracy drop (easy to hard) | No clear effect |
| Use high-quality filtered pre-training data | CLIP on CounterAnimal · CounterAnimal 1 vs. 1000 | accuracy drop (easy to hard): 17.59 -> 9.39 | Helped |
| Use ImageNet-1K supervised training instead of CLIP pre-training | ViT-B/16 · CounterAnimal 1 vs. 1000 | accuracy drop (easy to hard): 16.12 -> 9.99 | Helped |
| Fine-tune CLIP with the CLIP objective rather than use zero-shot inference | RN-50 OpenAI · ColoredCOO | accuracy drop (in-distribution to out-of-distribution): 1.34 -> 94.89 | Hurt |
| Fine-tune CLIP with object-and-background prompts rather than use zero-shot inference | RN-50 OpenAI · ColoredCOO | accuracy drop (in-distribution to out-of-distribution): 1.34 -> 93.89 | Hurt |
Surprising findings
- ImageNet supervised models are more robust to CounterAnimal's background shift than CLIP models, contrary to prior findings on ImageNet variants.
- Fine-tuning CLIP with the CLIP objective gives no additional robustness over ERM on ColoredCOCO with synthetic spurious features.
Limitations
- Dataset covers only 45 animal classes and 13k photos from iNaturalist, limiting scale and semantic scope.
- Spurious discovery relies on a single proxy CLIP model (LAION400M ViT-B/32), so the benchmark may be biased toward that model.
- Theory studies a simplified linear encoder setting with one invariant and one spurious feature.
Experimental setup
Datasets: ImageNet-1K iNaturalist LAION400M LAION2B DataComp1B DFN2B COCO (ColoredCOCO) MNIST (MultiColoredMNIST)
Benchmarks: CounterAnimal
Metrics: class-wise average accuracy accuracy drop between easy and hard groups top-1 accuracy
Baselines: ImageNet supervised models (AlexNet, VGG, ResNet, ViT, ConvNext) MiniGPT4 LLaVA-1.5 CLIP-OpenAI CLIP-LAION2B CLIP-DataComp1B CLIP-DFN2B
Builds on: CLIP ImageNet-1K OpenCLIP
Closest papers by meaning
- Demystifying CLIP Data (ICLR 2024 · similarity 0.88)
- Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations (CVPR 2024 · similarity 0.88)
- CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIP (CVPR 2025 · similarity 0.88)
- What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insights (NeurIPS 2024 · similarity 0.87)
- Does CLIP’s generalization performance mainly stem from high train-test similarity? (ICLR 2024 · similarity 0.87)
- CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classification (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.