Research map / Label-efficient and robust learning / OOD and anomaly detection
Does CLIP’s generalization performance mainly stem from high train-test similarity?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Tests whether CLIP's OOD generalization stems from highly similar LAION training images by pruning LAION splits to match ImageNet-Train's similarity gap to OOD benchmarks and retraining CLIP ViT-B/32.
Problem
Whether CLIP's strong zero-shot out-of-distribution performance on ImageNet-based benchmarks is mainly explained by high train-test similarity between web-scale training data (LAION) and the test sets.
Approach
Introduces a similarity-gap alignment: prune LAION samples that are more similar to each test set than any ImageNet-Train image is, so the pruned training set matches ImageNet-Train's nearest-neighbor similarity distribution, then retrain CLIP from scratch to measure the residual OOD performance.
Key results
- Pruning LAION-200M to match ImageNet-Train's similarity gap (sketch-pruned) keeps ImageNet-Sketch accuracy at 44.78% vs 48.61% baseline, a ~10.8% relative drop, while ImageNet-Val is 68.34%.
- The sketch-pruned model reaches 44.78% on ImageNet-Sketch, far above the ~14% expected from ImageNet-Train's effective robustness line at that ImageNet-Val accuracy.
- A 100M far-pruned LAION-200M core set (all-far-pruned) averages 50.71% across six test sets vs 50.67% for LAION-200M, outperforming SemDeDup 100M (43.7%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove highly similar LAION samples to match ImageNet-Train similarity for ImageNet-Sketch (ablation) | CLIP ViT-B/32 trained on L-200M + ImageNet-Train · ImageNet-Sketch | top-1 accuracy: 50.21 -> 44.78 | Hurt |
| Remove highly similar LAION samples to match ImageNet-Train similarity for ImageNet-R (ablation) | CLIP ViT-B/32 trained on L-200M + ImageNet-Train · ImageNet-R | top-1 accuracy: 72.9 -> 69.48 | Hurt |
| Prune highly similar LAION samples to match ImageNet-Train similarity gaps across six test sets (ablation) | CLIP ViT-B/32 trained on L-200M + ImageNet-Train · ImageNet-Sketch | top-1 accuracy: 50.21 -> 44.12 | Hurt |
| Prune highly similar LAION samples to match ImageNet-Train similarity gaps across six test sets (ablation) | CLIP ViT-B/32 trained on L-200M + ImageNet-Train · ImageNet-R | top-1 accuracy: 72.9 -> 67.88 | Hurt |
| Prune highly similar LAION samples to match ImageNet-Train similarity gaps across six test sets (ablation) | CLIP ViT-B/32 trained on L-200M + ImageNet-Train · ImageNet-V2 | top-1 accuracy: 59.7 -> 58.61 | Hurt |
| Prune highly similar LAION samples to match ImageNet-Train similarity gaps across six test sets (ablation) | CLIP ViT-B/32 trained on L-200M + ImageNet-Train · ObjectNet | top-1 accuracy: 43.99 -> 44.39 | Helped |
| Far-prune to retain dissimilar samples across six ImageNet test sets, reducing training data to 100M (ablation) | CLIP ViT-B/32 trained on L-200M · six ImageNet test sets | average top-1 accuracy: 50.67 -> 50.71 | No clear effect |
| Use all-far-pruned (L-200M) instead of SemDeDup at 100M training examples | CLIP ViT-B/32 · six ImageNet test sets | average top-1 accuracy: 43.7 -> 50.71 | Helped |
| Near-prune similar training examples instead of random-pruning (ablation) | CLIP ViT-B/32 trained on LAION-200M · ImageNet-Sketch and ImageNet-Val | zero-shot top-1 accuracy | Hurt |
| Far-prune dissimilar training examples instead of random-pruning (ablation) | CLIP ViT-B/32 trained on LAION-200M · ImageNet-Sketch and ImageNet-Val | zero-shot top-1 accuracy | Helped |
| Near-prune based on ImageNet-Sketch or ImageNet-Val rather than far-prune (ablation) | CLIP ViT-B/32 trained on LAION-200M · MNIST and SVHN | zero-shot accuracy | Mixed |
| Use ViT-L-14 rather than CLIP ViT-B/16+ embeddings to compute perceptual similarity (ablation) | Similarity-based pruning and analysis · ImageNet-Train nearest-neighbor similarity to test sets | Spearman correlation: 0.93; 0.87; 0.90; 0.86; 0.93; 0.80 | No clear effect |
Surprising findings
- Removing highly similar training images to the test sets only marginally reduces CLIP's OOD performance, with the sketch-pruned model far exceeding the effective robustness line expected from ImageNet-Train.
- Far-pruned (dissimilar-removed) LAION subsets stay stable or slightly improve performance, suggesting dissimilar samples act like noise.
Limitations
- Only ImageNet-based benchmarks and CLIP ViT-B/32 are studied; other datasets and architectures left for future work.
- Pruning cannot guarantee removal of all highly similar images, since the perceptual similarity metric may miss some; the choice of similarity model matters less but is not exhaustive.
- Defining true OOD domains for pruning is ill-posed, so the study cannot measure true OOD performance directly.
Experimental setup
Datasets: LAION-400M ImageNet-1k train ImageNet-Val ImageNet-Sketch ImageNet-R ImageNet-A ImageNet-V2 ObjectNet CelebA Waterbirds MNIST SVHN
Benchmarks: ImageNet-Val ImageNet-Sketch ImageNet-A ImageNet-R ImageNet-V2 ObjectNet MNIST SVHN
Metrics: top-1 zero-shot accuracy nearest-neighbor similarity correlation relative performance change
Baselines: OpenAI CLIP ViT-B/32 OpenCLIP trained on LAION-400M ImageNet-trained ResNet-101 SemDeDup 100M rand-pruned LAION-200M splits
Builds on: CLIP OpenCLIP SemDeDup effective robustness
Closest papers by meaning
- When and How Does CLIP Enable Domain and Compositional Generalization? (ICML 2025 · similarity 0.89)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.88)
- What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insights (NeurIPS 2024 · similarity 0.87)
- A Sober Look at the Robustness of CLIPs to Spurious Features (NeurIPS 2024 · similarity 0.87)
- In Search of Forgotten Domain Generalization (ICLR 2025 · similarity 0.87)
- CLIPCEIL: Domain Generalization through CLIP via Channel rEfinement and Image-text aLignment (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.