Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Understanding the Robustness of Multi-modal Contrastive Learning to Distribution Shift
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically explains why multimodal contrastive learning (CLIP-style) is more robust to distribution shift than supervised learning, via intra-class contrasting, inter-class feature sharing, and rich captions, validated on synthetic/semi-synthetic MNIST and MSCOCO/CC-trained models on shifted ImageNets.
Problem
Why zero-shot multimodal contrastive learning (MMCL) such as CLIP yields representations and classifiers that are more robust to distribution shift than supervised learning (SL), which was not theoretically understood.
Approach
Uses a linearized MMCL loss analyzed via cross-covariance/SVD and two linear data models to show MMCL learns high-variance core features (intra-class contrasting) and exploits features annotated in other classes (inter-class feature sharing), while SL latches onto low-variance spurious features; also shows caption richness is necessary for these gains.
Key results
- Theory: SL accuracy on the true distribution is at most 2/3 under data model 1, while MMCL achieves at least 81% (with 85% best possible) when sigma_core=1.
- Under data model 2, SL accuracy is bounded by 60% for alpha=10, beta=1/3, whereas MMCL attains 100% zero-shot accuracy under a stated condition on m, beta, alpha.
- Experiments show MMCL-trained models are more OOD-robust than SL on MSCOCO and Conceptual Captions evaluated on shifted ImageNets; replacing captions with labels or removing intra-class contrasting reduces robustness.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MMCL instead of SL | Data Model 1 · Data Model 1 true distribution | accuracy: 2/3 -> 81% | Helped |
| Use MMCL instead of SL | Data Model 2 (α=10, β=1/3) · Data Model 2 true distribution | accuracy: 60% -> 100% | Helped |
| Include variation in the core feature in captions (increase π_core) | MMCL · Semi-synthetic MNIST binary classification, OOD test | OOD accuracy | Helped |
| Include variation in the spurious feature in captions (increase π_spu) | MMCL · Semi-synthetic MNIST binary classification, OOD test | OOD accuracy | No clear effect |
| Simplify captions to labels only (ablation) | CLIP · Six shifted ImageNet versions | OOD accuracy | Hurt |
| Remove contrasting between image-text pairs from the same class (ablation) | CLIP loss · OOD evaluation with encoders trained from scratch | OOD accuracy | Hurt |
| Mention features unrelated to the image's latent class in captions, with π > π̃ instead of π < π̃ | MMCL · Data Model 2 true distribution | zero-shot accuracy: 50% -> 100% | Helped |
| Share features across classes (β > 0 rather than β = 0) | MMCL · Data Model 2 true distribution | zero-shot accuracy | Helped |
Surprising findings
- Although CLIP's ID accuracy is lower than SL's on the small-scale setup, OOD accuracies appear comparable, with the advantage expected to grow with data scale.
- Linear probing on CLIP image features reduces its robustness advantage over supervised methods.
Limitations
- Theory relies on linearized contrastive loss, linear encoders, and Gaussian/synthetic data models with strong assumptions.
- Real-data experiments use small-scale setting (3-layer MLP on frozen encoders) and computational resource constraints limit scale.
Experimental setup
Datasets: MNIST MSCOCO Conceptual Captions ImageNet shifted variants (six)
Benchmarks: six shifted ImageNet variants semi-synthetic MNIST color-spurious test set
Metrics: zero-shot accuracy out-of-distribution accuracy ID-OOD accuracy relationship
Baselines: supervised learning (cross-entropy / logistic loss) CLIP with labels-as-captions SimCLR SupCon
Builds on: CLIP linearized contrastive loss analysis spurious correlation analysis of supervised learning (max-margin/SVM implicit bi
Closest papers by meaning
- On the Comparison between Multi-modal and Single-modal Contrastive Learning (NeurIPS 2024 · similarity 0.88)
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.88)
- Continual Multimodal Contrastive Learning (NeurIPS 2025 · similarity 0.87)
- Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP (ICLR 2024 · similarity 0.87)
- CLIP the Bias: How Useful is Balancing Data in Multimodal Learning? (ICLR 2024 · similarity 0.86)
- Generalized Contrastive Learning for Universal Multimodal Retrieval (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.