Research map / Vision-language and multimodal / CLIP and zero-shot
Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives a theoretical analysis of CLIP's transferable representation learning and zero-shot transfer under a conditional-independence latent-shared-feature model, shows square loss fails for zero-shot, and proposes a positive-pair L2 regularizer that improves CLIP on CC3M zero-shot and linear probing.
Problem
Why CLIP's contrastive vision-language pretraining yields representations that transfer to zero-shot classification, which lacks theoretical explanation, and how batch-level false negatives and modality misalignment affect learning.
Approach
Model image and text as conditionally independent given a sparse discrete shared feature; show near-optimal contrastive training aligns modalities and enables zero-shot prediction with prompts, contrast this with square loss, and add a positive-pair alignment regularizer that enlarges the margin.
Key results
- Adding the positive-pair regularizer to CLIP improves average zero-shot top-1 accuracy on six datasets from 39.20% to 41.26%, with a 3.62% gain on CIFAR10.
- Training with only the L2 regularization term yields near-random zero-shot accuracy (average 3.47%), showing square loss fails for zero-shot transfer.
- CLIP+Reg improves average linear probing accuracy over CLIP on seven datasets from 72.75% to 74.29%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add positive-pair regularization to CLIP | CLIP · Zero-shot classification, six datasets | accuracy: 39.20 -> 41.26 average; improves on 5 out of 6 datasets | Helped |
| add positive-pair regularization to CLIP | CLIP · Linear probing, seven datasets | accuracy: 72.75 -> 74.29 average | Helped |
| add positive-pair regularization to CLIP | CLIP · Flickr30K test retrieval | Average R@1/R@5/R@10: 70.47 -> 72.40 | Helped |
| add positive-pair regularization to CLIP | CLIP · MSCOCO test retrieval | Average R@1/R@5/R@10: 46.86 -> 47.14 | Helped |
| use only the L2 regularization objective, without contrastive loss (ablation) | CLIP · Zero-shot classification average | accuracy: 39.20 -> 3.47 | Hurt |
| use only the L2 regularization objective, without contrastive loss (ablation) | CLIP · Linear probing average | accuracy: 72.75 -> 6.51 | Hurt |
| regularize off-diagonal batch pairs as negatives rather than regularizing positive pairs (ablation) | CLIP · Zero-shot classification, six datasets | accuracy: 39.20 -> 39.52 average | No clear effect |
| add positive-pair regularization to CyCLIP | CyCLIP · MSCOCO test retrieval | Average R@1/R@5/R@10: 46.86 -> 47.23 | Helped |
| add positive-pair regularization to CLIP | CLIP with TinyViT image encoder · Zero-shot classification average | accuracy: 31.02 -> 32.05 | Helped |
| decrease temperature from 0.07 to 0.01 | CLIP · MSCOCO training batches | margin | Hurt |
Surprising findings
- Training with the square (L2) loss alone gives near-random zero-shot accuracy, even though the representation is useful for downstream classification.
- Off-diagonal (treated-as-negative) regularization gives only marginal and unstable gains, since in-batch off-diagonal pairs may share features.
Limitations
- Theory assumes a discrete sparse shared feature and one-to-one image-text mapping, which may not hold for real data.
- Experiments use relatively small training data (CC3M subset) and small batch sizes compared with industry-scale CLIP training.
- Zero-shot results on Flowers102 and Oxford-IIIT Pet are near random, so these were omitted.
Experimental setup
Datasets: CC3M MSCOCO Flickr30K
Benchmarks: CIFAR10 CIFAR100 STL10 Food101 ImageNetV2 DTD Flowers102 Oxford-IIIT Pet Flickr30K MSCOCO
Metrics: top-1 accuracy linear probing accuracy recall@1/5/10 margin distribution
Baselines: CLIP CyCLIP square loss (L2) only
Builds on: CLIP CyCLIP SimSiam UniCL
Closest papers by meaning
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.90)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.89)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.89)
- TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives (NeurIPS 2024 · similarity 0.88)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.88)
- AmorLIP: Efficient Language-Image Pretraining via Amortization (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.