Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
OT-CLIP: Understanding and Generalizing CLIP via Optimal Transport
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reinterprets CLIP's InfoNCE training as inverse entropic optimal transport and generalizes the loss to an OT-based family (FGW, unbalanced, DBOT), plus OT-based graph-matching inference for zero-shot, long-tailed and selective classification.
Problem
CLIP's InfoNCE contrastive loss lacks a principled explanation of its alignment mechanism, and zero-shot inference treats classification as independent point matching, ignoring structure among samples and labels.
Approach
Shows CLIP training equals a bilevel inverse entropic OT problem whose inner constraints are the row and column normalizations of InfoNCE; replaces these constraints with variants of regularized OT (fused Gromov, unbalanced, DBOT) to form an OT-CLIP loss family, and recasts zero-shot inference as fused Gromov graph matching.
Key results
- On CC3M pretraining, FGromov OT loss reaches 20.6% average zero-shot accuracy versus 19.2% for InfoNCE across the Table 1 datasets.
- FGromov OT-ISoftmax raises zero-shot average accuracy from 44.3% (vanilla softmax) to 49.9% across 20 datasets with OpenAI CLIP.
- On selective zero-shot classification at 50% selective rate, Unbalanced OT improves average accuracy from 54.9% to 56.9% over softmax.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace InfoNCE with FGromov OT for representation learning | OT-CLIP · CC3M pretraining; average across 18 zero-shot datasets | top-1 accuracy: 19.2 -> 20.6 | Helped |
| Replace InfoNCE with unbalanced OT for representation learning (ablation) | OT-CLIP · CC3M pretraining; ImageNet1K zero-shot | top-1 accuracy: 16.3 -> 17.1 | Helped |
| Use entropic OT instead of InfoNCE for representation learning (ablation) | OT-CLIP image encoder · COCO instance segmentation | mAP: 0.162 -> 0.166 | Helped |
| Use unbalanced OT instead of InfoNCE for representation learning (ablation) | OT-CLIP image encoder · CIFAR100 full fine-tuning | top-1 accuracy: 68.0 -> 68.4 | Hurt |
| Use FGromov OT instead of InfoNCE for text-encoder training (ablation) | OT-CLIP text encoder · STS tasks; average score | average score: 57.29 -> 58.20 | Helped |
| Use FGromov OT-ISoftmax instead of vanilla softmax for inference | OpenAI CLIP · Zero-shot classification; average across 19 datasets | top-1 accuracy: 44.3 -> 49.9 | Helped |
| Use FGromov OT for long-tailed inference instead of vanilla softmax | OpenAI CLIP · ImageNet; zero-shot, long-tailed test distribution | top-1 accuracy: 58.6 -> 63 | Helped |
| Use FGromov OT instead of vanilla softmax for inference | Fine-tuned OpenAI CLIP · ImageNet; uniform test distribution | top-1 accuracy: 44.4 -> 59.9 | Helped |
| Use unbalanced OT instead of softmax for selective inference | OpenAI CLIP · Selective zero-shot classification; average across 19 datasets | top-1 accuracy: 54.9 -> 56.9 | Helped |
| Use partial OT instead of softmax for selective inference | OpenAI CLIP · Selective zero-shot classification; average across 19 datasets | top-1 accuracy: 54.9 -> 56.6 | Helped |
Surprising findings
- Vanilla entropic OT and some OT variants do not consistently beat InfoNCE; for example, DBOT and Unbalanced OT underperform on several STS tasks for the text encoder.
- CyCLIP reaches only 31.3% linear-probe accuracy on CIFAR100 versus 45.4% for InfoNCE, showing baseline instability.
Limitations
- Gains over InfoNCE on CC3M pretraining are modest and vary by dataset and OT variant.
- Only a subset of OT variants is derived; other divergences and regularizers are left as future work.
- Fused Gromov inference is non-convex and relies on iterative approximations.
Experimental setup
Datasets: CC3M ImageNet-1K CIFAR100 Places365 COCO STS12-16 STS-B SICK-R
Benchmarks: ImageNet-1K CIFAR-10 CIFAR-100 SUN397 Food-101 Pets Caltech-101 STL-10 Country211 PCAM ImageNet-A ImageNet-R
Metrics: top-1 accuracy mAP mAR Spearman correlation
Baselines: InfoNCE HNS Triplet CyCLIP Softmax Classifier Normalization Class-Aware Bias
Builds on: CLIP InfoNCE Inverse entropic OT (Shi et al., 2023) DBOT Fused Gromov-Wasserstein
Closest papers by meaning
- A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR 2024 · similarity 0.90)
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.89)
- AmorLIP: Efficient Language-Image Pretraining via Amortization (NeurIPS 2025 · similarity 0.89)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.88)
- TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives (NeurIPS 2024 · similarity 0.88)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.