Research map / Vision-language and multimodal / CLIP and zero-shot
Modeling Caption Diversity in Contrastive Vision-Language Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Llip, a contrastive vision-language pretraining method whose image embedding is a text-conditioned mixture of K learnable visual mixture tokens, trained with a SigLIP-style loss on MetaCLIP data.
Problem
CLIP maps each image to a single vector and pushes all caption representations toward it, which ignores that an image admits many valid, diverse captions and limits the richness of visual representations.
Approach
The visual encoder outputs K learnable mixture tokens; a multi-head cross-attention whose queries come from the caption text feature produces caption-dependent mixing weights, yielding a contextualized image embedding contrasted with the text embedding under a SigLIP-style sigmoid loss, so the image representation is one-to-many rather than invariant.
Key results
- Llip ViT-G/14 reaches 83.5% ImageNet zero-shot top-1, +1.4% over MetaCLIP ViT-G/14 (82.1%).
- On ViT-B/32 with the same MetaCLIP data, Llip_64 averages 67.5 across 22 classification tasks vs 63.5 for re-implemented SigLIP (+4.7% average gain as stated in text).
- Llip ViT-B/16 improves MSCOCO image-to-text R@1 from 59.7 (SigLIP) to 63.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| condition visual mixture weights on text features instead of using a learned shared query (ablation) | Llip, ViT-B/32 · ImageNet zero-shot | top-1 accuracy | Helped |
| add 63 extra register tokens while using only the first learned token (ablation) | SigLIP, ViT-B/32 · ImageNet zero-shot | top-1 accuracy | No clear effect |
| add 63 extra learned tokens and uniformly average the 64 tokens (ablation) | SigLIP, ViT-B/32 · ImageNet zero-shot | top-1 accuracy | Helped |
| use a learned shared query for non-uniform token mixing instead of uniform averaging (ablation) | Llip token aggregation, ViT-B/32 · ImageNet zero-shot | top-1 accuracy | No clear effect |
| increase the number of mixture tokens (ablation) | Llip · ImageNet zero-shot | top-1 accuracy | Helped |
| increase cross-attention softmax temperature to large values (ablation) | Llip · ImageNet zero-shot | top-1 accuracy | Hurt |
| include image patch token outputs alongside mixture tokens in cross-attention (ablation) | Llip, ViT-B/32 · ImageNet zero-shot | top-1 accuracy | Mixed |
| use 32 mixture tokens rather than 1 mixture token with image patch tokens in cross-attention (ablation) | Llip, ViT-B/32 · ImageNet zero-shot | top-1 accuracy | Helped |
| use caption-conditioned visual representations instead of SigLIP contrastive representations | Llip64, ViT-B/32 · ImageNet zero-shot | top-1 accuracy: 67.3 -> 70.4 | Helped |
| use caption-conditioned visual representations instead of MetaCLIP representations | Llip64, ViT-G/14 · ImageNet zero-shot | top-1 accuracy: 82.1 -> 83.5 | Helped |
| use Llip instead of MetaCLIP | ViT-L/14 · Geographic diversity, Overall Top5 | accuracy: 85.26 -> 83.55 | Hurt |
| use Llip instead of MetaCLIP | ViT-L/14 · Scene understanding, CLEVR | accuracy: 25.9 -> 25.5 | Hurt |
Surprising findings
- Adding extra learned tokens with uniform averaging alone (no text conditioning) already gives a significant ImageNet gain over the SigLIP baseline, and non-uniform learned mixing showed no significant difference from uniform.
Limitations
- Zero-shot inference requires encoding each image with every candidate caption, increasing inference cost with the number of classes.
- Pretraining cost increases with the number of mixture tokens, which adds computation to the vision transformer.
- Comparisons to literature baselines use different architectures, data, and recipes; DFN outperforms Llip on ImageNet with larger data and higher resolution.
Experimental setup
Datasets: MetaCLIP-2.5B Flickr30k MSCOCO ImageNet CIFAR10 CIFAR100 CUB Food-101 STL-10 Caltech-101 MNIST SUN397
Benchmarks: ImageNet CIFAR10 CIFAR100 CUB Food-101 STL-10 Caltech-101 MNIST SUN397 Cars Aircraft Pets
Metrics: top-1 accuracy average zero-shot accuracy recall@1 recall@5 recall@10
Baselines: CLIP SigLIP MetaCLIP OpenCLIP EVA-CLIP DFN CLIPA-v2
Builds on: CLIP SigLIP Vision Transformer Registers MetaCLIP
Closest papers by meaning
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.90)
- MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data Utilization (ICML 2024 · similarity 0.90)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.90)
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.90)
- SuperCLIP: CLIP with Simple Classification Supervision (NeurIPS 2025 · similarity 0.90)
- TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.