Research map / Vision-language and multimodal / CLIP and zero-shot
Probabilistic Language-Image Pre-Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ProLIP, a probabilistic vision-language model pre-trained from scratch on DataComp 1B using only probabilistic objectives (PPCL plus a novel inclusion loss) with an uncertainty token for efficient uncertainty estimation.
Problem
Deterministic VLM embeddings assume one-to-one image-text correspondence, failing to capture the many-to-many multiplicity and aleatoric ambiguity of real image-text data; prior probabilistic VLMs are small-scale or lack zero-shot capability.
Approach
Map each image and text to a diagonal Gaussian, estimate variance via a dedicated [UNC] token with no extra parameters, train with a log-sigmoid probabilistic pairwise contrastive loss, and add an asymmetric inclusion loss enforcing distributional inclusion between image-text pairs and between original and masked inputs; uncertainty then drives prompt re-weighting (BPRW) and hierarchical image tra
Key results
- ProLIP ViT-B/16 trained on 12.8B seen samples reaches 74.6% ImageNet zero-shot accuracy, versus CLIP's 67.2% at 1.28B samples.
- At 1.28B seen samples, ProLIP ViT-B/16 slightly outperforms CLIP on the 38-task average (57.9 vs 57.1), with ImageNet 67.8 vs 67.2.
- Uncertainty-based Bayesian Prompt Re-Weighting with 9 few-shot images per class raises ImageNet zero-shot accuracy from 74.6% to 75.8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace deterministic CLIP embeddings with ProLIP probabilistic embeddings | ViT-B/16 · ImageNet | accuracy: 67.2 -> 67.8 | Helped |
| Replace deterministic CLIP embeddings with ProLIP probabilistic embeddings | ViT-B/16 · VTAB | score: 56.9 -> 58.5 | Helped |
| Replace deterministic CLIP embeddings with ProLIP probabilistic embeddings | ViT-B/16 · Retrieval | score: 53.4 -> 53.0 | Hurt |
| Replace deterministic CLIP embeddings with ProLIP probabilistic embeddings | ViT-B/16 · DataComp 38-task average | score: 57.1 -> 57.9 | Helped |
| Train ProLIP with 12.8B instead of 1.28B seen samples | ProLIP ViT-B/16 · ImageNet | accuracy: 67.8 -> 74.6 | Helped |
| Use inclusion-based rather than deterministic traversal (ablation) | ProLIP (1.28B) · HierarCaps retrieval | precision: 28.4 -> 35.9 | Helped |
| Use inclusion-based rather than deterministic traversal (ablation) | ProLIP (1.28B) · HierarCaps retrieval | R@1: 62.6 -> 62.9 | Helped |
| Use inclusion-based rather than deterministic traversal (ablation) | ProLIP (1.28B) · HierarCaps retrieval | R@1[ROOT]: 0.1 -> 15.3 | Helped |
| Filter prompts by uncertainty statistics (ablation) | ProLIP prompt selection · ImageNet | accuracy: +0.03pp | No clear effect |
| Use BPRW with visual information and no labeled examples | ProLIP prompt selection · ImageNet | accuracy: +0.12pp | Helped |
| Add 5 labeled images per class to BPRW | ProLIP prompt selection · ImageNet | accuracy: 74.7 -> 75.6 | Helped |
| Add 9 labeled images per class to BPRW | ProLIP prompt selection · ImageNet | accuracy: 74.7 -> 75.8 | Helped |
Surprising findings
- PCME++'s probabilistic matching loss fails to converge when used standalone without a deterministic loss, whereas PPCL converges on its own.
- Averaging prompt distributions with 1/N rather than 1/N^2 variance scaling gave slightly better ImageNet zero-shot accuracy (74.58 vs 74.51).
Limitations
- Large-scale pretraining was costly, so larger backbones (ViT-H, ViT-SO400M, ViT-G) were only obtained by fine-tuning SigLIP/DFN checkpoints rather than training from scratch.
- BPRW needs a few labeled images per class to reach its best gains, which goes beyond pure zero-shot setting.
- Image uncertainty is strongly dataset-dependent and the diagonal covariance assumption is a simplification.
Experimental setup
Datasets: DataComp 1B ConceptualCaptions 3M ConceptualCaptions 12M RedCaps HierarCaps COCO Caption
Benchmarks: ImageNet ImageNet distribution shifts VTAB DataComp 38-task suite Flickr30k MS-COCO Caption HierarCaps
Metrics: zero-shot accuracy recall@1 precision
Baselines: CLIP SigLIP PCME++ ProbVLM MAP
Builds on: CLIP PCME++ SigLIP ProbVLM
Closest papers by meaning
- Exploiting the Asymmetric Uncertainty Structure of Pre-trained VLMs on the Unit Hypersphere (NeurIPS 2025 · similarity 0.92)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.91)
- Probing Visual Language Priors in VLMs (ICML 2025 · similarity 0.89)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.89)
- Improved Probabilistic Image-Text Representations (ICLR 2024 · similarity 0.89)
- DH-Set: Improving Vision-Language Alignment with Diverse and Hybrid Set-Embeddings Learning (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.