Research map / Vision-language and multimodal / CLIP and zero-shot
Exploiting the Asymmetric Uncertainty Structure of Pre-trained VLMs on the Unit Hypersphere
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AsymVLM, a post-hoc probabilistic adapter that models only text embeddings as vMF or power-spherical distributions on the unit hypersphere while keeping image embeddings deterministic, trained with an InfoNCE objective over spherical log-likelihoods.
Problem
Post-hoc probabilistic adaptation of deterministic pretrained VLMs (CLIP, SigLIP) ignores the asymmetric uncertainty between text and images and the fact that embeddings lie on the unit hypersphere, leading to suboptimal uncertainty estimates and retrieval.
Approach
Treat text embeddings as random variables on the unit sphere (vMF or power spherical), keep image embeddings deterministic, and optimize a symmetric-in-batch InfoNCE over the spherical log-likelihood matrix; this generalizes the CLIP cosine-similarity loss with a concentration term that yields a per-text uncertainty 1/kappa.
Key results
- On MS-COCO CLIP, AsymVLM_vMF reaches t2i Recall@1 0.561 and i2t Recall@1 0.392 versus 0.500 and 0.304 for PFE* and PCME++*.
- On CC-200k, AsymVLM_vMF raises i2t Recall@1 to 0.383 vs 0.336 for PFE* and 0.302 for ProbVLM, and it is the only method whose Recall@1 decreases strictly monotonically with uncertainty.
- In robust fine-tuning on MS-COCO, AsymVLM_PS reaches 0.847 CIFAR-10 zero-shot accuracy vs 0.684 for deterministic fine-tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use asymmetric text-probabilistic, image-deterministic adaptation with vMF distributions | CLIP ViT-B/32 · MS-COCO, Flickr-30k, and CC-200k cross-modal retrieval | Recall@1 | Helped |
| Replace vMF with power spherical distributions | AsymVLM · MS-COCO, Flickr-30k, and CC-200k retrieval | Recall@1 | No clear effect |
| Fine-tune with AsymVLM rather than deterministic fine-tuning | CLIP ViT-B/32 fine-tuned on MS-COCO · CIFAR-10 zero-shot classification | accuracy: 0.684 -> 0.847 | Helped |
| Fine-tune with AsymVLM rather than deterministic fine-tuning | CLIP ViT-B/32 fine-tuned on MS-COCO · ImageNet-1k zero-shot classification | accuracy: 0.395 -> 0.507 | Helped |
| Add dummy prompt 'a photo' for none-of-the-above classification | AsymVLM vMF · CIFAR-100 negative samples | negative accuracy: 0.009 -> 0.587 | Helped |
| Model uncertainty in image embeddings instead of text embeddings (ablation) | AsymVLM · CC-200k validation retrieval | Recall@1 | Hurt |
| Use symmetric probabilistic adapters for both modalities (ablation) | AsymVLM · CC-200k validation retrieval | Recall@1 | Hurt |
| Use symmetric probabilistic adapters for both modalities (ablation) | AsymVLM · CC-200k validation uncertainty quantification | Recall@1 across uncertainty levels | Hurt |
| Replace the spherical text distribution with a normalized Gaussian distribution (ablation) | AsymVLM · CC-200k validation retrieval | Recall@1 | Hurt |
| Replace the spherical text distribution with a normalized Gaussian distribution (ablation) | AsymVLM · CC-200k validation uncertainty quantification | Recall@1 trend as uncertainty increases | No clear effect |
Surprising findings
- Deterministic post-hoc adaptation (PFE*, PCME++*) yields identical retrieval to the pretrained CLIP because only variances are learned, so their recall equals the base model.
- AsymVLM(image) (probabilistic image side) shows no decrease in Recall@1 as uncertainty rises, and a symmetric variant performs much worse in recall.
Limitations
- Only text embeddings are modeled probabilistically; image uncertainty is not captured.
- Downstream applications (none-of-the-above rejection and robust fine-tuning) are only illustrative and further tasks are left unexplored.
- Uncertainty evaluation relies on correlation between uncertainty and retrieval recall rather than ground-truth uncertainty.
Experimental setup
Datasets: MS-COCO Flickr-30k Conceptual Captions (CC-200k subset) HierarCap CIFAR-10 CIFAR-100 STL-10 ImageNet-1k
Benchmarks: MS-COCO Flickr-30k CC-200k HierarCap CIFAR-10 CIFAR-100 STL-10 ImageNet-1k
Metrics: Recall@1 R^2 between uncertainty and retrieval error Spearman rank correlation zero-shot accuracy positive and negative accuracy
Baselines: PFE* PCME++* BayesVLM ProbVLM ProLIP Determ. FT
Builds on: CLIP ProbVLM BayesVLM PCME++ ProLIP
Closest papers by meaning
- Probabilistic Language-Image Pre-Training (ICLR 2025 · similarity 0.92)
- FUSE: Quantifying Uncertainty in Vision-Language Models by Bayesian Fusing Epistemic and Aleatoric Uncertainty (ICML 2026 · similarity 0.89)
- Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (ICML 2024 · similarity 0.89)
- From Head to Tail: Towards Balanced Representation in Large Vision-Language Models through Adaptive Data Calibration (CVPR 2025 · similarity 0.89)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.89)
- Mostly Text, Smart Visuals: Asymmetric Text-Visual Pruning for Large Vision-Language Models (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.