Research map / Vision-language and multimodal / CLIP and zero-shot
Improved Probabilistic Image-Text Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PCME++, a probabilistic image-text embedding with a closed-form sampled distance, pseudo-positives, and mixed-sample augmentation, improving image-text matching under false negatives.
Problem
Image-text matching is ambiguous due to many-to-many correspondences and sparse annotations (massive false negatives in COCO); existing probabilistic PCME is expensive due to Monte Carlo sampling and saturates under false negatives.
Approach
Replace the Monte Carlo matching probability with a closed-form expected squared Euclidean distance between Gaussian embeddings (CSD), train with a pairwise BCE loss plus pseudo-positive matching and mixup/cutmix on images with soft match labels.
Key results
- On ViT-L/14, PCME++ reaches COCO 1K R@1 81.8 and 5K R@1 64.3 (RSUM 554.7), above PCME (550.4) and InfoNCE (520.6).
- Ablation on ViT-B/32 shows the full combination of VIB, PP and MSDA reaches ECCV mAP@R 40.1 and RSUM 537.0, versus 535.9 for the baseline without these techniques.
- Uncertainty-based prompt filtering raises ImageNet zero-shot top-1 from 30.43 to 41.82 with best top-K per class.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PCME++ instead of PCME | ViT-L/14 image-text retrieval · COCO 1K | RSUM: 550.4 -> 554.7 | Helped |
| Add VIB, pseudo-positives, and MSDA (ablation) | PCME++ · ECCV Caption | mAP@R: 38.9 -> 40.1 | Helped |
| Add VIB (ablation) | PCME++ without PP or MSDA · ECCV Caption | mAP@R: 38.9 -> 39.3 | Helped |
| Add pseudo-positives (ablation) | PCME++ without VIB or MSDA · ECCV Caption | mAP@R: 38.9 -> 39.0 | No clear effect |
| Add MSDA (ablation) | PCME++ without VIB or PP · ECCV Caption | mAP@R: 38.9 -> 39.0 | No clear effect |
| Replace CSD with Wasserstein 2-distance (ablation) | PCME++ training objective · ECCV Caption | mAP@R: 39.3 -> 1.9 | Hurt |
| Replace CSD with PCME matching probability (ablation) | PCME++ training objective · ECCV Caption | mAP@R: 39.3 -> 39.1 | No clear effect |
| Increase the backbone from ViT-B/32 to ViT-L/14 | PCME++ · COCO 1K | RSUM: 537.0 -> 554.7 | Helped |
| Train with 50% noisy correspondences using PCME++ rather than PCME | ViT-B/32 · ECCV Caption, 50% noise | mAP@R: 35.2 -> 35.7 | Helped |
| Filter prompts using the best top-K prompts for each class (ablation) | PCME++ zero-shot classifier · ImageNet | Top-1 Acc: 30.43 -> 41.82 | Helped |
| Increase PP strength from α=0 to α=0.1 (ablation) | PCME++ with noisy correspondences · ECCV Caption, 50% noise | mAP@R: 35.7 -> 34.4 | Hurt |
Surprising findings
- Deterministic VSE∞ with ViT-L/14 collapses (RSUM 424.3) while probabilistic PCME++ improves with scale.
- Wasserstein 2-distance as training objective performs very poorly (RSUM 121.1), and KL/JS divergence collapse to near zero.
Limitations
- Gains on precision metrics (mAP@R, R-P) over VSE∞ with WSL backbone are not consistent; PCME++ is not best on all metrics.
- Noisy-correspondence setting requires PP weight tuning because PPs can be wrong under strong noise and model selection on noisy validation is unreliable.
- Prompt-filtering uses the ImageNet validation set, so it is not a true zero-shot evaluation.
Experimental setup
Datasets: MS-COCO Caption CC3M 12M RedCaps ImageNet
Benchmarks: COCO Caption 1K COCO Caption 5K ECCV Caption CxC ImageNet
Metrics: R@K RSUM mAP@R R-Precision top-1 accuracy
Baselines: VSE∞ PCME InfoNCE DAA P2RM CLIP zero-shot NCR DECL ViLT VinVL BLIP
Builds on: PCME VSE∞ CLIP GPO Mixup CutMix
Closest papers by meaning
- Probabilistic Language-Image Pre-Training (ICLR 2025 · similarity 0.89)
- Diffusion Bridge: Leveraging Diffusion Model to Reduce the Modality Gap Between Text and Vision for Zero-Shot Image Captioning (CVPR 2025 · similarity 0.87)
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.86)
- Expanding the Scope of Negatives: Boosting Image-Text Matching with Negatives Distribution Guided Learning (AAAI 2025 · similarity 0.85)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.85)
- T-MARS: Improving Visual Representations by Circumventing Text Feature Learning (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.