Research map / Vision-language and multimodal / CLIP and zero-shot
Revisiting the Role of Language Priors in Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Repurposes generative VLMs (BLIP, LLaVA-1.5) as zero-shot image-text scorers via P(text|image), and debiases language priors with a training-free alpha-reweighting estimated from Gaussian-noise images, achieving SOTA on compositionality retrieval benchmarks.
Problem
Zero-shot image-text retrieval with VLMs is hampered by language priors: some benchmarks contain implausible negatives that blind language models can exploit, while others use plausible negatives where the language prior hurts.
Approach
Use the generative likelihood P(text|image) of an image-conditioned LM as a match score (VisualGPTScore) and reweight it by P(text)^-alpha, with P_train(text) estimated Monte Carlo from a few Gaussian noise images, equivalent to PMI^k with k=1/alpha.
Key results
- VisualGPTScore (alpha=0) with BLIP reaches 89.1 Rel and 95.3 Attr on ARO and 93.3/91.0/91.0 on SugarCrepe Replace/Swap/Add, surpassing prior discriminative and fine-tuned methods.
- Debiasing with alpha=1 raises BLIP I-to-T R@1 on COCO from 19.7 to 46.2 and on Flickr30k from 34.6 to 58.7, but generative scores still trail ITMScore on these large-scale retrieval tasks.
- On Winoground, LLaVA-1.5-based VisualGPTScore with alpha=1 reaches 27.5 group score versus 12.5 for LLMScore with ChatGPT and 6.3 for VPEval.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| tune α for prior debiasing (ablation) | BLIP VisualGPTScore · SugarCrepe | accuracy: 93.3 -> 95.1; 91.0 -> 92.4; 91.0 -> 97.4 | Helped |
| set α=1 rather than α=0 (ablation) | BLIP VisualGPTScore · Winoground | Text Score: 27.5 -> 33.7 | Helped |
| tune α on a held-out validation set (ablation) | BLIP VisualGPTScore · Winoground | Text Score: 27.5 -> 36.6_(2.6) | Helped |
| set α=1 rather than α=0 (ablation) | BLIP VisualGPTScore · ARO-Flickr | accuracy: 99.5 -> 44.5 | Hurt |
| set α=1 rather than α=0 (ablation) | BLIP VisualGPTScore · Crepe | accuracy: 73.2 -> 20.6; 78.1 -> 28.3; 79.6 -> 35.6 | Hurt |
| estimate the text prior using Gaussian noise images rather than trainset images (ablation) | BLIP VisualGPTScore debiasing · Winoground | Text Score: 32.20_(1.6) -> 35.95_(0.5) | Helped |
| estimate the text prior using testset images rather than null images (ablation) | BLIP VisualGPTScore debiasing · COCO | R@1 / R@5: 24.4 / 52.6 -> 46.2 / 73.1 | Helped |
| estimate the text prior using testset images rather than null images (ablation) | BLIP VisualGPTScore debiasing · Flickr30k | R@1 / R@5: 27.8 / 62.2 -> 58.7 / 88.0 | Helped |
Surprising findings
- A blind language-only model (P_train(t) from BLIP on Gaussian noise images, or text-only Vera) already achieves SOTA-level accuracy on ARO-Flickr, implying the benchmark can be largely solved without images.
- Generative VLM scores are biased toward common texts even on LAION training-set retrieval, and optimal alpha is non-zero there.
Limitations
- Generative scores still lag ITMScore on large-scale retrieval (COCO, Flickr30k) and the model is biased toward common texts.
- Assumes the image likelihood P(i|t) is identical across train and test, a simplifying assumption that does not fully hold.
Experimental setup
Datasets: LAION-114M ARO Crepe SugarCrepe VL-CheckList Winoground EqBen COCO Flickr30K ImageNet1K
Benchmarks: ARO VL-CheckList SugarCrepe Crepe Winoground EqBen COCO Flickr30K ImageNet1K
Metrics: accuracy text score image score group score R@1 R@5
Baselines: CLIP LAION2B-CLIP LAION5B-CLIP NegCLIP BLIP-ITC BLIP-ITM DAC-LLM DAC-SAM SugarCrepe baselines CLIPScore TIFA VQ2
Builds on: BLIP PMI^k GPTScore CLIPScore ARO
Closest papers by meaning
- Probing Visual Language Priors in VLMs (ICML 2025 · similarity 0.92)
- The Neglected Tails in Vision-Language Models (CVPR 2024 · similarity 0.92)
- Unified Generative and Discriminative Training for Multi-modal Large Language Models (NeurIPS 2024 · similarity 0.92)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.91)
- Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy (ICLR 2024 · similarity 0.91)
- Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models (ICLR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.